We are currently using deeppavlov’s framework to extract custom named entities. We have trained a model on a labeled dataset but have encountered unexpected memory usage issues. The issues cause termination of the training / evaluation process.
For some reason during the training and evaluation stages the memory usage shoots up:
The maximum memory available is 8GB. We have GPU available and in both training and prediction / evaluation scripts have a line: os.environ[‘CUDA_VISIBLE_DEVICES’] = ‘0’. Training dataset is only 700 sentences.
Initially it was crashing during training but I reduced the batch size from 16 to 4 and it manages to go through a few epochs (still crashing at the end).
It is more bizarre with the evaluate_model function or python -m deeppavlov evaluate command. The process gets killed (I assume due to memory limitations) after loading vocabulary from a trained NER model. aka last Info log in ‘deeppavlov.core.data.simple_vocab’ at line 115.
Would you kindly point to a source of possible high memory usage and how to remedy it?