Ner ontonotes Bert model training with ontonotes dataset doesn't finish even after 4 days

DeepPavlov Library
1

I noticed something abnormal. I ran train model with ner ontonotes bert probas config. It did not stop even after 4 days. I’m using a i5 processor, two of them on windows 10. Number of iterations set to 1. No GPU in my machine. I’m using deeppavlov 0.12.1. validation patience has default value. Am I doing something incorrect or it is expected on my machine?

2

Hi!

Could you provide your configuration file and logs output?

Before training is started we run evaluation on validation set, it might that validation step takes too long on cpu-only machine. Also, do you see that cpu usage is high while training is running?

3

Hi Mr.YuraKoratov,
The logs and the config file are here sir. The training is done without the model but with tag.dict.

Best regards,
Ravi

(Attachment logs.zip is missing)

(Attachment ner_ontonotes_bert_probas.json is missing)

4

Looks like attachment is missing.

5

Hi Mr.YuraKoratov, I’m attaching the config file. I set the number of epochs to 1 before train_model(config).
Please see the logs in this mail. You know, I attached the logs in the form of zip file and the config file but zip isn’t allowed by yahoo. I thought only the logs need to be attached since I presumed the config file would have been sent to you. Sorry about that. Please take a look. I didn’t see the CPU usage btw for those four days.
Best regards,
Ravi

(Attachment ner_ontonotes_bert_probas.json is missing)

6

Still no attachment.
I think there is a problem that attached files via mail are not available on the DeepPavlov forum.
Could you try to add them via forum, not via mail?

7

Hi Mr.YuraKuratov, Attaching the logs and config file.

I don’t know how to attach directly to the topic. So I’m attaching the logs and config file in their order of appearance from my google drive.

8

Hi Mr.YuraKuratov, Please help.

1 Like
9

Hi mr.yurakuratov, please help.

10

Hi mr.yurakuratov, please help. I need to finish training before end of this week. Please tell me about how many iterations I should train. I may be annotating and adding a maximum of two or three thousand sentences to the base dataset and train.

11

Hello Mr.Yurakuratov, urgent help is needed. Please find time for this issue.

12

Hi!
As I see from logs, training is still going cause stopping criteria was not satisfied.
In your config file two stopping criteria are set:

  1. 30 epochs
  2. metrics on validation should not increase for 100 (patience) validations (each validation - every 40 batches).
    image
    image694×498 22.7 KB

On the plot ner_token_f1 metric is still increasing.