Модель нормально дообучалась на CPU. Захотел ускорить процесс и настроил CUDA. Tensorflow импортируется без ошибок, видеокарту видит. Уменьшил batch_size до 4, чтобы избежать переполнения видеопамяти, в остальном стандартная конфигурация ner_rus_bert.json. Ставлю на обучение и через несколько итераций получаю следующее:
2020-04-08 02:45:16.338 INFO in ‘deeppavlov.core.trainers.nn_trainer’[‘nn_trainer’] at line 307: Ran out of patience
I0408 02:45:16.338182 7940 nn_trainer.py:307] Ran out of patience
2020-04-08 02:45:17.494 INFO in ‘deeppavlov.core.data.simple_vocab’[‘simple_vocab’] at line 115: [loading vocabulary from C:\Users\User1\.deeppavlov\models\ner_rus_bert\tag.dict]
I0408 02:45:17.494161 7940 simple_vocab.py:115] [loading vocabulary from C:\Users\User1\.deeppavlov\models\ner_rus_bert\tag.dict]
2020-04-08 02:45:17.517901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-04-08 02:45:17.524495: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-04-08 02:45:17.530132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-04-08 02:45:17.533437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-08 02:45:17.537460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-04-08 02:45:17.539984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-04-08 02:45:17.543588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4712 MB memory) → physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-04-08 02:45:41.647 INFO in ‘deeppavlov.core.models.tf_model’[‘tf_model’] at line 51: [loading model from C:\Users\User1\.deeppavlov\models\ner_rus_bert\model]
I0408 02:45:41.647132 7940 tf_model.py:51] [loading model from C:\Users\User1\.deeppavlov\models\ner_rus_bert\model]
2020-04-08 02:45:57.471 DEBUG in ‘deeppavlov.metrics.fmeasure’[‘fmeasure’] at line 394: processed 2954 tokens with 144 phrases; found: 148 phrases; correct: 0.precision: 66.89%; recall: 68.75%; FB1: 67.81
LOC: precision: 84.00%; recall: 77.78%; F1: 80.77 25 ORG: precision: 48.39%; recall: 46.88%; F1: 47.62 62 PERSON: precision: 78.69%; recall: 90.57%; F1: 84.21 61I0408 02:45:57.471564 7940 fmeasure.py:394] processed 2954 tokens with 144 phrases; found: 148 phrases; correct: 0.
precision: 66.89%; recall: 68.75%; FB1: 67.81
LOC: precision: 84.00%; recall: 77.78%; F1: 80.77 25 ORG: precision: 48.39%; recall: 46.88%; F1: 47.62 62 PERSON: precision: 78.69%; recall: 90.57%; F1: 84.21 61{“valid”: {“eval_examples_count”: 344, “metrics”: {“ner_f1”: 67.8082, “ner_token_f1”: 83.3539}, “time_spent”: “0:00:14”}}
2020-04-08 02:46:11.932 DEBUG in ‘deeppavlov.metrics.fmeasure’[‘fmeasure’] at line 394: processed 3048 tokens with 113 phrases; found: 124 phrases; correct: 0.precision: 74.19%; recall: 81.42%; FB1: 77.64
LOC: precision: 74.47%; recall: 87.50%; F1: 80.46 47 ORG: precision: 60.00%; recall: 58.06%; F1: 59.02 30 PERSON: precision: 82.98%; recall: 92.86%; F1: 87.64 47I0408 02:46:11.932805 7940 fmeasure.py:394] processed 3048 tokens with 113 phrases; found: 124 phrases; correct: 0.
precision: 74.19%; recall: 81.42%; FB1: 77.64
LOC: precision: 74.47%; recall: 87.50%; F1: 80.46 47 ORG: precision: 60.00%; recall: 58.06%; F1: 59.02 30 PERSON: precision: 82.98%; recall: 92.86%; F1: 87.64 47{“test”: {“eval_examples_count”: 400, “metrics”: {“ner_f1”: 77.6371, “ner_token_f1”: 90.6801}, “time_spent”: “0:00:15”}}
2020-04-08 02:46:12.807 INFO in ‘deeppavlov.core.data.simple_vocab’[‘simple_vocab’] at line 115: [loading vocabulary from C:\Users\User1\.deeppavlov\models\ner_rus_bert\tag.dict]
I0408 02:46:12.807576 7940 simple_vocab.py:115] [loading vocabulary from C:\Users\User1\.deeppavlov\models\ner_rus_bert\tag.dict]
2020-04-08 02:46:12.828862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-04-08 02:46:12.835150: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-04-08 02:46:12.840317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-04-08 02:46:12.843330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-08 02:46:12.847133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-04-08 02:46:12.849532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-04-08 02:46:12.852399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4712 MB memory) → physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-04-08 02:46:36.444 INFO in ‘deeppavlov.core.models.tf_model’[‘tf_model’] at line 51: [loading model from C:\Users\User1\.deeppavlov\models\ner_rus_bert\model]
I0408 02:46:36.444068 7940 tf_model.py:51] [loading model from C:\Users\User1\.deeppavlov\models\ner_rus_bert\model]
Если я правильно понимаю, то тут обучение не завершилось, хоть явная ошибка не написана. И проблема похоже связана как-то с tensorflow и тем как распределяется нагрузка на видеокарты (которая у меня только одна). Подскажите, пожалуйста, что нужно предпринять? Изменить какие-то параметры в самом ner_rus_bert.json, evolve.py или где-то внутри tensorflow?