NER training ontonotes.bert ERROR

when i try to train ner.ontonotes_bert with my own dataset …I get the following error.

Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

2 root error(s) found.
(0) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [10,10] rhs shape= [37,37]
[[node save/Assign_76 (defined at C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_model.py:54) ]]
[[save/RestoreV2/_521]]
(1) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [10,10] rhs shape= [37,37]
[[node save/Assign_76 (defined at C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_model.py:54) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_76:
EMA/ner/Transition_Params/ExponentialMovingAverage (defined at C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:650)

Input Source operations connected to node save/Assign_76:
EMA/ner/Transition_Params/ExponentialMovingAverage (defined at C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:650)

Original stack trace for ‘save/Assign_76’:
File “c:\Users\Shaleen.vscode\extensions\ms-python.python-2020.1.58038\pythonFiles\ptvsd_launcher.py”, line 43, in
main(ptvsdArgs)
File “c:\Users\Shaleen.vscode\extensions\ms-python.python-2020.1.58038\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py”, line 432, in main
run()
File “c:\Users\Shaleen.vscode\extensions\ms-python.python-2020.1.58038\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py”, line 316, in run_file
runpy.run_path(target, run_name=‘main’)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File “c:\Users\Shaleen\Documents\Visual Studio 2019\Python.vscode\Sem4\Chatbot\train.py”, line 3, in
ner_model = train_model(configs.ner.ner_ontonotes_bert)
File "C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov_init
.py", line 32, in train_model
train_evaluate_model_from_config(config, download=download, recursive=recursive)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\commands\train.py”, line 121, in train_evaluate_model_from_config
trainer.train(iterator)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\trainers\nn_trainer.py”, line 333, in train
self.fit_chainer(iterator)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\trainers\fit_trainer.py”, line 104, in fit_chainer
component = from_params(component_config, mode=‘train’)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\common\params.py”, line 106, in from_params
component = obj(**dict(config_params, **kwargs))
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_backend.py”, line 78, in call
obj.init(*args, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_backend.py”, line 28, in _wrapped
return func(*args, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py”, line 529, in init
**kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py”, line 259, in init
self.load()
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_backend.py”, line 28, in _wrapped
return func(*args, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py”, line 457, in load
return super().load(exclude_scopes=exclude_scopes, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_model.py”, line 251, in load
return super().load(exclude_scopes=exclude_scopes, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\deeppavlov\core\models\tf_model.py”, line 54, in load
saver = tf.train.Saver(var_list)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saver.py”, line 825, in init
self.build()
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saver.py”, line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saver.py”, line 875, in _build
build_restore=build_restore)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saver.py”, line 508, in _build_internal
restore_sequentially, reshape)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saver.py”, line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py”, line 72, in restore
self.op.get_shape().is_fully_defined())
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\ops\state_ops.py”, line 227, in assign
validate_shape=validate_shape)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\ops\gen_state_ops.py”, line 70, in assign
use_locking=use_locking, name=name)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\util\deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\ops.py”, line 3616, in create_op
op_def=op_def)
File “C:\Users\Shaleen\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\framework\ops.py”, line 2005, in init
self._traceback = tf_stack.extract_stack()
File “C:\Users\Shaleen\Documents\Visual Studio 2019\Python.vscode\Sem4\Chatbot\train.py”, line 3, in
ner_model = train_model(configs.ner.ner_ontonotes_bert)

I tried looking through the github issues but I can’t find the solution anywhere

Hi @ShaleenAg,
The simplest solution would be to change the NER_PATH variable in your configuration file. This way save and load paths for models trained on your own data would not be the same as for the pre-trained models.

I’m sorry how do I do that? I havent been able to figure it out …besides I need the pre trained model…I was kinda hoping to fine tune it for my needs…I only have around 150 sentences or so …I cant really train it from scratch