How can I train my own model?

DeepPavlov Library Models
1

Target

I want to train my model for Cyber NER. But met some problem. DeepPavlov try to load existed models instead initialize a new one.

Error Log

2023-05-07 12:46:20.714 WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 66: TorchTrainer got additional init parameters ['pytest_max_batches', 'pytest_batch_size'] that will be ignored:
2023-05-07 12:46:29.126 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 104: [saving vocabulary to D:\Code\iscc-big-data\ner_model\tag.dict]
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForTokenClassification: ['cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2023-05-07 12:46:30.322 WARNING in 'deeppavlov.models.torch_bert.torch_transformers_sequence_tagger'['torch_transformers_sequence_tagger'] at line 312: Init from scratch. Load path D:\Code\iscc-big-data\ner_model\model_crf.pth.tar does not exist.
0it [00:00, ?it/s]
2023-05-07 12:46:30.328 WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 173: Got empty data iterable for scoring
Traceback (most recent call last):
  File "D:\Code\iscc-big-data\dp.py", line 23, in <module>
    ner_model = train_model(config)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\__init__.py", line 31, in train_model
    train_evaluate_model_from_config(config, install=install, download=download, recursive=recursive)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\core\commands\train.py", line 113, in train_evaluate_model_from_config
    trainer.train(iterator)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\core\trainers\nn_trainer.py", line 334, in train
    self.train_on_batches(iterator)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\core\trainers\torch_trainer.py", line 41, in train_on_batches
    super(TorchTrainer, self).train_on_batches(iterator=iterator)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\core\trainers\nn_trainer.py", line 274, in train_on_batches
    self._validate(iterator)
  File "D:\ProgramData\Miniconda3\envs\DeepPavlov\lib\site-packages\deeppavlov\core\trainers\nn_trainer.py", line 175, in _validate
    metrics = list(report['metrics'].items())
AttributeError: 'NoneType' object has no attribute 'items'

Code

from deeppavlov import configs, train_model
from deeppavlov.core.commands.utils import parse_config

train_file = 'train.txt'
valid_file = 'valid.txt'
test_file = 'test.txt'

config = config = parse_config('ner_ontonotes_bert')
config['dataset_reader']['data_path'] = 'opt/'
config['dataset_reader']['train'] = train_file
# config['dataset_reader']['valid'] = valid_file
# config['dataset_reader']['test'] = test_file
config["chainer"]["in"] = ["x", "y"]
config['chainer']['pipe'][0]['do_lower_case'] = False
config['chainer']['pipe'][1]['save_path'] = './ner_model/tag.dict'
config['chainer']['pipe'][1]['load_path'] = './ner_model/tag.dict'
config['chainer']['pipe'][2]['n_tags'] = 23
config['chainer']['pipe'][2]['save_path'] = './ner_model/model'
config['chainer']['pipe'][2]['load_path'] = './ner_model/model'

print(config)

ner_model = train_model(config)

Part of Trainset

Remote 5
Login 6
Service 6
( 0
RLS 0
) 0
1.0.0 3
does 0
not 0
properly 0
clear 0
account 1
information 1
when 0
switching 0

How can I modify the config in my code in order to create a brand new model for it?
Thank you all.

2

Dear, @sfc9982 ,

I apologize for the late response.

To train your own model changing the following config parameters should suffice:

config = parse_config('ner_ontonotes_bert')
config['dataset_reader']['data_path'] = ‘./opt/’
config['chainer']['pipe'][1]['save_path'] = './ner_model/tag.dict' 
config['chainer']['pipe'][1]['load_path'] = './ner_model/tag.dict'
config['chainer']['pipe'][2]['save_path'] = './ner_model/model'
config['chainer']['pipe'][2]['load_path'] = './ner_model/model'
del config[‘metadata’][‘download’]

Please make sure that your training data is found by the path you set in [‘dataset_reader’][‘data_path’] and delete our default training set from metadata (shown in the code above).

Hope this will help.

1 Like
3

Thanks, I will try it out.

1 Like