Упала точность обучения

Собрал dataset insult ru. Evaluation выдает результат

2024-07-23 08:13:01.310 WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 66: TorchTrainer got additional init parameters ['device'] that will be ignored:
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-23 08:13:04.34 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
172it [02:53,  1.01s/it]
{"train": {"eval_examples_count": 10961, "metrics": {"roc_auc": 0.996, "accuracy": 0.986, "f1_macro": 0.9842}, "time_spent": "0:02:54"}}
37it [00:36,  1.01it/s]
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9074, "accuracy": 0.8481, "f1_macro": 0.8289}, "time_spent": "0:00:37"}}
37it [00:36,  1.01it/s]
{"test": {"eval_examples_count": 2344, "metrics": {"roc_auc": 0.9087, "accuracy": 0.8515, "f1_macro": 0.8335}, "time_spent": "0:00:37"}}

После того как обновил dataset, добавил новые записи, evaluation выдал результат, модель перестала обучать. Пробавал рабивать с разными пропорциями dataset, результат всегда неудовлетворительный

WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 66: TorchTrainer got additional init parameters ['device'] that will be ignored:

Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
{"train": {"eval_examples_count": 11781, "metrics": {"roc_auc": 0.4848, "accuracy": 0.1035, "f1_macro": 0.1018}, "time_spent": "0:03:05"}}
27it [00:26, 1.04it/s]
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.1123, "f1_macro": 0.1065}, "time_spent": "0:00:27"}}
53it [00:51, 1.02it/s]
{"test": {"eval_examples_count": 3366, "metrics": {"roc_auc": 0.0, "accuracy": 0.1037, "f1_macro": 0.1025}, "time_spent": "0:00:52"}}

вот мой конфиг:

{
  "dataset_reader": {
    "class_name": "basic_classification_reader",
    "x": "Comment",
    "y": "Class",
    "data_path": "{PROJECT_PATH}/data/"
  },
  "dataset_iterator": {
    "class_name": "basic_classification_iterator",
    "seed": 42
  },
  "chainer": {
    "in": [
      "x"
    ],
    "in_y": [
      "y"
    ],
    "pipe": [
      {
        "class_name": "torch_transformers_preprocessor",
        "vocab_file": "{TRANSFORMER}",
        "do_lower_case": true,
        "max_seq_length": 64,
        "in": [
          "x"
        ],
        "out": [
          "bert_features"
        ]
      },
      {
        "id": "classes_vocab",
        "class_name": "simple_vocab",
        "fit_on": [
          "y"
        ],
        "save_path": "{MODEL_PATH}/classes.dict",
        "load_path": "{MODEL_PATH}/classes.dict",
        "in": [
          "y"
        ],
        "out": [
          "y_ids"
        ]
      },
      {
        "in": [
          "y_ids"
        ],
        "out": [
          "y_onehot"
        ],
        "class_name": "one_hotter",
        "depth": "#classes_vocab.len",
        "single_vector": true
      },
      {
        "class_name": "torch_transformers_classifier",
        "n_classes": "#classes_vocab.len",
        "return_probas": true,
        "pretrained_bert": "{TRANSFORMER}",
        "save_path": "{MODEL_PATH}/model",
        "load_path": "{MODEL_PATH}/model",
        "optimizer": "AdamW",
        "optimizer_parameters": {
          "lr": 1e-05
        },
        "learning_rate_drop_patience": 5,
        "learning_rate_drop_div": 2.0,
        "in": [
          "bert_features"
        ],
        "in_y": [
          "y_ids"
        ],
        "out": [
          "y_pred_probas"
        ]
      },
      {
        "in": [
          "y_pred_probas"
        ],
        "out": [
          "y_pred_ids"
        ],
        "class_name": "proba2labels",
        "max_proba": true
      },
      {
        "in": [
          "y_pred_ids"
        ],
        "out": [
          "y_pred_labels"
        ],
        "ref": "classes_vocab"
      }
    ],
    "out": [
      "y_pred_labels"
    ]
  },
  "train": {
    "device": "gpu",
    "epochs": 100,
    "batch_size": 64,
    "metrics": [
      {
        "name": "roc_auc",
        "inputs": [
          "y_onehot",
          "y_pred_probas"
        ]
      },
      "accuracy",
      "f1_macro"
    ],
    "validation_patience": 5,
    "val_every_n_epochs": 1,
    "log_every_n_epochs": 1,
    "show_examples": false,
    "evaluation_targets": [
      "train",
      "valid",
      "test"
    ],
    "class_name": "torch_trainer"
  },
  "metadata": {
    "NAME": "Russian Language Toxic Comments",
    "variables": {
      "ROOT_PATH": "./data",
      "PROJECT_PATH": "{ROOT_PATH}/rlt",
      "MODELS_PATH": "{PROJECT_PATH}/models",
      "MODEL_PATH": "{MODELS_PATH}/classifiers",
      "TRANSFORMER": "bert-base-uncased",
      "LOGS_PATH": "{PROJECT_PATH}/logs"
    }
  }
}

Здравствуйте! Подскажите вы обучали модель на обновлённой версии датасета прежде чем запустить валидацию?

Да, конечно, сначало обучил датасет, потом валидировал

Возможно обучение прервалось на ранних стадиях, метрики на тренировочной выборке слишком низкие. Можете проверить логи обучения? Качество модели на тренировочной выборке поднималось выше тех значений которые вы прислали?

я несколько раз пытался обучить на новом датасете, разбивая его на 15%-15%-70% и 10%-20%-70% (valid-test-train)

вот лог от train:

2024-07-23 19:56:31.777 WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 66: TorchTrainer got additional init parameters ['device'] that will be ignored:
2024-07-23 19:56:33.747 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 104: [saving vocabulary to /www/kolas/data/rlt/models/classifiers/classes.dict]
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-23 19:56:34.712 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
37it [00:37,  1.03s/it]
2024-07-23 19:57:13.886 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 198: Initial best roc_auc of 0.9074
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9074, "accuracy": 0.8481, "f1_macro": 0.8289}, "time_spent": "0:00:39", "epochs_done": 0, "batches_seen": 0, "train_examples_seen": 0, "impatience": 0, "patience_limit": 5}}
172it [09:48,  3.42s/it]
1it [00:01,  1.00s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.96, "accuracy": 0.9844, "f1_macro": 0.9765}, "time_spent": "0:10:28", "epochs_done": 1, "batches_seen": 172, "train_examples_seen": 10961, "loss": 0.12014580381557692}}
37it [00:36,  1.02it/s]
2024-07-23 20:07:39.763 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.9074
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9043, "accuracy": 0.854, "f1_macro": 0.8317}, "time_spent": "0:11:04", "epochs_done": 1, "batches_seen": 172, "train_examples_seen": 10961, "impatience": 1, "patience_limit": 5}}
172it [09:42,  3.39s/it]
1it [00:00,  1.01it/s]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 1.0, "accuracy": 1.0, "f1_macro": 1.0}, "time_spent": "0:20:48", "epochs_done": 2, "batches_seen": 344, "train_examples_seen": 21922, "loss": 0.10207617190276641}}
37it [00:36,  1.03it/s]
2024-07-23 20:17:59.394 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.9074
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9072, "accuracy": 0.8494, "f1_macro": 0.8331}, "time_spent": "0:21:24", "epochs_done": 2, "batches_seen": 344, "train_examples_seen": 21922, "impatience": 2, "patience_limit": 5}}
172it [09:37,  3.36s/it]
1it [00:00,  1.03it/s]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 1.0, "accuracy": 1.0, "f1_macro": 1.0}, "time_spent": "0:31:02", "epochs_done": 3, "batches_seen": 516, "train_examples_seen": 32883, "loss": 0.09254124161224206}}
37it [00:35,  1.03it/s]
2024-07-23 20:28:13.633 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.9074
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9062, "accuracy": 0.8587, "f1_macro": 0.838}, "time_spent": "0:31:38", "epochs_done": 3, "batches_seen": 516, "train_examples_seen": 32883, "impatience": 3, "patience_limit": 5}}
172it [10:01,  3.50s/it]
1it [00:00,  1.04it/s]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 1.0, "accuracy": 1.0, "f1_macro": 1.0}, "time_spent": "0:41:41", "epochs_done": 4, "batches_seen": 688, "train_examples_seen": 43844, "loss": 0.08036662430805695}}
37it [00:35,  1.04it/s]
2024-07-23 20:38:52.122 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.9074
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9033, "accuracy": 0.8621, "f1_macro": 0.844}, "time_spent": "0:42:17", "epochs_done": 4, "batches_seen": 688, "train_examples_seen": 43844, "impatience": 4, "patience_limit": 5}}
172it [18:58,  6.62s/it] 
1it [00:00,  1.00it/s]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 1.0, "accuracy": 1.0, "f1_macro": 1.0}, "time_spent": "1:01:17", "epochs_done": 5, "batches_seen": 860, "train_examples_seen": 54805, "loss": 0.06288564873339478}}
37it [00:38,  1.03s/it]
2024-07-23 20:58:30.344 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.9074
2024-07-23 20:58:30.344 INFO in 'deeppavlov.core.models.torch_model'['torch_model'] at line 206: ----------Current LR is decreased in 2.0 times----------
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9063, "accuracy": 0.8596, "f1_macro": 0.8416}, "time_spent": "1:01:55", "epochs_done": 5, "batches_seen": 860, "train_examples_seen": 54805, "impatience": 5, "patience_limit": 5}}
2024-07-23 20:58:31.130 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 326: Ran out of patience
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-23 20:58:32.333 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
172it [02:55,  1.02s/it]
{"train": {"eval_examples_count": 10961, "metrics": {"roc_auc": 0.996, "accuracy": 0.986, "f1_macro": 0.9842}, "time_spent": "0:02:56"}}
37it [00:36,  1.02it/s]
{"valid": {"eval_examples_count": 2350, "metrics": {"roc_auc": 0.9074, "accuracy": 0.8481, "f1_macro": 0.8289}, "time_spent": "0:00:37"}}
37it [00:36,  1.02it/s]
{"test": {"eval_examples_count": 2344, "metrics": {"roc_auc": 0.9087, "accuracy": 0.8515, "f1_macro": 0.8335}, "time_spent": "0:00:37"}}
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-23 21:02:41.818 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
2024-07-23 21:02:43.65 INFO in 'deeppavlov.core.models.torch_model'['torch_model'] at line 175: Saving model to /www/kolas/data/rlt/models/classifiers/model.pth.tar.

Насколько я понимаю это логи обучения на первой версии датасета, у вас остались логи обучения на обновлённом датасете?

да, по ошибке обучал на стром датасете. вот логи от последней версии:

2024-07-24 20:17:05.898 WARNING in 'deeppavlov.core.trainers.fit_trainer'['fit_trainer'] at line 66: TorchTrainer got additional init parameters ['device'] that will be ignored:
2024-07-24 20:17:09.912 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 104: [saving vocabulary to /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/classes.dict]
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-24 20:17:10.980 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
2024-07-24 20:17:10.982 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 151: Init from scratch. Load path /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/model.pth.tar does not exist.
27it [00:27,  1.02s/it]
2024-07-24 20:17:38.640 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 198: Initial best roc_auc of 0.0
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.3345, "f1_macro": 0.254}, "time_spent": "0:00:28", "epochs_done": 0, "batches_seen": 0, "train_examples_seen": 0, "impatience": 0, "patience_limit": 5}}
185it [27:54,  9.05s/it] 
1it [00:01,  1.01s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.0, "accuracy": 0.8438, "f1_macro": 0.7714}, "time_spent": "0:28:24", "epochs_done": 1, "batches_seen": 185, "train_examples_seen": 11781, "loss": 2.0333578541472153}}
27it [00:26,  1.02it/s]
2024-07-24 20:46:00.522 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.0
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.7445, "f1_macro": 0.6627}, "time_spent": "0:28:50", "epochs_done": 1, "batches_seen": 185, "train_examples_seen": 11781, "impatience": 1, "patience_limit": 5}}
185it [10:42,  3.47s/it]
1it [00:01,  1.01s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.0, "accuracy": 0.8281, "f1_macro": 0.7972}, "time_spent": "0:39:33", "epochs_done": 2, "batches_seen": 370, "train_examples_seen": 23562, "loss": 0.8027426006020726}}
27it [00:26,  1.01it/s]
2024-07-24 20:57:10.343 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.0
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.7398, "f1_macro": 0.7255}, "time_spent": "0:40:00", "epochs_done": 2, "batches_seen": 370, "train_examples_seen": 23562, "impatience": 2, "patience_limit": 5}}
185it [11:00,  3.57s/it]
1it [00:01,  1.02s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.0, "accuracy": 0.6094, "f1_macro": 0.5975}, "time_spent": "0:51:01", "epochs_done": 3, "batches_seen": 555, "train_examples_seen": 35343, "loss": 0.665641714914425}}
27it [00:26,  1.02it/s]
2024-07-24 21:08:38.234 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.0
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.5663, "f1_macro": 0.5644}, "time_spent": "0:51:28", "epochs_done": 3, "batches_seen": 555, "train_examples_seen": 35343, "impatience": 3, "patience_limit": 5}}
185it [10:51,  3.52s/it]
1it [00:01,  1.01s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.0, "accuracy": 0.8281, "f1_macro": 0.7418}, "time_spent": "1:02:20", "epochs_done": 4, "batches_seen": 740, "train_examples_seen": 47124, "loss": 0.5809152403191941}}
27it [00:26,  1.01it/s]
2024-07-24 21:19:57.579 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.0
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.8093, "f1_macro": 0.7561}, "time_spent": "1:02:47", "epochs_done": 4, "batches_seen": 740, "train_examples_seen": 47124, "impatience": 4, "patience_limit": 5}}
185it [11:00,  3.57s/it]
1it [00:01,  1.03s/it]
{"train": {"eval_examples_count": 64, "metrics": {"roc_auc": 0.0, "accuracy": 0.875, "f1_macro": 0.8259}, "time_spent": "1:13:49", "epochs_done": 5, "batches_seen": 925, "train_examples_seen": 58905, "loss": 0.5243978982036178}}
27it [00:27,  1.02s/it]
2024-07-24 21:31:26.880 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 211: Did not improve on the roc_auc of 0.0
2024-07-24 21:31:26.880 INFO in 'deeppavlov.core.models.torch_model'['torch_model'] at line 206: ----------Current LR is decreased in 2.0 times----------
2024-07-24 21:31:26.880 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 151: Init from scratch. Load path /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/model.pth.tar does not exist.
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.8134, "f1_macro": 0.7626}, "time_spent": "1:14:16", "epochs_done": 5, "batches_seen": 925, "train_examples_seen": 58905, "impatience": 5, "patience_limit": 5}}
2024-07-24 21:31:26.882 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 326: Ran out of patience
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-24 21:31:27.766 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
2024-07-24 21:31:27.768 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 151: Init from scratch. Load path /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/model.pth.tar does not exist.
185it [03:11,  1.03s/it]
{"train": {"eval_examples_count": 11781, "metrics": {"roc_auc": 0.6178, "accuracy": 0.0036, "f1_macro": 0.0034}, "time_spent": "0:03:12"}}
27it [00:27,  1.04s/it]
{"valid": {"eval_examples_count": 1683, "metrics": {"roc_auc": 0.0, "accuracy": 0.0036, "f1_macro": 0.0035}, "time_spent": "0:00:28"}}
53it [00:57,  1.08s/it]
{"test": {"eval_examples_count": 3366, "metrics": {"roc_auc": 0.0, "accuracy": 0.0065, "f1_macro": 0.0062}, "time_spent": "0:00:58"}}
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-07-24 21:36:04.713 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 96: Unable to place component TorchTransformersClassifierModel on GPU, since no CUDA GPUs are available. Using CPU.
2024-07-24 21:36:04.715 WARNING in 'deeppavlov.core.models.torch_model'['torch_model'] at line 151: Init from scratch. Load path /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/model.pth.tar does not exist.
2024-07-24 21:36:04.719 INFO in 'deeppavlov.core.models.torch_model'['torch_model'] at line 175: Saving model to /Users/alexanderme/Documents/www/kolas/data/rlt/models/classifiers/model.pth.tar.

Разобрался в чем проблема… В dataset были добавлены очень короткие сообщения по несколько слов. После того как удалил все сообщения короче 5 слов, обучение прошло успешно