How to add validation loss?

1

I am using BertClassifier and rusentiment config file and modifying it slightly. I also integrated tensorboard to display metrics however, it gives me only “training loss” but not “Validation loss”, how can I add it to my config file? I am solving multi-class classification problem if that helps.

I tried to like this:
{
“name”: “log_loss”,
“inputs”: [
“y_onehot”,
“y_pred_labels”
]
}

but it is throwing an error.

same question about “precision and recall”. I know we have f1 weighted and f1 macro however, I just need a precision and recall metrics separately as well. Thank you

1 Like
2

Hi!

Looks like y_true that comes into sklearn.metrics.log_loss is list of np.arrays and it fails to work with it.
To workaround, you can use y or y_ids as y_true in log loss computation. This change will compute log loss for your case:

{
        "name": "log_loss",
        "inputs": [
          "y",
          "y_pred_probas"
        ]
      }

Unfortunately, there is no ready to use precision and recall metrics in DeepPavlov.
You can create file my_metrics.py and define precision, recall metrics like it was done for different correlations: link.
Add importing of my_metrics.py:

"metadata": {
    "imports": ["my_metrics"],

and use them in your config file as regular metrics.

3

Thank you.
Where to store the my_metric.py file so the config file can recognizes it?

4

such structure worked for me:

.
├── my_metrics.py
└── rusentiment_convers_bert.json

running python -m deeppavlov ... also from this dir.