How to generate confidence scores uisng configs.squad.squad_bert_infer model

DeepPavlov Library
1

Hi,

I am using the below code to generate the answers using BERT model.
But how do I generate the respective confidence scores/metrics for each answer ? Is there any method that can show the metrics/probability or confidence score for each answer?

from deeppavlov import build_model, configs
bert_model = build_model(configs.squad.squad_bert_infer, download=False)

q1 = “For what time period did Pradhan Mantri Vaya Vandana Yojana got opened?”
answers = bert_model([sample_text], [q1])

Answer:
[[‘one year i.e. from 4th May 2017 to 3rd May 2018’], [685], [103742.2265625]]

2

Hi!

In your example:

[[‘one year i.e. from 4th May 2017 to 3rd May 2018’], [685], [103742.2265625]]

model outputs an answer, answer position in text in characters and unnormalised logit, which can be used as confidence score.

If you want some value in [0, 1] as score, than you can make few changes in squad_bert.json:

modify to:
→ "out": ["ans_start_predicted", "ans_end_predicted", "logits", "score"]
and

modify to:
→ "out": ["ans_predicted", "ans_start_predicted", "score"]

and use the same bert_squad_infer model (it depends on bert_squad).

1 Like
3

Thanks much! It worked. Another quick thing, if I have to get top 3 answers and their probability scores using Infer model, how do I get them ? Please help!

4

I replied you here: Get top 3 answers and their respective confidence scores using squad_bert_infer_model