Hi folks, I would like to know how Hugging Face estimates the confidence score, displayed when we use QA model from the Hugging Face “pipeline”. What I know is, QA model tries to predict 2 tokens (starting and ending index). The span between starting and ending index represents the answer.
For each token, we understand that the confidence score is represented by the max probability in the output vector. But how to estimate confidence scores when there are 2 output vectors (one for starting token and the other one for ending token) ?
1 Like
That score is obtained by multiplying the probabilities for start and end tokens: taking the softmax of start_logits to get probabilities, the softmax of end_logits then the value corresponding to the predicted start index end index respectively, and finally multiplying those two numbers.
1 Like
Okay, got it. Thanks a lot @sgugger .
Following up on this, @sgugger do you have any suggestion on how to extract confidence score from OpenGVLab/InternVL2_5-8B-MPO? I need to have a confidence on my answers for a Invoice QA project.
1 Like