Confidence Score in Setfit fine-tuned model

🤗Transformers
1

Hi Community,

I have successfully trained a Setfit Model for multi-label text classification. However, at inference the prediction output doesn’t give confidence score along with the predicted class.

Could you please guide me on how I can get confidence score along with the predicted class label.

Thanks
Abhijit

1 Like
2

Hi Abhijit,
I am not sure if I am answering your question but if you instantiated a HF trainer class object as say trainer and your model was called model then maybe something like this?

trainer.model.predict(test_dataset['text']) #gives predicted label with highest prob
trainer.model.predict_proba(test_dataset['text']) #gives all probabilities- for each label
np.argmax(trainer.model.predict_proba(test_dataset['text'])) #gives the label with the highest prob

That may not be correct nor answering your question though.

4 Likes
3

Thanks @AmandaHattaway for reply. YESSS!!! it does help. “predict_proba” is my guy. I need to write some more code to get my desired output but this will work. Thank you again.

1 Like
4

Hi @MathyIIT , I’m still not able to get the predicted probability, could you pls share your piece of code here ?

5

Hi @MathyIIT
can you share your approach, it is the same I am struggling with now

6

As mentioned, once you get your model out of the trainer, in addition to being able to call model.predict() to get the label with the highest confidence interval, you can also call model.predict_proba() to get back a list of confidence intervals.