Confidence Score in Setfit fine-tuned model

🤗Transformers
1

Hi Community,

I have successfully trained a Setfit Model for multi-label text classification. However, at inference the prediction output doesn’t give confidence score along with the predicted class.

Could you please guide me on how I can get confidence score along with the predicted class label.

Thanks
Abhijit

1 Like
2

Hi Abhijit,
I am not sure if I am answering your question but if you instantiated a HF trainer class object as say trainer and your model was called model then maybe something like this?

trainer.model.predict(test_dataset['text']) #gives predicted label with highest prob
trainer.model.predict_proba(test_dataset['text']) #gives all probabilities- for each label
np.argmax(trainer.model.predict_proba(test_dataset['text'])) #gives the label with the highest prob

That may not be correct nor answering your question though.

4 Likes
3

Thanks @AmandaHattaway for reply. YESSS!!! it does help. “predict_proba” is my guy. I need to write some more code to get my desired output but this will work. Thank you again.

1 Like
4

Hi @MathyIIT , I’m still not able to get the predicted probability, could you pls share your piece of code here ?

5

Hi @MathyIIT
can you share your approach, it is the same I am struggling with now