Hi I was doing intent classification model using bert-base-uncased and I am making my own dataset. I have around 10 intents but none of them are fallback intents as the questions will be too much to add. I trained and researched and saw accuracy score and I need to keep that score too. however, I want to have a fallback intent but I’m not sure how I should do it to be efficient. I was thinking of using confidence score but I can’t seem to find a way to calculate the score. the only confidence score I researched up was for question answering and that is not what I need. Is there a confidence score for load_metrics? or do I have to manually calculate it? If so, how?
This is what I added in my trainer
