Binary Classification Given Large Number of Tags

I am currently running Zero Shot Learning using pipeline("zero-shot-classification") with a large number of candidate tags. If enough tags have a high enough scores, that sequence sentence would be classified as matching. Otherwise, it would be classified as non matching.

I am currently taking the mean of the scores of all candidate tags, and if they exceed a certain threshold, it would be classified as matching. However, this is not an accurate approach.

Is there are more accurate “mathematical formula” that yields better results?

Thank you,

Stephen Wong