Negative test in Text classification

I am using one of the pre-trained model from Huggingface for Topic classification. I have 3 classes after training is completed, when I have done positive test its working fine. However, if i am trying to input some random sentence model is still giving highest probability for one of the three classes( but input sentence is not at all relevant). I understand this the nature of softmax function. Is there way to solve the issue?. As far as I research Training random sentences with other class is the solution but is there any other way?