TAPAS question answering for missing informations?

Hello! I have experimented a bit with the pre-trained TAPAS models available, and they actually work quite well. I tried asking questions that don’t have an answer in the table and still get an output, as expected.
My idea was to filter questions without answers by looking at the probabilities in the output of the model. Unfortunately, even when there isn’t any answer in the table, the model outputs extremely confident answers (probability > 0.9).
How can I treat questions without answers? I don’t have a lot of computer power so fine-tuning will be very time-consuming and hard for me. Is there an alternative way?