Test a Model's knowledge

I’m currently training Semantic Textual Similarity using the SentenceTransformers Library.
Models are BERT and Paraphrase MPNet.

Is there any way to check or even proof if a model understands a specific word or sequence?

I tried masking or calculating the similarity to specific categories related to the word but the results were hardly distinguishable.

Thanks in advance