Sentence similarity models not capturing opposite sentences

I have tried different models for sentence similarity, namely:

  • distilbert-base-uncased
  • bert-base-uncased
  • sentence-transformers/all-mpnet-base-v2

I used them together with the packages sentence-similarity and sentence-transformers, which simplify the programming.

I have also tried Universal Sentence Encoders (en_use_md and en_use_cmlm_lg).

However, while these models generally correctly detect similarity for equivalent sentences, they all fail when inputting negated sentences. E.g., these opposite sentences:

  • “I like rainy days because they make me feel relaxed.”
  • “I don’t like rainy days because they don’t make me feel relaxed.”

return a similarity of 0.993 with the model distilbert-base-uncased.

However, sentences that could be considered very similar:

  • “I like rainy days because they make me feel relaxed.”
  • “I enjoy rainy days because they make me feel calm.”

return a similarity of 0.996, which is barely higher.

To my understanding, opposite sentences should have a small similarity, especially when semantics are taken into account.

My question is: Are there any models/approaches that are able to capture the affirmative/negative nature of sentences when calculating similarity?

This relates back to a discussion that was had on Twitter not too long ago. The problem is that “similarity” is ill-defined. You can read through the thread. I did not add much but the discussion between Nils Reimers and Yoav Goldberg is interesting.

It is a good mind exercise to think outside of what you’d want it to mean, and what the models are actually paying attention to. In your example, it is likely that content words receive the most attention and are responsible for a lot of the “meaning” (representation vector). On top of that, lexical overlap inevitably contributes to this value as well. That means that “(rainy) days” and “because they make me feel” already overlap. Yes, in terms of semantics and the whole context the meaning is different, but for the model these sentences are very “similar”. These models do not (necessarily) do sentiment analysis and comparison which seems to be what you are after.

You may wish to look for sentiment models instead.

Thank you for your answer. The Twitter thread was very interesting.

Actually, I’m not really after sentiment analysis; the models also fail with factual sentences with opposite meanings. What I would like to achieve is inferring when two sentences are opposite and assign them a low similarity score (again, this is within the discussion of what “similarity” means, since someone could argue that two sentences that only differ in a “not” or a “don’t” are still very similar).