I have a dataset with sentences. The task that I want to complete is unsupervised, therefore I cannot fine tune the pre-trained model. I need to have a sentence embedding which reflects the semantic meaning of the sentence. Until now I tried BERT, using the CLS as sentence embedding. Unfortunatelly, computing the cosine similarity between different embeddings I discovered that all the different sentences are really similar to each other (cosine similarity always around 0.8). This cannot be possible since the sentences in my dataset have really different semantic meaning. Reading on the internet i found out that there is no general consensus about considering the CLS as sentence embedding. Additionally, some paper shows how the pre-trained BERT (no fine tuning) leads to poor sentence embeddings. Which is the best approach in this case? How should I embed the sentences in my dataset knowing that I cannot fine tune a pre trained LLM? Thank you in advance!
I have a similar issue, the most viable option seems to be getting word embeddings and then pooling (mean etc) on the vectors to generate a proxy sentence vector. Finally this discussion seems most promising
PS Im hoping you might have found an answer by now and can help me out
SentenceTransformers Documentation SentenceTransformers offer an option to get sentence embeddings. You can get embeddings for entire sentences.