I have a dataset with sentences. The task that I want to complete is unsupervised, therefore I cannot fine tune the pre-trained model. I need to have a sentence embedding which reflects the semantic meaning of the sentence. Until now I tried BERT, using the CLS as sentence embedding. Unfortunatelly, computing the cosine similarity between different embeddings I discovered that all the different sentences are really similar to each other (cosine similarity always around 0.8). This cannot be possible since the sentences in my dataset have really different semantic meaning. Reading on the internet i found out that there is no general consensus about considering the CLS as sentence embedding. Additionally, some paper shows how the pre-trained BERT (no fine tuning) leads to poor sentence embeddings. Which is the best approach in this case? How should I embed the sentences in my dataset knowing that I cannot fine tune a pre trained LLM? Thank you in advance!