Hey, I’ve been trying to train Sentence2Vec embeddings and I’ve been wondering what do you think about my approach.
I would be glad to learn about different possible pitfalls in my approach and how to solve them.
- I have a small unique corpus of about 4 million sentences in my test language
- I have a smaller subset (150k) of labeled sentences of whether the sentence is toxic or not.
For this question’s sake assume there isn’t any pretrained model or another existing corpus I can use.
The target is to be able to cluster sentences that have similar meanings together.
- Train a RoBERTa language model
- Fine tune it for classification of toxic or not
- Use a hidden layer of that classifier as an embedding model
- Cluster using embeddings and HDBSCAN.
That’s about it, I tried to keep it as clear as I can.
Thanks for anyone that read up until here!!