Training BERT for word embedding

[Nlpeva]

Please feel free to share more details. It really depends a lot on the context around the sentences/words you want to embed.
Hopefully these two discussions can help:

Generate raw word embeddings using transformer models like BERT for downstream process

Hi, I am new to using transformer based models. I have a few basic questions, hopefully, someone can shed light, please. I’ve been training GloVe and word2vec on my corpus to generate word embedding, where a unique word has a vector to use in the downstream process. Now, my questions are: Can we generate a similar embedding using the BERT model on the same corpus? Can we have one unique word with its vector? BERT is contextual, not sure how the vector will look like for the same word which i…

Obtaining word-embeddings from Roberta

Hello Everyone, I am fine-tuning a pertained masked LM (distil-roberta) on a custom dataset. Post-training, I would like to use the word embeddings in a downstream task. How does one go about obtaining embeddings for whole-words when the model uses sub-word tokenising. For example, tokeniser.tokenize(‘floral’) will give me [‘fl’, ‘oral’]. So, if ‘floral’ is not even a part of the vocabulary, how do I obtain its embedding? When I do this: tokens = tokenizer.encode("floral") word = tokenizer.e…