Hi all.
I would like to extract the word embeddings for some input tokens (single words, not sentences) from DistilBertForMaskedLM models that I trained with the Hugging Face library. Does the Feature Extraction pipeline compute only contextualized embeddings? Or, when the pipeline receives a single token, it computes a decontextualized embedding? (using the mean of the word’s embeddings from all it contexts).