Produce similar embeddings to another model with BERT

luka · December 15, 2021, 7:58am

I have a dataset in the form (input_text, embedding_of_input_text), where embedding_of_input_text is an embedding of dimension 512 produced by another model (DistilBERT) when given as input input_text.

I would like to fine-tune BERT on this dataset such that it learns to produce similar embeddings (i.e. a kind of mimicking).

Furthermore, by default BERT returns embeddings of dimension 768, while here embedding_of_input_text are embeddings of dimension 512.

Which is the correct way to to that within the HuggingFace library?

Topic		Replies	Views
PyTorch Bilinear messing with HuggingFace BERT?! Beginners	0	626	February 22, 2022
Accessing uncontextualized BERT word embeddings Beginners	2	1501	October 30, 2020
How to obtain [CLS] embeddings from fine-tuned BERT model (using Transformers Trainer) Beginners	1	2632	June 27, 2022
Saving Manually Resized Embeddings for a Pretrained Bert Model (I believe I am asking this correctly) Beginners	0	84	November 7, 2024
BERT modified embeddings 🤗Transformers	0	392	January 8, 2022

Produce similar embeddings to another model with BERT

Related topics