I have a dataset in the form
(input_text, embedding_of_input_text), where
embedding_of_input_text is an embedding of dimension 512 produced by another model (DistilBERT) when given as input
I would like to fine-tune BERT on this dataset such that it learns to produce similar embeddings (i.e. a kind of mimicking).
Furthermore, by default BERT returns embeddings of dimension 768, while here
embedding_of_input_text are embeddings of dimension 512.
Which is the correct way to to that within the HuggingFace library?