What is the `tie_word_embeddings` option exactly doing?

🤗Transformers
#1

Hi,

for some models there is this tie_word_embeddings parameter. I think it is for the text 2 text models.
Can someone please explain what exactly this parameter is doing?

Many thanks
Philip

Using tie_weights() always? (intend to separate source/target embedding)
Should we use tie_weights always? (intend to separate source/target embedding)
#2

No this is for all models that have a language modeling head (so even masked language models like BERT or causal language models like GPT-2). The idea is that the embedding weights (vocab_size by hidden_size) are tied with the decoder (hidden_size by vocab_size) so the model only learns one representation of the words (that is a big matrix!)

4 Likes