Change embedding layer in a model

Languages at Hugging Face
1

I am trying to learn embeddings using a language model (bert-model-uncased), and use the embeddings from this model as an embedding layer in another language model -

this is my base model-

MarianMTModel(
  (model): MarianModel(
    (shared): Embedding(67028, 512, padding_idx=67027)
    (encoder): MarianEncoder(
      (embed_tokens): Embedding(67028, 512, padding_idx=67027)
      (embed_positions): MarianSinusoidalPositionalEmbedding(512, 512)
      (layers): ModuleList(
        (0-5): 6 x MarianEncoderLayer(
          (self_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (self_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (activation_fn): SiLUActivation()
          (fc1): Linear(in_features=512, out_features=2048, bias=True)
          (fc2): Linear(in_features=2048, out_features=512, bias=True)
          (final_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (decoder): MarianDecoder(
      (embed_tokens): Embedding(67028, 512, padding_idx=67027)
      (embed_positions): MarianSinusoidalPositionalEmbedding(512, 512)
      (layers): ModuleList(
        (0-5): 6 x MarianDecoderLayer(
          (self_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (activation_fn): SiLUActivation()
          (self_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (encoder_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (encoder_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (fc1): Linear(in_features=512, out_features=2048, bias=True)
          (fc2): Linear(in_features=2048, out_features=512, bias=True)
          (final_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=512, out_features=67028, bias=False)
)

how do i do that ?

this is what i am dong right now -

        new_embedding_layer = nn.Embedding(
            marian_model.config.vocab_size,
            marian_model.config.hidden_size,
            padding_idx=marian_model.config.pad_token_id
        )
        new_embedding_layer.weight = nn.Parameter(bert_embedding_weights[:marian_model.config.vocab_size])

        # Replace the MarianMTModel's embedding layer with the new embedding layer
        marian_model.model.encoder.embed_tokens = new_embedding_layer
2

Did you ever figure out a better solution to this? I’m looking to do something similar.

3

I am doing something similar, but the model stops generating output offer changing the embedding. Did you face a similar issue?