Bitext Alignment (Translation Source and Target Alignment)

Models
1

Hello!

Looking for some guidance on handling Bitext Alignment. Similar to how Microsoft handles it with their Translation Service. For reference I’m using the Helsinki pre-trained models and have come across papers say alignment can be derived from the hidden states or the decoder attentions. I am returning them but can’t make any sense of the returned tensors. Looking for some documentation or examples on how to make sense of the decoder_attentions.

generated = translation_model.generate(return_dict_in_generate = True, **prepare
translation_model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-es', return_dict=True, output_attentions=True, output_scores=True, output_hidden_states=True)
2

If you want to extract a sentence representation (for alignment) out of the encoder output, you should consider using a pooling layer to obtain the sentence vector: sentence-transformers/Pooling.py at ec76488000f94efdba911356b8924cc46db0c2ee · UKPLab/sentence-transformers · GitHub you can refer to the pooling class by sentence-transformers.

1 Like
3

Thank you @morenolq for taking the time to respond! I’ll check this out. :bowing_man:t4: