Bitext Alignment (Translation Source and Target Alignment)


Looking for some guidance on handling Bitext Alignment. Similar to how Microsoft handles it with their Translation Service. For reference I’m using the Helsinki pre-trained models and have come across papers say alignment can be derived from the hidden states or the decoder attentions. I am returning them but can’t make any sense of the returned tensors. Looking for some documentation or examples on how to make sense of the decoder_attentions.

generated = translation_model.generate(return_dict_in_generate = True, **prepare
translation_model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-es', return_dict=True, output_attentions=True, output_scores=True, output_hidden_states=True)

If you want to extract a sentence representation (for alignment) out of the encoder output, you should consider using a pooling layer to obtain the sentence vector: sentence-transformers/ at ec76488000f94efdba911356b8924cc46db0c2ee · UKPLab/sentence-transformers · GitHub you can refer to the pooling class by sentence-transformers.

1 Like

Thank you @morenolq for taking the time to respond! I’ll check this out. :bowing_man:t4: