Hi all, I was reading through the encoder decoder transformers and saw how loss was generated. But I’m just wondering how it is internally generated? Is it something like the following: Suppose I have the following pair: ("How are you?", "I am doing great"). In this case, is it calculating the cro…

padding tokens in the labels should be replaced by -100 so the cross_entriopy loss ignores the pad tokens when computing the loss. and the loss is actually computed like this shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous() # prediction_scores is logits labels = labels[:, 1:].…

Encoder Decoder Loss

🤗Transformers

sachin March 15, 2021, 7:22am 2

I posted a longer version of above with reproducible code: deep learning - Sequence to Sequence Loss - Stack Overflow

Topic		Replies	Views
Understanding the encoder-decoder loss calculation VS CLM loss Beginners	0	346	February 21, 2024
Seq2Seq Loss computation in Trainer Beginners	9	6039	October 28, 2021
Does the transformer automatically shift by one position when calculating the autoregressive loss during the forward pass? Beginners	1	28	March 20, 2025
Getting CrossEntropy loss from beam search scores 🤗Transformers	0	402	June 21, 2022
GPT-2 shift logits and labels 🤗Transformers	5	5871	May 12, 2023

Encoder Decoder Loss

Related topics