Use t5-small ft a English to German Bidirectional translation model

[John6666]

I wonder if there are any problems specific to T5…?
Or maybe the learning rate is too low.
Or maybe something is wrong and the model is not actually learning anything. (I saw a case like that a few days ago, but I can’t find it…)
There was also a post that mentioned bleu.

What is loss function for T5

Could you please guide me how loss function for T5 is computed? I mean it is a seq2seq model. Suppose it must map a sequence of X tokens to Y tokens, but it generates Z tokens. How Y and Z are compared to calculate loss?

Pytorch T5 training loss not changing

T5 Model Problems - Constant Loss (doesn't go down)

I’ve been trying to train T5 on a custom dataset similar to Squad v2 by modifying the T5 on TPU colab written by Suraj Patil. (I have Colab Pro so I train on a high-ram TPU instance). The link I have shared is training on Squad v2 directly, which seems to have the same problem. link: Google Colaboratory However, no matter how much I train the error seems to stay constant, i.e. the model does not seem to learn. This seems to be the case both for loss recorded during training, as well as loss du…