T5 Finetuning Tips

[valhalla]

Hey there, you guys might find this interesting, we have just fixed the fp16 issue for some of the T5 models

More details below

T5 fp16 issue is fixed

We have just fixed the T5 fp16 issue for some of the T5 models! (Announcing it here, since lots of users were facing this issue and T5 is one most widely used model in the library) TL;DR: Previously, there was an issue when using T5 models in fp16; it was producing nan loss and logits. Now on the master, this issue is fixed for the following T5 models and versions. Now you should be able to train and use these models for inference in fp16 and see a decent speed-up! T5v1 : t5-small , t5-…