Hey guys, sorry about the super late response.
T5 pre-training is not implemented with Transformers, AFAIK it’s only available in the original T5 repo.
What we need to implement this with Transformers is the T5 style denoising dataset. It’s in my todo-list to implement this hopefully early next month.