Example of how to pretrain T5?

[lewtun]

Hi @mralexis, there’s a GitHub issue that might help you: How do I pre-train the T5 model in HuggingFace library using my own text corpus? · Issue #5079 · huggingface/transformers · GitHub

In particular T5ForConditionalGeneration is probably what you are looking for doing pretraining: T5 — transformers 4.3.0 documentation