Hi all. I’m creating a pretrianed T5 model with: T5ForConditionalGeneration.from_pretrained("t5-small") How is this model pretrained? It seems to me that the model weights I get here were trained at least on the GLUE dataset (and probably others). I’d like it to only be pretrained on C4. Are tho…

How is T5 pretrained?

tstandley July 10, 2021, 6:39pm 3

Yes, I’ve read the paper. My question is about the Huggingface implementation.

Topic		Replies	Views
Example of how to pretrain T5? 🤗Transformers	15	16093	March 16, 2023
How to use architecture of T5 without pretrained weights Beginners	2	2404	September 14, 2022
T5 training from scratch Beginners	5	2182	November 5, 2020
Missing pretraining datasets for T5 models 🤗Hub	0	913	January 27, 2022
Using T5 pre-trained weight for Text style transfer Models	1	1618	June 25, 2021