How much data is required to pretrain a model

antoine2323231 October 17, 2022, 1:57pm 1

Hello, I am looking to find an estimate of the amount of data generally required to pre-train a transformers model? Is there models which are less data intensive?

Topic		Replies	Views
Format of data during pre-training 🤗Datasets	1	274	October 7, 2020
Training HF transformer models on custom (not text) data 🤗Transformers	0	125	May 26, 2023
Transformers with protein data Beginners	0	279	July 6, 2022
An example on training the decision transformer from scratch Models	2	608	September 14, 2022
Transformers, limiting output to 200 words 🤗Transformers	0	207	August 23, 2022

How much data is required to pretrain a model

Related Topics