Pretrain gpt2 example

scienceapptest June 11, 2021, 9:49pm 1

Are there any docs or notebooks that show how to train GPT2 from scratch with a custom text corpus?

Topic		Replies	Views
Train GPT2 on wikitext from scratch Beginners	5	3880	October 25, 2021
Train gpt-2 from scratch in Italian Beginners	0	884	September 8, 2022
Finetune GPT2 in tensorflow on custom data example programmatically Beginners	0	489	July 23, 2020
GPT2 Training from scratch in German 🤗Transformers	3	2322	October 3, 2020
PreTrain GPT2 from scratch in Punjabi Flax/JAX Projects	2	424	June 29, 2021