I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the simpletransformers package, but it has memory issues at the moment and after a few epochs my colab session crashes.

How pretrain ELECTRA on custom dataset?

valhalla September 6, 2020, 10:53am 6

1 Like

Topic		Replies	Views
ELECTRA training reimplementation and discussion Research	14	6324	September 17, 2023
PreTrain ELECTRA from scratch in Portuguese Flax/JAX Projects	2	1271	November 8, 2021
Anyone have idea how we can finetune a model using Trainer API? 🤗Transformers	0	345	April 22, 2022
Dataset for training BlenderBot 🤗Transformers	1	2315	May 1, 2021
PreTrain Electra/T5 for Korean from scratch Flax/JAX Projects	3	638	July 2, 2021