After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning :computer: Code: ele…

ELECTRA training reimplementation and discussion

vblagoje October 16, 2020, 8:52am 9

This is no easy feat, I know it first hand as I am doing something similar with BERT pre-training from scratch. Any reason why you didn’t use HF Trainer?

Topic		Replies	Views
How pretrain ELECTRA on custom dataset? Beginners	5	4151	September 6, 2020
ELECTRA: Accounting for mask tokens that are correctly predicted by MLM 🤗Transformers	9	1283	May 15, 2021
ELECTRA Paper Doubts Research	0	220	September 8, 2023
Retrain Electra model with different embedings from scratch Beginners	0	319	April 1, 2022
Electra-base returns always same output 🤗Transformers	0	226	February 5, 2023

ELECTRA training reimplementation and discussion

Related topics