So far, I’ve been using pre-trained models. For my task, it seems that I am required to perform pre-training on GLUE task just to see how it performs. I wanted to confirm what modifications need to be done to do this ? I’m not sure about using the same tokenizer. I want to randomly initialize it an…

Tips for PreTraining BERT from scratch

prajjwal1 September 24, 2020, 2:31pm 17

I’ve opened a new issue which is about pre-training. Training on GLUE part is resolved. Thanks for asking.

Topic		Replies	Views
Pre-training a BERT model from scratch with custom tokenizer Intermediate	5	2182	January 11, 2022
Differences between Config.from_pretrained and Model.from_pretrained 🤗Transformers	1	789	July 20, 2021
Weights not downloading Beginners	3	1502	May 24, 2021
Metrics mismatch between BertForSequenceClassification Class and my custom Bert Classification Beginners	3	761	December 10, 2020
How do i take only "BERT" weights from BertForSequenceClassification model? 🤗Transformers	0	1261	February 16, 2022