Learning rate for further pretraining BERT on masked language modeling task

Gianluca · September 16, 2021, 8:41pm

I want to further pretrain BERT on my corpus. Is there a standard or typical value for the learning rate that is used when training on the masked language modeling task?

Topic		Replies	Views
Is masking still used when finetuning a BERT model? Beginners	1	1159	July 29, 2020
Which model can use to pre-train a BERT model? Beginners	1	391	December 22, 2021
Fine tunning pretrained bert with new vocabulary Beginners	0	364	October 1, 2020
Pre-training BERT Models	0	249	February 10, 2023
What's the inner mechanism of Masked Language Model in BERT Beginners	0	205	March 31, 2021

Learning rate for further pretraining BERT on masked language modeling task

Related Topics