Tutorial on Pretraining BERT

Beginners
1

I would like to pretrain a BERT model with custom data. For example, data in my native language. Is there any example script about pretraining a BERT model? I found the tutorial with the EsperBERTo model but this is not a BERT model as far as I understand.

2

You can have a look at the example scripts here: https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling

1 Like