Pre-training a BERT model from scratch with custom tokenizer

sgugger · January 11, 2022, 1:04pm

Looks like your evaluation DataLoader does not contain labels as the compute_metrics is never called. If it was you would get an error since you are not taking the argmax of your predictions before sending them to the accuracy.

Topic		Replies	Views
Fine tune Masked Language Model on custom dataset Beginners	5	6082	August 20, 2020
Getting the MLM accuracy for the BERT model I am training from scratch Beginners	7	5385	October 5, 2023
Where in the code does masking of tokens happen when pretraining BERT Beginners	5	7306	August 17, 2020
Calculating accuracy during fine-tuning the BERTForMaskedLM 🤗Transformers	6	2797	October 1, 2020
Tips for PreTraining BERT from scratch 🤗Transformers	19	9867	December 10, 2020

Pre-training a BERT model from scratch with custom tokenizer

Related topics