I want to fine-tune Roberta on MLM task on my own data. However, for each word, I also have an additional vector with 10 elements. So whenever I predict a masked token, I want the loss to be: aMLM loss + bvector_prediction_loss. How can I do it? didnt find any example or tutorial?

Fine-tune MLM in Roberta custom loss (additional component)

nielsr January 21, 2024, 12:12pm 3

Topic		Replies	Views
How loss is calculated in MLM training 🤗Transformers	0	765	April 1, 2022
How can I check mlm accuracy during training RoBERTa? Beginners	7	2428	August 30, 2021
Label smoothing and compute_metrics in Trainer Beginners	1	788	September 7, 2021
Pre-trained from scratch RoBERTa is not fine-tuned. (using pytorch and DDP) Beginners	0	234	November 2, 2023
Accuracy of Masked LM training Beginners	0	722	June 15, 2022