I want to fine-tune Roberta on MLM task on my own data. However, for each word, I also have an additional vector with 10 elements. So whenever I predict a masked token, I want the loss to be: aMLM loss + bvector_prediction_loss. How can I do it? didnt find any example or tutorial?

Fine-tune MLM in Roberta custom loss (additional component)

nielsr January 21, 2024, 12:12pm 3

Topic		Replies	Views
How loss is calculated in MLM training 🤗Transformers	0	827	April 1, 2022
Fine tune Masked Language Model on custom dataset Beginners	5	5855	August 20, 2020
Further pre-training the tokenizer? 🤗Tokenizers	0	801	April 30, 2022
How do I use a fine-tuned Trainer model for inference correctly? 🤗Transformers	0	930	June 9, 2023
RoBERTa MLM fine-tuning Beginners	1	1740	November 24, 2021