Hi, In the roberta paper, the model is trained by dynamic masking of sentences. If a roberta model is further pre-trained using the run_mlm.py , is the sentences going to be dynamically masked during pre-training using this file or is it statically masked like vanilla BERT? The script is found he…

They are talking about dynamic masking that was mentioned in the RoBERTa paper . The original BERT paper did not use dynamic masking, but rather static masking meaning they masked all sentences before training and used those same masks across multiple epochs. Dynamic masking is better because it mask…

Is the huggingface run_mlm Script dynamically masked?

🤗Transformers

anon81579828 June 1, 2022, 11:46pm 9

Thanks a lot for your reply. Really helped clear things up

Topic		Replies	Views
Does all masking during training take place in data_collator.py? 🤗Transformers	0	118	November 11, 2023
How can I see the masked words during pre-learning by MLM? 🤗Transformers	0	252	February 7, 2022
Masking specific token in each input sentence during Masked language modelling 🤗Transformers	0	1050	October 18, 2021
Creating masked sentences 🤗Datasets	1	411	March 2, 2022
Sequence masking 🤗Transformers	0	382	April 25, 2022

Is the huggingface run_mlm Script dynamically masked?

Related topics