What would be the best strategy to mask only specific words during the LM training? My aim is to mask only words of interest which I have previously collected in a list. The issue arises since the tokenizer, not only splits a single word in multiple tokens, but it also adds special characters if t…

Mask only specific words

RylanSchaeffer November 6, 2021, 7:30pm 4

@Gabrer did you find a solution?

Topic		Replies	Views
Mask modelling on specific words Beginners	1	1064	March 25, 2021
Masking specific token in each input sentence during Masked language modelling 🤗Transformers	0	1064	October 18, 2021
Sequence masking 🤗Transformers	0	390	April 25, 2022
Whole-word masking for T5 Beginners	2	554	November 28, 2023
RobertaTokenizer: How to enable masking of custom special tokens 🤗Transformers	1	991	April 24, 2021