How to do selective masking in Language modeling

🤗Transformers
1

Hi Huggingfacers

I have a number of questions regarding finetuning a language model:

  1. How to mask a selective portion of a given input sentence instead of masking randomly.
  2. For example, if I am using ALBERT as a model, and I am aiming to do a different kind of loss function than the standard MLM loss for the masked tokens, how to access the model output of the masked tokens
2

I think the answer is similar another post: Selective masking in Language modeling

3

Closing this, since you opened another thread for this and received an answer.