Selective masking in Language modeling

abdallah197 · August 13, 2020, 12:06am

Hi Huggingfacers

I have a number of questions regarding finetuning a language model:

How to mask a selective portion of a given input sentence instead of masking randomly.
For example, if I am using ALBERT as a model, and I am aiming to do a different kind of loss function than the standard MLM loss for the masked tokens, how to access the model output of the masked tokens

leoapolonio · August 13, 2020, 2:26pm

Refer to: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=M1oqh0F6W3ad

Mask Code:

github.com

huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L157


    if are_tensors_same_length:
        return torch.stack(examples, dim=0)
    else:
        if self.tokenizer._pad_token is None:
            raise ValueError(
                "You are attempting to pad samples but the tokenizer you are using"
                f" ({self.tokenizer.__class__.__name__}) does not have one."
            )
        return pad_sequence(examples, batch_first=True, padding_value=self.tokenizer.pad_token_id)

def mask_tokens(self, inputs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
    """
    Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.
    """

    if self.tokenizer.mask_token is None:
        raise ValueError(
            "This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer."
        )

    labels = inputs.clone()

Topic		Replies	Views
How to do selective masking in Language modeling 🤗Transformers	3	502	August 13, 2020
Masking specific token in each input sentence during Masked language modelling 🤗Transformers	0	1001	October 18, 2021
How to make a model predict on only some tokens Beginners	1	565	June 16, 2022
Documentation: Transformers Language Modeling Section Beginners	0	320	May 14, 2022
Using a dataset with already masked tokens Beginners	2	672	February 3, 2021

Selective masking in Language modeling

Related Topics