Default mBERT tokenizes a sentence as ['[CLS]', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']. I want to change this behaviour and add a language specific token after the CLS token like this: ['[CLS]', '__en__', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']
I know TemplateProcessing can be used to achieve this if the language token doesn’t change
from tokenizers.processors import TemplateProcessing
tokenizer._tokenizer.post_processor = TemplateProcessing(
single=f"{_lang_token} $A [SEP]",
pair=f"{_lang_token} $A [SEP] $B:1 [SEP]:1",
special_tokens=[("[SEP]", tokenizer.convert_tokens_to_ids("[SEP]")),
(_lang_token, tokenizer.convert_tokens_to_ids(_lang_token))],
)
But in my case, the language token changes with every batch. What is the best way to add these tokens? Creating TemplateProcessing objects every time seems inefficient.