Hi, I have added custom tokens using this code: # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased') num_added_toks = tokenizer.add_tokens(['😎', '🤬'])…

Adding New Tokens - IndexError: index out of range in self

anon58275033 June 11, 2021, 3:59pm 3

Hello @lewtun, how do I tokensize the corpus after adding the new tokens? Also, this issue in conjunction with this issue I am having: [HELP] How to include emojis in masked language modelling?

Topic		Replies	Views
"IndexError: index out of range in self" in BertForPreTraining Beginners	0	1021	January 31, 2022
[HELP] How to fix IndexError: index out of range in self Beginners	1	1496	March 31, 2023
IndexError: index out of range in self while training a language model from scratch 🤗Transformers	0	252	April 9, 2024
IndexError: index out of range in self on train() Beginners	0	1189	June 19, 2023
"IndexError: index out of range in self" for bert LM example on https://huggingface.co/transformers/quickstart.html Beginners	2	6208	October 29, 2020