Adding New Tokens - IndexError: index out of range in self

anon58275033 · June 11, 2021, 3:59pm

Hello @lewtun, how do I tokensize the corpus after adding the new tokens? Also, this issue in conjunction with this issue I am having: [HELP] How to include emojis in masked language modelling?

Topic		Replies	Views
BERT encoding for batch of Sentence Pairs raise IndexError: index out of range in self Beginners	1	348	November 16, 2022
Fine-tune transformers for language model Beginners	2	547	August 14, 2022
Maybe there is a bug in BertTokenizer? 🤗Transformers	0	315	March 19, 2021
IndexError while training Roberta with a custom tokenizer Beginners	8	860	December 17, 2023
Model.generate() -- IndexError: too many indices for tensor of dimension 2 Beginners	3	4531	November 4, 2021