Hi, @sgugger - thanks for that. Would it be possible for you to take a look at this issue I am having? Here is the link: [HELP] How to include emojis in masked language modelling?
I understand that I need to add custom tokens, which are emojis in my case, but I do not know how to do it. I believe that I need to use this code somewhere:
# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
num_added_toks = tokenizer.add_tokens(['🤗'])
print('We have added', num_added_toks, 'tokens')
# Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
model.resize_token_embeddings(len(tokenizer))