[HELP] Special tokens not appearing as predicted tokens!

Beginners
1

Hello,

I require some urgent help!

I trained a masked language model on a Twitter dataset, with each tweet containing one emoji. Then, I used the following code to add the emojis as special tokens:

num_added_toks = tokenizer.add_tokens(['😃',
'😄',
'😁',
'😆',
'😅',
'😂',
'🤣',
'🧔🏿‍♂️'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer

From adding the special tokens, I added 3311 different emojis successfully, which increased the embedding to (53575, 768) as shown below:

We have added 3311 tokens

Embedding(53575, 768)

Now, here’s the issue I am facing… When I add the <mask> token to a sentence and input the top_k as the total number of embeddings, which is 53575, not a single emoji shows up in the predictions.

I used this line of code:

mask_filler("Are you happy today <mask>", top_k=53575)

As you can see in the code above, the top_k is 53575, the total number of embeddings which should include the 3311 emojis I added, right?

However, when I make the predictions and scroll through the list of 53575, not a single emoji is there!

Why is this is happening? Like, I have added the emojis to the vocabulary, but they are simple not there when making predictions.

SEE FULL CODE HERE: MLM-EMOJIS/mlm_emojis.ipynb at main · saucyhambon/MLM-EMOJIS · GitHub

2

I see one emoji in the last predictions. Mostly, it’s that the model is not used to seeing those, so it probably needs to be trained longer.

3

Hello @sgugger. I appreciate the response. How much data would you recommend?

4

So if I use more data to train, will all the emojis show up as predictions? Only asking, because I am doing this as part of my MSc project, and I am having no success.

5

@sgugger Shouldn’t all tokens have at least some output probability? The output predictions are over the full size of the vocabulary after all?

6

@BramVanroy That is what I was thinking! If emojis are in the vocab, which they are, they should be predicted tokens when making predictions right? Like, my vocab size with the emojis is 53575, and when making predictions, there is one 1 predicted emoji in the vocab size of 53575 - which means something is going wrong. What do you think?

7

I don’t understand the question: if there is one prediction that is an emoji, clearly they have output probabilities. It’s just that for the sentence used (“I am feeling good today my friend ”), those probabilities are lower than punctuation symbols, which seems logical to me given the fast BERT has learned to do that over a huge corpus and a long training, a habit it won’t lose in a small fine-tuning of three epochs over 1700 samples.

8

Yes, but even changing the sentence, I still get that same predicted emoji.

9

Update: So…I trained my model on 70,000 records, and I still only get 1 emoji. Something is definitely not right.

10

If OP adds 3311 emojis to the vocabulary, then you’d expect that the probability vector for the mask token in Are you happy today <mask> includes probabilities for all the original tokens of the model as well as probabilities for all the 3311 emojis.

OP seems to suggest that the output probabilities for <mask> do not include probabilities for those added emojis, i.e. the output probabilities do not have the same size as the vocabulary size. That is at least what I understand from OP’s post.

1 Like
11

When how did the pipeline predict an emoji if it’s not in the vocab?

There is no shape computation anywhere in the notebook, just a pipeline use, so there is no way to see if the model outputted probabilities for each emoji for sure (I’m pretty sure it did, they are just low).

12

I would think the same, but OP wrote that they tried

mask_filler("Are you happy today <mask>", top_k=53575)

where 53575 is the resized embedding size, and that in that output they can only find one emoji.

1 Like
13

That is correct indeed. After adding the emojis to the vocab and resizing it, I’d expect all 3311 emojis to appear when viewing the top_k=53575. Hence, that 53575 contains all the emojis as far as I am aware; therefore, they should be predictions.

14

Not sure that’s a use case the pipeline properly supports, as it’s a very large value. In any case to debug further, you should dig into the predictions manually and go through the model and its output.

15

What do you mean by ‘dig into the predictions manually?’ I have already gone through all 53575 predicted tokens when making predictions, and there was only 1 emoji as you know.