Padding to the left of the inputs, GPT2LMHeadModel gives different answer

[wangkuiyi]

When I input the tokenized string Hello, my dog is cute, GPT2LMHeadModel predicts Hello, my dog is cute. I'm not sure if she's a puppy.

If I add four 0s to the left of the token IDs and use the attention mask [0,0,0,0,1,1,1,1,1,1,1], the prediction changes to !!!!Hello, my dog is cute hello hello hello hello hello hello hello hello hello hello.

I’d thought that these two inputs should lead me to the same result?

I have this notebook to reproduce the above experiment Google Colab

[joaogante]

Hi @wangkuiyi In your examples, you forgot to consider the position_ids – regardless of the padding you insert, you want the position_ids that correspond to a given word to be invariant

Further resources:

• Glossary: position IDs
• How GPT2, the model itself, infers position IDs if you don’t pass them (spoiler: it’s based on your input_ids)
• How generate attempts to handle it (spoiler: it uses the attention_mask as a guide)

[wangkuiyi]

Thank you, Joao @joaogante !