Batch generation with GPT2

[lqtrung]

@lqtrung

Hello,

Correct me if I’m wrong but for GPT2, during training mode, when padding_side='left', position_ids should be passed correctly or else the position_ids is created as if right padding is used … making it inconsistent, right ?

I’m referring to this line where the position_ids is created if it is not passed …
transformers/modeling_gpt2.py at main · huggingface/transformers (github.com)

Thank you !

Hi @joaogante , thank you for the response.

I believe that the position_ids is properly prepared during generation as you said because the prepare_inputs_for_generation is called …

But my question is about during training where that function is not called and the gpt2 modeling script does not compute position_ids based on the attention mask (so it is not correct when ‘left’ padding is used …)

So I’m not sure about the recommended practice:

1. Is ‘right’ padding always used during training … and ‘left’ padding is only used during batch generation ?
2. Or the training and generation should have the same padding scheme and in this case the gpt2 modeling script should handle the position_ids better ?