How to do batch generation with the GPT2 model?

Batch generation with GPT2

deathcrush February 28, 2022, 10:41am 6

@patrickvonplaten @ttj I think this is a good question! Could we discuss on how to do batch inference with past_key_values?

Topic		Replies	Views
What is the correct format of input when fine-tuning GPT2 for text generation with batch input? Models	0	509	January 22, 2024
Different model.generate() predictions between batched and unbatched/padded token inputs 🤗Transformers	2	2265	August 26, 2023
The effect of padding_side 🤗Transformers	14	15806	August 7, 2025
Variable length batch decoding 🤗Transformers	11	3957	March 31, 2024
Padding to the left of the inputs, GPT2LMHeadModel gives different answer Intermediate	2	1292	February 21, 2023