How to do batch generation with the GPT2 model?

Batch generation with GPT2

sgugger July 7, 2022, 5:48pm 8

It looks like you are looking for max_new_tokens?

2 Likes

Topic		Replies	Views
What is the correct format of input when fine-tuning GPT2 for text generation with batch input? Models	0	509	January 22, 2024
Different model.generate() predictions between batched and unbatched/padded token inputs 🤗Transformers	2	2265	August 26, 2023
The effect of padding_side 🤗Transformers	14	15806	August 7, 2025
Variable length batch decoding 🤗Transformers	11	3957	March 31, 2024
Padding to the left of the inputs, GPT2LMHeadModel gives different answer Intermediate	2	1292	February 21, 2023