Hi All, Just want to know, is there any way to batch decode variable length sentences. For example [S1, S2] , where S1 has 5 words abd S2 has 10 words . Can we decode it using GPT2 , BERT etc?

Variable length batch decoding

valhalla October 5, 2020, 12:28pm 10

Not a TF user

Great, can you share your fix if possible, lots of other people are interested in batched prediction for GPT.

1 Like

Topic		Replies	Views
Decoder generate with prompts of variable lengths? 🤗Transformers	0	664	May 25, 2022
Issue with Decoding in HuggingFace 🤗Tokenizers	2	3878	March 24, 2022
Batch generation with GPT2 🤗Transformers	12	17189	January 16, 2024
Parallelize Mistral/ llama2 output 🤗Transformers	1	154	May 25, 2024
Llama2 pad token for batched inference Models	7	15664	March 31, 2024