I was previously using trainer.predict() to run my inference but it does not offer that much flexibility and control over generation. For instance, I want to have more than one generation for each sample and it is not possible with trainer.predict(). So I changed to model.generate(). The problem is …

Model.generate() is extremely slow while using beam search

Thang July 22, 2022, 2:19pm 2

Did you any solution? I face the same problem now.

Topic		Replies	Views
Multiple gpu not properly parallelized during model.generate() 🤗Transformers	4	1643	October 9, 2022
How to parallelize model.generate? 🤗Transformers	1	822	September 7, 2022
Beam_search bottlenecks inference with only 1 used cpu 🤗Transformers	1	839	October 13, 2022
Speed up beam search for item generation DeepSpeed	1	955	October 4, 2023
Can trainer.predict() return multiple generations for each sample? 🤗Transformers	2	767	January 18, 2022