seq2seq decoding is inherently slow and using onnx is one obvious solution to speed it up. The onnxt5 package already provides one way to use onnx for t5. But if we export the complete T5 model to onnx, then we can’t use the past_key_values for decoding since for the first decoding step past_key_…

Speeding up T5 inference 🚀

kira March 15, 2021, 9:29am 14

thank you! @valhalla. created a new thread here.

Topic		Replies	Views
Boost inference speed of T5 models up to 5X & reduce the model size by 3X 🤗Transformers	2	5598	June 8, 2023
How to convert mT5 and ByT5 to ONNX format? 🤗Transformers	4	2055	December 22, 2021
Improving decoding speed by onnx conversion model Beginners	0	240	November 17, 2021
When exporting seq2seq models with ONNX, why do we need both decoder_with_past_model.onnx and decoder_model.onnx? 🤗Optimum	12	4543	March 7, 2024
Transformers / T5 , jit trace, script, quantize 🤗Transformers	2	571	April 18, 2023