I’m using a VisionEncoderDecoderModel and I want to reuse the encoded image to decode multiple times (say, 30+ times per image). However, I don’t want to rerun the encoder every time I call model.generate(). Is there a way to cache the encoder state and reuse it? Or is there another efficient way to…

Hi! You can pass into “generate” an argument called “encoder_outputs” which will be used by the decoder then, instead of running encoder every time. Optionally you can pass in “decoder_input_ids”, otherwise it will be initialized from BOS token. from transformers import AutoTokenizer, AutoModelForS…

Caching encoder state for multiple encoder-decoder `.generate()` calls?

🤗Transformers

system Closed April 12, 2024, 7:41pm 3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Control EncoderDecoderModel to generate tokens step by step 🤗Transformers	8	2600	June 8, 2022
Why model.generate does encoding multiple times 🤗Transformers	1	564	September 20, 2022
Avoid recalculating hidden states between generate calls? 🤗Transformers	3	1195	March 30, 2023
Cache T5 encoder results within batch when training 🤗Transformers	0	484	March 6, 2021
What is the purpose of 'use_cache' in decoder? 🤗Transformers	5	23698	July 4, 2023

Caching encoder state for multiple encoder-decoder `.generate()` calls?

Related topics