How to use inputs_embeds in generate()?

I have fine-tuned a T5 model to accept a sequence of custom embeddings as input. That is, I input inputs_embeds instead of input_ids to the model’s forward method.

However, I’m unable to use inputs_embeds with T5ForConditionalGeneration.generate(). It complains that bos_token_id has to be given if not inputting input_ids, but even if I provide a bos_token_id, it still doesn’t run. I considered running the encoder separately, but there is no way I can pass the encoder output to generate() either.

It will be very useful if generate() can accept inputs_embeds or encoder output, so that we can use the decoding strategies provided in the GenerationMixin.


Not an expert, but I just follow the example from this:, and it works for me.

I guess you can use model_kwargs argument for .generate() function.
put encoder_outputs to model_kwargs, where encoder_outputs is encoder output of encoder-decoder model (like T5 and BART). I looked at the code and think this is the way.

Hi, did you solve this problem? I have met the same one but it seems that there is still no solution

Just encountered this problem too… anyone have a solution? thanks!