I have fine-tuned a T5 model to accept a sequence of custom embeddings as input. That is, I input inputs_embeds instead of input_ids to the model’s forward method.
However, I’m unable to use inputs_embeds with T5ForConditionalGeneration.generate(). It complains that bos_token_id has to be given if not inputting input_ids, but even if I provide a bos_token_id, it still doesn’t run. I considered running the encoder separately, but there is no way I can pass the encoder output to generate() either.
It will be very useful if generate() can accept inputs_embeds or encoder output, so that we can use the decoding strategies provided in the GenerationMixin.