Rewriting generate function for manual decoder input

🤗Transformers
2

Hey @marcoabrate,

The current version of generate (and also the one of v4.1.1.) already includes the possibility to provide user specific decoder_input_ids. You just have to add it to generate().

E.g. the following code works as expected

from transformers import T5ForConditionalGeneration, T5TokenizerFast

model = T5ForConditionalGeneration.from_pretrained("t5-small")
tokenizer = T5TokenizerFast.from_pretrained("t5-small")

input_ids = tokenizer("translate English to German: How are you?", return_tensors="pt").input_ids
decoder_input_ids = tokenizer("<pad> Wie geht", return_tensors="pt", add_special_tokens=False).input_ids

output = model.generate(input_ids, decoder_input_ids=decoder_input_ids, num_beams=4, num_return_sequences=4)

print("With decoder_input_ids num_beams=4", tokenizer.batch_decode(output, skip_special_tokens=True))

output = model.generate(input_ids, num_beams=4, num_return_sequences=4)

print("Without decoder_input_ids num_beams=4", tokenizer.batch_decode(output, skip_special_tokens=True))

Also see this notebook for the answers of this specific use case: https://colab.research.google.com/drive/11js9We6ZtjN15hb3-PoFZBXJrcSOo_Qa?usp=sharing

This feature was enabled by the generate refactor: Big `generate()` refactor

Does this correspond to what you were trying to achieve or are you looking for some other behavior?

1 Like