Get the top k token probabilities of t5 first token

Beginners
1

During inference time, is there a way to extract the top k tokens by probabilities for the first token generated by T5 ?

I have fine-tuned T5 model using the T5ForConditionalGeneration method and now I’m calling a forward call:

output = model(input_ids=source_ids,attention_mask=source_masks,return_dict=True)

But it raises an error:
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

In the T5 training documentation, it says that the forward call " automatically creates the correct decoder_input_ids" which seems that it is not true in my case. How do I handle that forward call?

1 Like