How to get the logits for the T5 model when using the `generate` method for inference?

Beginners
1

I’m currently using HuggingFace’s T5 implementation for text generation purposes. More specifically, I’m using the T5ForConditionalGeneration to solve a text classification problem as generation.

The model’s performance is overall very satisfactory after training, but what I am wondering is how I can get the logits for generation?

I’m currently performing inference as is suggested in the documentation via model.generate(**tokenizer_outputs), but this simply outputs the IDs themselves without anything else.

The reason why I want the logits is because I want to measure the model’s confidence of generation. I’m not 100% certain if my approach is correct, but I’m thinking that if I can get the logit values of each generated token and average them, I could get the overall confidence score of the generated sequence.

Would anybody know how I could do this? Thanks.

2

You can use the arguments output_scores=True, return_dict_in_generate=True in generate

3

Hmm neither option seems to work. Is outputs = model.generate(tokens['input_ids'], output_scores=True) the correct syntax?

Apparently you have to set both arguments to True. I was only setting one. My mistake misreading the answer.

4

you can use compute_transition_scores to get scores of each token
find more info here Generation