[Announcement] GenerationOutputs: Scores, Attentions and Hidden States now available as outputs to generate

[patrickvonplaten]

This PR: https://github.com/huggingface/transformers/pull/9150 added GenerationOutputs similar to ModelOutpus for Pytorch Transformers.

=> Check out the corresponding tweet with an example: https://twitter.com/SimonBrandeis/status/1346858472000937984 .

The community has been asking for quite some time for this feature, see

• https://github.com/huggingface/transformers/issues/7654
• https://github.com/huggingface/transformers/issues/8656
• https://github.com/huggingface/transformers/issues/3891

When setting return_dict_in_generate the PyTorch .generate() method now returns GenerationOutputs that are very similar in style to ModelOutputs. The usual generation output_ids can then be accessed under outputs["sequences"].

For all “non-beam search” generations (greedy_search and sample), one has now access to

• all attentions of every layer at every generation step (be careful memory might blow up here) if output_attentios=True
• all hidden_states of every layer at every generation step if output_hidden_states=True.
• scores. Now the scores correspond to the processed logits -> which means the models lm head output after applying all processing functions (like top_p or top_k or repetition_penalty) at every generation step in addition if output_scores=True.

For all “beam_search” methods:

• all attentions and all hidden_states of every layer at every generation step if output_attentions and output_hidden_states are set to True
• scores now correspond to all processed lm head logits + the current beam_scores for each output token. This score (next_token_scores + beam_scores) is the most important score at every generation step so we decided to output this score.
• sequence_scores In addition to the three outputs above for beam search output_scores=True also returns the final “beam score” for each returned sequence (see https://twitter.com/SimonBrandeis/status/1346858472000937984)

This should make it easier to analyze the generation of transformer models and should also allow the user to build “confidence” graphs from the scores and sequence_scores.

For more in-detail information please check out the docs: https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputs

We would be very happy for some feedback from the community if the GenerationOutputs meets the expectations .

[patrickvonplaten]

This might be interesting as well:

Generation Probabilities: How to compute probabilities of output scores for GPT2

Now that it is possible to return the logits generated at each step, one might wonder how to compute the probabilities for each generated sequence accordingly. The following code snippet showcases how to do so for generation with do_sample=True for GPT2: import torch from transformers import AutoModelForCausalLM from transformers import AutoTokenizer gpt2 = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True) tokenizer = AutoTokenizer.from_pretrained("gpt2") input_ids …