[Announcement] GenerationOutputs: Scores, Attentions and Hidden States now available as outputs to generate

This PR: https://github.com/huggingface/transformers/pull/9150 added GenerationOutputs similar to ModelOutpus for Pytorch Transformers.

=> Check out the corresponding tweet with an example: https://twitter.com/SimonBrandeis/status/1346858472000937984 .

The community has been asking for quite some time for this feature, see

When setting return_dict_in_generate the PyTorch .generate() method now returns GenerationOutputs that are very similar in style to ModelOutputs. The usual generation output_ids can then be accessed under outputs["sequences"].

For all “non-beam search” generations (greedy_search and sample), one has now access to

  • all attentions of every layer at every generation step (be careful memory might blow up here) if output_attentios=True
  • all hidden_states of every layer at every generation step if output_hidden_states=True.
  • scores. Now the scores correspond to the processed logits -> which means the models lm head output after applying all processing functions (like top_p or top_k or repetition_penalty) at every generation step in addition if output_scores=True.

For all “beam_search” methods:

  • all attentions and all hidden_states of every layer at every generation step if output_attentions and output_hidden_states are set to True
  • scores now correspond to all processed lm head logits + the current beam_scores for each output token. This score (next_token_scores + beam_scores) is the most important score at every generation step so we decided to output this score.
  • sequence_scores In addition to the three outputs above for beam search output_scores=True also returns the final “beam score” for each returned sequence (see https://twitter.com/SimonBrandeis/status/1346858472000937984)

This should make it easier to analyze the generation of transformer models and should also allow the user to build “confidence” graphs from the scores and sequence_scores.

For more in-detail information please check out the docs: https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputs

We would be very happy for some feedback from the community if the GenerationOutputs meets the expectations :hugs:.

8 Likes

This might be interesting as well: