Why is the average length of generated summaries during Hugging Face text summarization training much smaller than the actual average length of the training data?

🤗Transformers
1

Hello,
I’m using Hugging Face’s text generation model to train a model on my own dataset for text summarization, and I’m noticing that the average length of generated summaries during training is much smaller than the actual average length of the training data.

Here are some details about my setup:

  • I’m using Seq2SeqTrainer API in Hugging Face’s Transformers library to train the google/flan-t5-small model for text summarization.
  • The average length of the target output text in my training data is around 270 words, while the average length of the summaries is only around 20 words.
  • The final rouge1 score I get after training for 20 epochs is 25.

In this context, I would like to ask several questions:

  1. Why the model is generating shorter texts of average 20 words while the training data has average 270 words?
  2. How can I force the model to generate longer summaries during training.
  3. I believe that if the model increases the length of generation then it will increase the rouge1 score too. Please share your feedback on this too.

Thank you.

2 Likes
2

Just curious were you able to resolve/understand this? If so how?