Generating longer summaries using transformers

🤗Transformers
1

I am using a test article as input and trying to generate a reasonably long summary, but I keep getting very short, single line, summaries from a very long article. I would appreciate any help.

from transformers import AutoTokenizer, \
                         AutoModelForSeq2SeqLM

file = open("Data/article.txt", "r")
article = file.read()

model_name = 'google/pegasus-xsum'

tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer(article, max_length=510, truncation=True, return_tensors='pt').input_ids
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
outputs = model.generate(inputs, max_new_tokens=510, do_sample=False)
summary = tokenizer.decode(outputs[0], min_length=50, max_length=200, skip_special_tokens=True)
print(summary)
2

Apparently models ending in ‘xsum’ are designed for a single-line summary. I will modify the test and report.

Thanks

3

Oh.

I was testing to see if I could find anything. You can add length_penalty=3 to force it to generate longer answer but it starts to keep repeating itself. You can also add repetition_penalty=1.5 to reduce the amount of repetition. But the result isn’t a very good summary.

edit: yea, much better result when I switched to google/pegasus-x-large

4

@wsfung2008 Thank you for looking into this question. Apparently all models ending in ‘xsum’ are designed for a single line summary, so this was my fault for not understanding the model’s purpose.

However I did run into a problem, using models designed for longer Abstractive summaries and posted another issue. Perhaps if you look at that one, you may come up with a solution.

see my post titled “Generating Abstractive summaries”

Thanks

1 Like