Hi all,
I am getting to know HuggingFace pipelines and trying to find a model for summarizing reviews. I tried the following models: sshleifer/distilbart-xsum-12-1, t5-base, ainize/bart-base-cnn, gavin124/gpt2-finetuned-cnn-summarization-v2 and google/pegasus-xsum. My experience is below:
sshleifer/distilbart-xsum-12-1
summarizer = pipeline("summarization", model='sshleifer/distilbart-xsum-12-1') # 443M
summarizer("An apple a day, keeps the doctor away", min_length=5, max_length=20)
[{'summary_text': ' A doctor in the UK has been diagnosed with cancer.'}]
summarizer("""
Your max_length is set to 20, but your input_length is only 11.
Since this is a summarization task, where outputs shorter than
the input are typically wanted, you might consider decreasing max_length
manually, e.g. summarizer('...', max_length=5)""")
[{'summary_text': ' If you want to know what it is a good idea for you.'}]
t5-base
summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf") # 892M
summarizer("An apple a day, keeps the doctor away", min_length=5, max_length=20)
[{'summary_text': 'an apple a day, keeps the doctor away from the doctor .'}]
summarizer("""summarize: This was a really good movie, although the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! So it was not bad to watch.""", min_length=5, max_length=15)
[{'summary_text': 'the second part of this movie was quite boring . the movie was'}]
ainize/bart-base-cnn
summarizer = pipeline("summarization", model="ainize/bart-base-cnn") # 558M
summarizer("""This was a really good movie, although the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! So it was not bad to watch.""", min_length=5, max_length=15)
[{'summary_text': '"This was a really good movie, although the second part'}]
gavin124/gpt2-finetuned-cnn-summarization-v2
summarizer = pipeline("summarization", model="gavin124/gpt2-finetuned-cnn-summarization-v2", framework="pt")
summarizer("This was a really good movie, although the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! So it was not bad to watch.")
[{'summary_text': 'This was a really good movie, although the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! So it was not bad to watch. I'}]
google/pegasus-xsum
summarizer = pipeline("summarization", model="google/pegasus-xsum", framework="pt") # 2.28G
summarizer("This was a pretty bad movie: the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! ")
[{'summary_text': 'This is the second part of a two-part review of the horror film The Conjuring.'}]
GPT-3.5
summarize: An apple a day, keeps the doctor away.
'The saying "an apple a day, keeps the doctor away" emphasizes the idea that consuming an apple every day can help maintain good health and prevent the need for medical attention.'
summarize in 15 tokens: """This was a really good movie, although the second part was quite boring, me and my wife almost fall asleep. But the beginning was crazy! So it was not bad to watch."""
'Good movie, boring part almost slept, crazy beginning, not bad.'
These make the impression that the open source models cannot really summarize. Is that the state of the art or I am missing something? Should I do something, like extra train the (pretrained) model with some other corpus to be able to summarize?
Thank you,