I’m working on a project that is supposed to use abstractive and/or extractive summarization methods to summarize a given set of reports. I’ve decided to start with PRIMERA as it seemed to fit my intention (summarize multiple documents at once; MDS), however I’m having issues with implementing and making the model work properly.
I’m working within a Google Colab notebook as this was one of the requirements set to me in the beginning of the project.
Here’s what I’ve tried so far:
- Using
pipelinefrom the HugginFacetransformerslibrary, based on PRIMERA’s HF page and HF’s Pipeline tutorial. After reading the input texts, I’m concatenating them together, removing any newlines, tabs or extra spaces and then passing into the pipeline. The result doesn’t seem satisfactory as the summary is very short and doesn’t even contain information from other documents (seems to focus on only one document). I’ve tried both ‘text2text-generation’ and ‘summarization’ parameters within pipeline.
Here’s the code I’m using for the pipeline implementation:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline(task="text2text-generation", model="allenai/PRIMERA-multinews")
# Combine selected texts in a long string
concatenated_report = ' '.join(chosen_texts)
# Remove newline or extra spaces
concatenated_report = ' '.join(concatenated_report.split())
print(concatenated_report)
# Use model
result = pipe(concatenated_report)
print(result)
I’m not sure what I’m doing wrong as I’m getting a very small summary that doesn’t even properly summarizes one document. Here’s an example of an output:
A senior Russian draft officer and former submarine commander accused by Ukraine of deadly strikes on its territory has been shot dead while jogging in the southern Russian city of Krasnodar. Stanislav Rzhitsky, 42, was killed on Monday by an unidentified gunman during a morning run in a park near the Olimp sports centre, local police said. Russian FSB security services said on Tuesday that a 64- year-old man was arrested on suspicion of carrying out the attack. At the time of his death, Rzhitsky was serving as the deputy head of the Krasnodar c ity administration’s mobilisation. According to the Russian daily newspaper Kommersant, Rzhitsky was previously the commander of the Krasnodar submarine, named after the city, in the Russian navy. The Ukrainian army said in a Telegram post on Tuesday that Rzhitsky was in command of a submarine that carried out a deadly missile attack on the Ukrainian city of Vinnytsi a in July 2022 , killing 23 civilians. Rzhitsky’s father told the Baza, a Russian Telegram channel with links to the security services, that his son left the military in December 2021, before Russia invaded Ukraine. Several close friends of Rzhitsky similar ly told the independent Russian station Dozdh that Rzhitsky left the army before the war started. Baza further reported that the killer could have tracked Rzhitsky’s runs on the Strava app, as he regularly followed the same 4- mile route while running. The Ukrainian defence ministry’s main directorate of intelligence, GUR, did not claim responsibility for Rzhitsky’s death. But in a cryptic message, GUR shared a series of details about the killing, including the time of the attack (6am) and the lack of witnes ses because of rain. “Due to heavy rain, the park was deserted, so there were no witnesses who could provide details or identify the attacker,” the GUR said. In a later statement, the agency’s chief, Kyrylo Budanov, rejected the claim that Ukrainian intelligence was behind the killing. “Statements by some media and politicians that the GUR has something to do with the death of Stanislav Rzhitsky have no basis,” Budanov said. Ukraine typically declines to claim responsibility for attacks on Russia or Russian- annexed Crimea, though Kyiv officials have frequently celebrated such attacks with cryptic or mocking remarks. US intelligence has previously said the Ukrainian government authorised last year’s car bomb attack that killed Darya Dugina , the daughter of the prominent Russian nationalist Alexander Dugin. The Britis h defence secretary, Ben Wallace, has suggested Ukraine needs to put more emphasis on thanking the west for its assistance after Volodymyr Zelenskiy’s complaints on Tuesday that his country had not been issued a firm timetable or set of conditions for joining Nato. “Whether we like it or not, people want to see a bit of gratitude,” Wallace said at a briefing in the margins of the Nato summit in the Lithuanian capital, Vilnius, when asked about the Ukrainian president’s comments that it was “absurd” for Kyiv to be told it would be welcome in the alliance but not given a date or exact conditions. “Sometimes you are asking countries to give up their own stocks [of weapons],” Wallace said. “Sometimes you have to persuade lawmakers on the [Capitol] Hill in America.” The US national security adviser, Jake Sullivan, told a public forum at the summit he believed “the American people do deserve a degree of gratitude from the United States government for their willingness to step up and from the rest of the world as well” . He said: “The United States of America has stepped up to provide an enormous amount of capacity to help ensure that Ukraine’s brave soldiers have the ammunition, air defence, the infantry, fighting vehicles, the mine clearing equipment and so much else to be able to effectively defend against Russia’s onslaught and to take territory back as well.” Zelenskiy, speaking in Vilnius on Wednesday, said it was “understandable that Ukraine cannot join Nato when at war” but it would have been ideal if there had been an invitation for Kyiv to join the alliance. He welcomed the results of the Nato summit, in particular the recognition that Ukraine did not need to follow a membership action plan, as well as the positive news on defence packages announced during the summit. Wallace revealed at the briefing that he had travelled to Ukraine last year to be presented with a shopping list of weapons. “You know, we’re not Amazon,” he said. “I told them that last year, when I drove 11 hours to be given a list.” But he said he understood Zelenskiy was speaking to his own public and that, despite his complaint on Tuesday, the final summit deal was a good one for Ukraine. There was an acceptance that “Ukraine belongs at Nato” and that amounted to an effective invitation for membership once t he conflict had died down. Britain, the US and global allies were due to unveil new security assurances for Ukraine on Wednesday. A declaration by the G7 industrialised countries “will set out how allies will support Ukraine over the coming years to end the war and deter and respond to any future attack”, a British government statement said. In practice, this would come as bilateral agreements with Kyiv on long- term military and financial aid to keep Ukraine’s army and economy running. A White House official said the US would start such talks with Kyiv soon. Russia, which says Nato’s eastward expansion is an existential threat to its own security, swiftly lashed out. Dmitry Medvedev, the deputy secretary of Russia’s powerful security council and an anti -western hawk, said increased military assistance to Ukraine from Nato was bringing a third world war a step closer. Musician Alex Yatsun’s house was shelled by Russian forces, but he has focused the trauma and apocalyptic feeling into atmospheric tracks that help him get ‘out of reality’ When the Russian invasion of Ukraine began in 2022, Alex Yatsun was living just 30km from the Russian border in the northernmost part of Kharkiv. “When I woke up that day I started living in a completely different reality,” he recalls. “There were bombs falling every hour.” - Yatsun’s family evacuated but he soon returned north to more dangerous territory to volunteer at a medical centre. “But my house was hit by shelling,” he recalls. A photograph on his Instagram shows the aftermath: huge chunks of wall blown out, smashed windows, a mangled front door. “That was the moment I decided to move closer to central Ukraine.” Despite the difficult and often harrowing backdrop of the last year, the 24- year-old has managed to produce a new 21- track compilation album as DJ Sacred, entitled Dungeon Rap: the Evolution . It’s a follow -up to 2019’s, Dungeon Rap: the Intr oduction . Dungeon rap is a new hybrid style of hip- hop created by Yatsun. It merges Memphis rap – a style of lo -fi atmospheric southern hip- hop made popular by the likes of Three Six Mafia in the late 80s and early 90s – and dungeon synth, a sub- genre that blossomed in 1990s Scandinavia blending dark ambient and black metal. Yatsun’s compilations feature a variety of his musical aliases, such as DJ Bishop, DJ Armok and Pillbox. While other artists also feature as guests, this approach allowed him a variety of musical personalities to explore while presenting the image of a more engaged and thriving scene around the infant genre. “I came to the conclusion it would be very difficult to develop a whole new genre of music only by using one name,” he says. “I thought I could reinvent the process by having these different approaches. DJ Armok is more reliant on bass and more heavily connected to Memphis rap, whereas Pillbox is more about the ethereal and transcendental – like a sublime melancholic feeling.” The res ult is moody and atmospheric stuff. Synths hum doomily, lo- fi beats crackle and snap, while sampled vocals are often slowed to a woozy drawl, blending in as texture rather than leading the charge. Yatsun became so obsessed with Memphis rap, and its “evil” sound, that he spent months breaking down hundreds of tracks, collecting their beats, vocals, melodies and synth lines. He’s assembled them in mega packs, containing thousands of samples, that can be used as the basis for other people to make their own mut ated twist on the genre. There is also a feeling of inescapable darkness to the music. Elements of dread hover like ominous clouds about to unleas h a storm – but the storm itself never comes. “The whole idea of this music is the end of times,” Yatsun says. “It corresponds to my life and my beliefs that the future is bleak.” Yatsun also cites the late theorist and music critic Mark Fisher, and his writings on hauntology, as being linked to the work. “It relates to the idea of a lost future,” he says. “Since the 2013 uprising, most of my life I’ve been living in a condition without stability or any clear vision of a future. I can’t predict my future be cause I don’t even believe in the idea of one, because right now it’s being cancelled. Through this music I have nostalgia for times I have never experienced. It’s completely out of reality – more like a fantasy.” Yatsun’s approach to creating a new genre out of thin air seemingly worked. There are now dungeon rap artists and labels popping up in other countries like Finland, Germany and the UK. Some of these new artists embracing the genre – like DJ Gravelord, Dominus Soul and Lord Crucifix – feature as gu ests on the latest compilation. Yatsun suggests that the themes of alienation and a sense of being trapped in a futureless existence are resonating. “It’s not something that is just localised in Ukraine,” he says. “It’s something we are all experiencing, especially the younger generation. It’s now much clearer that what we were promised, in terms of neoliberalism and a better life and future, has all started to decay. We are all feeling this.” There’s perhaps also an element of relating to a deeper darkness at the core of the music that is as universally personal as it is political. “If you look at my most popular tracks, most of them were written during very difficult times for me,” Yatsun says. “I often have strong, negative emotions in myself and I’m tryi ng to battle through them by making art. Hence why my music is pretty depressing sometimes.” However, what was pretty much an isolated incident of a teenager experimenting with different sounds to create a new hybrid genre has steadily bloomed into a conne cted network that now spans continents. “Dungeon rap is my brainchild,” he says. “The Introduction release was like looking at my child taking its first steps. For this one, it’s like looking at my child growing up and learning to speak.”
[{'generated_text': '– A senior Russian draft officer and former submarine commander accused by Ukraine of deadly strikes on its territory'}]
- I’ve tried loading the model directly. Here, I combined the code from PRIMERA’s HF page (at the bottom of the page) and the code from their example notebook on GitHub (can’t add more links but it can be easily found if you follow the github repo on the HF page). However, I’ve encountered several issues with this approach:
- Just copying the code from the notebook above doesn’t work. The notebook crashes after running the model for 1-2 min due to a lack of RAM (I’ve got 12GB available here). Similarly, the structure of the dataset used by them for pretraining the model, multinews, is a bit different in structure than what I am using with the documents uploaded (articles are stored in a dictionary, and each element is a string with multiple articles on the same topic concatenated using
"|||||"- I don’t need all of this as I’ll only have one report per topic). Trying to match the same ‘structure’ still results in a crash. - If I change the processing part, I get multiple summaries for some reason (for example, if I upload 3 docs I’ll get 3 summaries while I was expecting just one). Sometimes it can combine info from multiple documents, while majority will just all use info primarily from one doc.
- Additionally, majority of the summaries “take into account” all of the documents. As an example when uploading 3 documents, the model seems to focus its attention on one document, which it uses to build ~70-80% of the summarization and then the remainder adds from another document, leaving the last one unused at all (note, all of them are on different topics but I’m trying to combine them together). I’m not sure if that’s a problem with me processing the document or creating the token matrix which is used by the model - I’ve checked and I think I can see the
<DOC_SEP>, <EOS> and <BOS>tokens.
Here’s the code I used for processing and generating the summaries. I’m sorry for the mess and there might be very evident mistakes here, I’ve already tried changing it multiple times and I don’t really remember how it looked when it worked the best:
# Importing necessary modules
from transformers import (
AutoTokenizer,
LEDForConditionalGeneration,
LEDConfig,
)
from datasets import load_dataset, load_metric
import torch
# Initializing variables
TOKENIZER = AutoTokenizer.from_pretrained('allenai/PRIMERA-multinews')
CONFIG = LEDConfig.from_pretrained('allenai/PRIMERA-multinews')
MODEL = LEDForConditionalGeneration.from_pretrained('allenai/PRIMERA-multinews')
MODEL.gradient_checkpointing_enable()
PAD_TOKEN_ID = TOKENIZER.pad_token_id
DOCSEP_TOKEN_ID = TOKENIZER.convert_tokens_to_ids("<doc-sep>")
input_ids_all = []
for doc in chosen_texts:
# Will strip each document of newlines, extra spaces or tabs.
doc = doc.replace("\n", " ")
doc = " ".join(doc.split()).strip()
# Append to a new list
## concat with global attention on doc-sep
input_ids = []
input_ids.extend(
TOKENIZER.encode(
doc,
truncation=True,
max_length= 4096 // len(chosen_reports),
)[1:-1]
)
input_ids.append(DOCSEP_TOKEN_ID)
input_ids = (
[TOKENIZER.bos_token_id]
+ input_ids
+ [TOKENIZER.eos_token_id]
)
input_ids_all.append(torch.tensor(input_ids))
input_ids = torch.nn.utils.rnn.pad_sequence(
input_ids_all, batch_first=True, padding_value=PAD_TOKEN_ID
)
I’m not exactly sure if what I’m doing here is correct - am I processing the documents wrong and it creates the token matrix wrong, leading to a higher weighting of a document over the others? How can I understand if I’m doing anything wrong? I’ve tried printing the token/tensor matrices but I’m not sure I really understand what’s happening in there.
- The last option would remain to use the inference API to use the model. Would that be an option or should I best rely on using the
transformerslibrary? I’ve tried using it by following the instructions on the model’s page, but I’m just getting a response similar to thepipelinesoption (super short and not even complete)
I’d really appreciate if anyone with the experience of using this model (or any other that uses HF’s transformers library) can help me out with this. I’ve never done anything like this before and I’m not sure what I’m doing wrong here (or what I’m doing in general hahah).