I have the following code to run text summarisation:
First I create the pipeline
summariser = pipeline("summarization", device=0, model="sshleifer/distilbart-cnn-12-6", revision="a4f8f3e", max_length=max_tokens, batch_size=batch_size)
And then I run it (here I break after one cycle to check the size of the output)
summaries =  for out in tqdm(summariser(KeyDataset(ds_to_shorten, "description"), batch_size=batch_size)): summaries.extend(out) break
len(out) is always 1, meaning that it is not returning batches of data, am I right?
What am I doing wrong?