Length_penalty not influencing results (Bart, Pegasus)

🤗Transformers
1

Hello,

I am experimenting with the generative parameters of the two models Bart and Pegasus. In particular, I am having trouble with the length_penalty parameter, since changing it does not change the output of the model.

I am summarizing two different chapters of a book (# tokens around 1k) and this is the code I am using:

model.generate(
    b0ch1sec1_text_enc,
    min_length = 150,
    max_length = 350,
    num_beams = 2,
    length_penalty = lp,
    early_stopping = True)[0]

With lp varying from 0.1 to 2 and model being either bart-large-cnn or pegasus-large.

Do you have any idea why the output does not change at all?

3 Likes
2

For those who were following this post, I tried in a more rigorous way with some (around 10) articles from the CNN/DM and the length_penalty parameter does change the output (actually a lot, from 200 to 500 tokens). However, it is still a mystery for me why sometimes it does not influence it at all.

The code I have used:

import torch
torch.manual_seed(42)

summ = {}
model.to('cuda')
for i, a in enumerate(articles):
    summ[i] = []
    for lp in [0.1, 1, 2]:
        summ[i].append(
            tokenizer.decode(
                model.generate(
                    tokenizer.encode(a, truncation=True, return_tensors='pt').to('cuda'),
                    length_penalty=lp)[0],
                skip_special_tokens=True))