Effect of selecting both `do_sample` and `num_beams` in summarization pipeline

🤗Transformers
1

I am using a summarization pipeline to generate summaries using a fine-tuned model. The summarizer object is initialised as follows:

    summarizer = pipeline(
        "summarization", 
        model=model, 
        tokenizer=tokenizer, 
        num_beams=5, 
        do_sample=True, 
        no_repeat_ngram_size=3,
        max_length=1024,
        device=0,
        batch_size=8
    )

According to the documentation, setting num_beams=5 means that the top 5 choices are retained when a new token in the sequence is generated based on a language model, and the model moves forward discarding all other possibilities, so that 5 options are carried over all the time. However, this option seems to apparently be incompatible with do_sample=True where it seems that new tokens are picked based on some random strategy (which doesn’t have to be uniformly random of course, but I don’t know the details of this process). Could anyone explain clearly how num_beams=5 and do_sample=True would work together (no error is raised so I assume this is a valid configuration).

Many thanks

1 Like