I am using a summarization pipeline to generate summaries using a fine-tuned model. The
summarizer object is initialised as follows:
summarizer = pipeline( "summarization", model=model, tokenizer=tokenizer, num_beams=5, do_sample=True, no_repeat_ngram_size=3, max_length=1024, device=0, batch_size=8 )
According to the documentation, setting
num_beams=5 means that the top 5 choices are retained when a new token in the sequence is generated based on a language model, and the model moves forward discarding all other possibilities, so that 5 options are carried over all the time. However, this option seems to apparently be incompatible with
do_sample=True where it seems that new tokens are picked based on some random strategy (which doesn’t have to be uniformly random of course, but I don’t know the details of this process). Could anyone explain clearly how
do_sample=True would work together (no error is raised so I assume this is a valid configuration).