GPT2: many bad_words_ids leading to slow text generation?

the-pale-king · September 4, 2021, 5:55pm

For many models, like GPT2, the generate function accepts bad_words_ids. We’re currently passing about 2500 tokenized phrases into this, and finding that it works well, but also finding that it slows down inference considerably… with 2500 phrases, we find that a generate that would take 250ms without the bad_words_ids, takes about 500ms.

Maybe there is just no solution for this, and we need to simply curtail our usage of bad_words_ids?

We were also looking at this code:

github.com

huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/generation_logits_process.py#L345

    
      
                      )
                      for hypo_idx in range(num_hypos)
                  ]
          
          
        for i, banned_tokens in enumerate(banned_batch_tokens):
                      scores[i, banned_tokens] = -float("inf")
          
          
        return scores
          
          

          
class NoBadWordsLogitsProcessor(LogitsProcessor):
              """
              :class:`transformers.LogitsProcessor` that enforces that specified sequences will never be sampled.
          
          
    Args:
                  bad_words_ids (:obj:`List[List[int]]`):
                      List of list of token ids that are not allowed to be generated. In order to get the tokens of the words
                      that should not appear in the generated text, use :obj:`tokenizer(bad_word,
                      add_prefix_space=True).input_ids`.
                  eos_token_id (:obj:`int`):
                      The id of the `end-of-sequence` token.

A the bad_words_ids are required to be passed in as a list. If somehow we could use a tensor instead, and put that tensor on the GPU, would that speed it up?

Any other suggestions?

thanks

Topic		Replies	Views
Exclude words from GPT-2 generate( ) 🤗Transformers	3	1470	April 26, 2023
Good word list in generate function 🤗Transformers	1	443	March 23, 2023
Add custom constraint in generate() 🤗Transformers	0	286	September 6, 2023
Implementing bad_words_ids does not work with OPT models! Beginners	0	382	July 25, 2022
Best practices for improving text generation speed? Beginners	0	1755	April 30, 2022

GPT2: many bad_words_ids leading to slow text generation?

Related Topics