GPT2: many bad_words_ids leading to slow text generation?

the-pale-king · September 4, 2021, 5:55pm

For many models, like GPT2, the generate function accepts bad_words_ids. We’re currently passing about 2500 tokenized phrases into this, and finding that it works well, but also finding that it slows down inference considerably… with 2500 phrases, we find that a generate that would take 250ms without the bad_words_ids, takes about 500ms.

Maybe there is just no solution for this, and we need to simply curtail our usage of bad_words_ids?

We were also looking at this code:

github.com

huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/generation_logits_process.py#L345

    
      
                      )
                      for hypo_idx in range(num_hypos)
                  ]
          
          
        for i, banned_tokens in enumerate(banned_batch_tokens):
                      scores[i, banned_tokens] = -float("inf")
          
          
        return scores
          
          

          
class NoBadWordsLogitsProcessor(LogitsProcessor):
              """
              :class:`transformers.LogitsProcessor` that enforces that specified sequences will never be sampled.
          
          
    Args:
                  bad_words_ids (:obj:`List[List[int]]`):
                      List of list of token ids that are not allowed to be generated. In order to get the tokens of the words
                      that should not appear in the generated text, use :obj:`tokenizer(bad_word,
                      add_prefix_space=True).input_ids`.
                  eos_token_id (:obj:`int`):
                      The id of the `end-of-sequence` token.

A the bad_words_ids are required to be passed in as a list. If somehow we could use a tensor instead, and put that tensor on the GPU, would that speed it up?

Any other suggestions?

thanks

Topic		Replies	Views
Exclude words from GPT-2 generate( ) 🤗Transformers	3	1774	April 26, 2023
Good word list in generate function 🤗Transformers	1	629	March 23, 2023
Implementing bad_words_ids does not work with OPT models! Beginners	0	478	July 25, 2022
How to restrict T5 model to generate tokens only from the input text? Intermediate	0	424	June 6, 2023
Prohibit GPT-2 from generating some words on a condition 🤗Transformers	7	1127	April 25, 2021

GPT2: many bad_words_ids leading to slow text generation?

Related topics