Tokenizer has call function nad it accepts List[List[string]] so one can tokenize multiple samples at once efficiently. I am looking for a function that reverts this behavior. There are convert_ids_to_tokens() and convert_tokens_to_string() but they only accept List[int] and List[string]. I do not w…

Efficient detokenization method

sgugger January 28, 2021, 10:04pm 4

For multiple sentences there is its counterpart batch_decode.

1 Like