What does the parameter 'clean_up_tokenization_spaces' do in the tokenizer.decode function?

Beginners
1

As titled. The documentation is a bit vague.

Whether or not to clean up the tokenization spaces.

2

It should remove space artifacts inserted while encoding the sequence. E.g., if you have state-of-the-art it will be encoded as state - of - the - art. The cleanup should remove those spaces between -. Hope it helps!

8 Likes
3

But may be ‘-’ these like tokens is important for response or output

You sure about that

1 Like