Hugging Face Forums
What does the parameter 'clean_up_tokenization_spaces' do in the tokenizer.decode function?
Beginners
ivnle
May 2, 2022, 2:28am
#1
As titled. The
documentation
is a bit vague.
Whether or not to clean up the tokenization spaces.