Hugging Face Forums
How to avoid PreTrainedTokenizerFast.decode to add space between tokens
🤗Transformers
John6666
April 22, 2025, 5:55am
2
Hmm…
clean_up_tokenization_spaces
?
show post in topic
Related topics
Topic
Replies
Views
Activity
What does the parameter 'clean_up_tokenization_spaces' do in the tokenizer.decode function?
Beginners
1
8660
December 5, 2022
How to decode with spaces?
🤗Tokenizers
0
1820
April 28, 2022
How to make tokenizer add the spaces correctly when decoding a sequence when set add_prefix_space=False
🤗Tokenizers
0
557
October 9, 2023
BPEDecoder no spaces after special tokens
Intermediate
4
2004
April 19, 2023
Tokenizer: what function removes spaces between '<' and '>'?
🤗Tokenizers
0
45
December 9, 2024