Hugging Face Forums
How can I check the implementation of tokenizer.decode()
🤗Transformers
HaiShi
September 30, 2024, 8:12am
7
Thanks for the reply, I will try what you say
1 Like
show post in topic
Related topics
Topic
Replies
Views
Activity
PreTrainedTokenizerFast.convert_tokens_to_string always assumes the presence of decoder
Intermediate
2
68
November 7, 2024
How to avoid PreTrainedTokenizerFast.decode to add space between tokens
🤗Transformers
3
60
April 22, 2025
Decoding sequence of tokens produces question marks instead of actual tokens
🤗Tokenizers
1
29
September 3, 2024
Decode token IDs into a list (not a single string)
🤗Tokenizers
4
4429
March 11, 2025
Return_offsets_mapping when decoding
🤗Tokenizers
3
55
April 25, 2025