Hugging Face Forums
Implementing custom tokenizer components (normalizers, processors)
🤗Tokenizers
saattrupdan
November 30, 2021, 12:41pm
2
For anyone else looking, this
can
be done, and it’s answered in this question:
show post in topic
Related topics
Topic
Replies
Views
Activity
How to add additional custom pre-tokenization processing?
🤗Tokenizers
6
5183
March 7, 2023
Custom PostProcessor?
🤗Tokenizers
0
915
November 10, 2022
Modifying normalizer for pretrained tokenizers don't consistently work
🤗Tokenizers
2
118
June 12, 2024
Tokenizer post_processor help
🤗Tokenizers
1
1366
October 27, 2022
Adding atomic / indivisible tokens to BPE tokenizer
🤗Tokenizers
8
34
July 3, 2025