I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since…

How to add additional custom pre-tokenization processing?

anthony February 20, 2021, 6:21pm 5

Thanks for letting me know! Just fixed it.

Topic		Replies	Views
Implementing custom tokenizer components (normalizers, processors) 🤗Tokenizers	1	2892	November 30, 2021
Error creating custom pre_tokenizer 🤗Tokenizers	3	49	January 2, 2025
Custom PostProcessor? 🤗Tokenizers	0	918	November 10, 2022
Save tokenizer with argument 🤗Tokenizers	2	1966	October 26, 2022
Writing custom tokenizer and wrapping it in tokenizer object 🤗Tokenizers	2	794	June 26, 2023