Implementing custom tokenizer components (normalizers, processors)

[saattrupdan]

For anyone else looking, this can be done, and it’s answered in this question:

How to add additional custom pre-tokenization processing?

I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_to…