Adding atomic / indivisible tokens to BPE tokenizer

[John6666]

Hmm… Like this?

#tokenizer.pre_tokenizer = AtomicUnitPreTokenizer(atomic_units)
tokenizer.pre_tokenizer = PreTokenizer.custom(AtomicUnitPreTokenizer(atomic_units))

Creating a custom pre-tokenizer

Cannot inject custom PreTokenizer into Tokenizer

How to add additional custom pre-tokenization processing?

I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_to…