Error creating custom pre_tokenizer

John6666 · January 1, 2025, 11:44pm

There may be a workaround.

Cannot inject custom PreTokenizer into Tokenizer

opened 07:56PM - 23 Sep 24 UTC

Hey, I want to train a Tokenizer that operates on a custom PreTokenizer. I tr…ied a mix of [this documentation post](https://huggingface.co/docs/tokenizers/pipeline) and [this example](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom_components.py). My resulting code looks like this: ```python class GlyLESPreTokenizer: def __init__(self, *args, **kwargs): pass def __new__(cls, *args, **kwargs): return super().__new__(cls) def glyles_split(self, iupac: str): iuapc = iupac.strip().replace(" ", "") token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}"))) GlyLESParser(token).start() idx = 0 output = [] for i in range(1, len(token.tokens) - 2): txt = str(token.tokens[i].text) output.append((txt, (idx, idx + len(txt)))) idx += len(txt) return output def pre_tokenize_str(self, input_: str): return self.glyles_split(input_) iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc" # This returns a list of 33 token GlyLESPreTokenizer().pre_tokenize_str(iupac) # This however only returns a list with one token that is the entire input string. pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre_tokenize_str(iupac) ``` The final idea is to use it in such setting: ```python tokenizer = Tokenizer(models.Model()) tokenizer.normalizer = normalizers.Strip() tokenizer.pre_tokenizer = pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()) ``` Can someone help me to understand how to use the `pre_tokenizers.PreTokenizer.custom` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from [GlyLES](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer. Thank you for any help, comment, or feedback in advance. Roman

Topic		Replies	Views
How to add additional custom pre-tokenization processing? 🤗Tokenizers	6	5203	March 7, 2023
Train_from_iterator throwing TypeError: expected string or buffer errir 🤗Tokenizers	2	24	January 3, 2025
Writing custom tokenizer and wrapping it in tokenizer object 🤗Tokenizers	2	794	June 26, 2023
How to create a tokenizers from a custom pretrained tokenizer? 🤗Transformers	1	601	February 24, 2021
Save tokenizer with argument 🤗Tokenizers	2	1966	October 26, 2022

Error creating custom pre_tokenizer

Related topics