class IngenxPreTokenizer(PreTokenizer): def __init__(self): super().__init__() self.base_tokenizer = IngenxTokenizer() def pre_tokenize(self, pretok: PreTokenizedString): # text = pretok.normalized processed = self.base_tokenizer.process_text(pretok) n…

Error creating custom pre_tokenizer

John6666 January 2, 2025, 8:08am 4

It doesn’t seem wrong…

Most of the tokenizer library is written in Rust, so the Python parts that were added afterwards might as well not inherit from classes.

# class IngenxPreTokenizer(PreTokenizer):
# class IngenxPreTokenizer():

Topic		Replies	Views
How to add additional custom pre-tokenization processing? 🤗Tokenizers	6	5203	March 7, 2023
Train_from_iterator throwing TypeError: expected string or buffer errir 🤗Tokenizers	2	24	January 3, 2025
Writing custom tokenizer and wrapping it in tokenizer object 🤗Tokenizers	2	794	June 26, 2023
How to create a tokenizers from a custom pretrained tokenizer? 🤗Transformers	1	601	February 24, 2021
Save tokenizer with argument 🤗Tokenizers	2	1966	October 26, 2022