Hello everyone, I trained a character-level model using a custom tokenizer class (not from Hugging Face). Now I want to create a HF tokenizer so users can consume for inference. However, I only see two character-level tokenizers: Canine and ByT5. These tokenizers use “ord” to get the id, instead of…

Character level tokenizer with specific order

samirchar February 7, 2025, 4:49pm 6

@John6666
I’m trying this but failing with this error: Custom Tokenizer Error - Please Help!

Any idea?

1 Like

Topic		Replies	Views
Character-level tokenizer Beginners	6	9145	May 8, 2024
Creating a Custom Token Vocabulary for GPT-2 🤗Tokenizers	1	181	January 7, 2025
How does one create a custom hugging face model with a already working tokenizer? 🤗Transformers	1	938	July 29, 2024
Fine-Tuning a Text2Text Model using different tokenizer 🤗Transformers	5	56	January 20, 2025
How does one create a HF tokenizer with only a fraction of the vocabulary but without changing the model? Beginners	0	167	February 1, 2024