This category is for any question related to the :hugs: Tokenizers library . You can also file an issue .

About the Tokenizers category

sgugger July 7, 2020, 4:16pm 1

This category is for any question related to the Tokenizers library. You can also file an issue.

Topic	Replies	Views
Using the Tokenizers library in a Unity project 🤗Tokenizers	595	May 4, 2022
Can't get to the source code of `tokenizer.convert_tokens_to_string` 🤗Tokenizers	281	February 28, 2023
Unable to register my own tokenizer 🤗Tokenizers	133	November 21, 2023
Tokenizer from own vocab 🤗Tokenizers	361	July 11, 2022
Easy way to compare tokenizers 🤗Tokenizers	236	May 1, 2023