This category is for any question related to the :hugs: Tokenizers library . You can also file an issue .

About the Tokenizers category

sgugger July 7, 2020, 4:16pm 1

This category is for any question related to the Tokenizers library. You can also file an issue.

Topic		Replies	Views
About the Transformers category 🤗Transformers	1	241	July 7, 2020
About the datasets category 🤗Datasets	1	346	July 7, 2020
Using the Tokenizers library in a Unity project 🤗Tokenizers	0	649	May 4, 2022
Can't get to the source code of `tokenizer.convert_tokens_to_string` 🤗Tokenizers	0	336	February 28, 2023
Unable to register my own tokenizer 🤗Tokenizers	0	176	November 21, 2023