I understand that there are a few different tokenizers; e.g., DistilBertTokenizerFast, etc. However, I don’t understand the concept of “Auto” in tokenizer selection. Using IMDB text as an example. What do I get if I use AutoTokenizer to tokenize the text?

AutoTokenizer vs. regular Tokenizer

vett93 June 2, 2021, 5:30pm 3

Thanks. In this case, are there reasons to use DistilBertTokenizerFast?

Topic		Replies	Views
Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer? 🤗Transformers	2	3278	July 10, 2021
Importing a DistilBertTokenizer does not work using AutoTokenizer Beginners	0	658	November 8, 2023
Custom DistilBertTokenizer training 🤗Transformers	3	659	November 13, 2020
How to convert Tokenizer to TokenizerFast? Beginners	1	550	September 30, 2020
Is there any difference in the tokenized output if I load the tokenizer from a different pretrained model Beginners	2	383	September 3, 2020