AutoTokenizer vs. regular Tokenizer

Beginners
1

I understand that there are a few different tokenizers; e.g., DistilBertTokenizerFast, etc. However, I don’t understand the concept of “Auto” in tokenizer selection. Using IMDB text as an example. What do I get if I use AutoTokenizer to tokenize the text?

2

The AutoTokenizer will work on any checkpoint and pick the proper architecture for you (whereas DistilBNertTokenizerFast will only work for distilbert checkpoints).

3

Thanks. In this case, are there reasons to use DistilBertTokenizerFast?