Hi everyone, I’m trying to understand what is the difference between the DistilBertTokenizerFast and the DistilBertTokenizer . From the documentation it looks like that the DistilBertTokenizerFast “Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’s tokenizers library).” whereas the Di…

Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer?

juroylim July 10, 2021, 9:51am 2

@ayalaall I also have the same thought as you.

I also want to know about this. Are they the same in terms of function?

1 Like

Topic		Replies	Views
Difference between tokenizer and tokenizerfast Beginners	4	4197	December 22, 2023
How to convert Tokenizer to TokenizerFast? Beginners	1	544	September 30, 2020
Custom DistilBertTokenizer training 🤗Transformers	3	657	November 13, 2020
Newbie: Main difference between tokenizers? 🤗Tokenizers	0	832	May 6, 2021
AutoTokenizer vs. regular Tokenizer Beginners	2	1587	June 2, 2021