Hello, there I wonder if the tokenizer of t51.1 is the same as that of normal t5. [image] Or is the pre-tokenization step not the same?

T5 tokenizer vs t51.1 tokenizer

Nevermetyou March 1, 2024, 2:35am 1

Hello, there

I wonder if the tokenizer of t51.1 is the same as that of normal t5.

Or is the pre-tokenization step not the same?

Topic		Replies	Views
SentencePieceUnigramTokenizer 🤗Tokenizers	0	523	September 22, 2022
Easy way to compare tokenizers 🤗Tokenizers	0	237	May 1, 2023
Can i use BPE tokenizer for T5 model not pretrain Beginners	0	286	April 13, 2022
Bos token for T5? 🤗Transformers	0	487	November 6, 2022
Difference between vocab_size in model T5forConditionalGeneration "t5-small" and its corresponding Tokenizer "t5-small" 🤗Transformers	1	544	June 30, 2023