I built a tokenizer and trained an LM from scratch following this link . Then I used this to train a clip using this tokenizer. The training went fine. Now, when I load this clip for evaluation, I get this error: ValueError: The backend_tokenizerprovided does not match the expected format. The CLI…

CLIP: The `backend_tokenizer` provided does not match the expected format

prabhatkr May 9, 2024, 4:52pm 3

I took some time to study and explore the CLIP. I was using FAST tokenizer having tokenizer.json only. I converted it to Slow and now it works fine. Thanks @Molbap

1 Like

Topic		Replies	Views
Why does PreTrainedTokenizerFast return a list instead of tokenizers.Encoding instance? Beginners	1	315	February 6, 2023
Questions when doing Transformer-XL Finetune with Trainer Beginners	3	1053	October 6, 2021
Convert a Python Tokenizer into a TokenizerFast Beginners	0	338	May 20, 2022
Error with new tokenizers (URGENT!) 🤗Tokenizers	16	50741	July 22, 2024
Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library 🤗Tokenizers	1	1078	August 30, 2021

CLIP: The `backend_tokenizer` provided does not match the expected format

Related topics