Error with new tokenizers (URGENT!)

🤗Tokenizers
1

Hi, recently all my pre-trained models undergo this error while loading their tokenizer:

Couldn't instantiate the backend tokenizer from one of: (1) a tokenizers library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

I tried to pip install sentencepiece but this does not solve the problem. Do you know any solution? (I am working on Google Colab)

Note: In my humble opinion, changing so important things so fast can generate very dangerous problems. All my students (I teach DL stuff) and clients are stuck on my notebooks. I can understand that after a year a code can become outdated, but not just after two months. This requires a lot of maintenance work from my side!

3 Likes
2

There were some breaking changes in the V4 release, please find the details here:

2 Likes
3

Thank you so much @FL33TW00D. Two steps were required.

  1. latest pip version 20.3.3 (On Colab I had installed by default 19 and something);
  2. use_fast = False.

Working a lot on Italian or personal custom models, all of them were disabled all of a sudden!

3 Likes
4

Hi @denocris I am facing the same problem. I am a student and I depend on this for my presentation tomorrow. I am also using Google Colab. Can you please explain in steps how to fix this sentencpiece problem? Thanks in advance

1 Like
5

I resolved it.

  1. Uninstalled transformers
  2. Installed transformers sentencepiece like this : !pip install --no-cache-dir transformers sentencepiece
  3. Use_fast= False like this: tokenizer = AutoTokenizer.from_pretrained(“XXXXX”, use_fast=False)
20 Likes
6

Sorry @Ogayo, I have just read. I am happy you solved it. It is quite annoying to have these kind of issues from one day to another. I had this error during a live presentation. A couple of days before the notebook was working well!

7

Thanku @Ogayo

1 Like
8

Thank you @Ogayo !!

1 Like
9

THANKS A LOT @Ogayo

1 Like
10

Thank you very much @Ogayo.

1 Like
11

Hi @Ogayo ,im getting the same error after resolving the following steps also
what should i do ?

1 Like
12

Same problem here.
Solved it using:
pip install sentencepiece

13

I tried using use_fast=False but got the following error:


SyntaxError: keyword argument repeated: use_fast

I think it’s deprecated now! but the installation part works, so thank you.