Value error : sentencepiece

Beginners
1

Hi,

I have value error: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer.

I installed and updated sentencepiece (0.1.95) but still getting the same error. Can someone help?
Thank you!

2

Hi @Katarina, what happens if you try installing transformers in a new environment with

pip install transformers[sentencepiece]

Does that solve the problem?

3 Likes
3

Hi, thanks on answer…I tried with this but still the same error

4

Can you please share the code you are running and the full stack trace / error message?

5

Acctually it is working after I restarted my enviroment…thank you!

2 Likes
6

This works:

pip install sentencepiece
1 Like
7

use this one-

pip install transformers[sentencepiece] datasets

8

this has to do with you shell actually. You are probably using zsh if you are on Mac, and you are getting zsh: no matches found: transformers[sentencepiece].

Solution: pip install "transformers[sentencepiece]"

8 Likes
9

Fix: pip install transformers[sentencepiece]
if it is installed and still getting error, restart the kernel

2 Likes
10

Kernel Restart solved this problem.

3 Likes
11

Thank you!

12

Use pip install transformers[sentencepiece]

And restart the kernel. It will work :slight_smile:

13

I also found that using conda install works. I had to do conda install sentencepiece and conda install protobuf to get it to work for me.

14

Besides conda install sentencepiece, I found that I can do conda install sentence-transformers which pull in sentencepiece also.

15

Yes, I met this issue either.
My platform is below:
MacOS 15.0.1
Python 3.12.6
pip 24.2

Then I executed this command:

pip3 install sentencepiece

And restarted Kernel, then it works

1 Like