Value error : sentencepiece

Beginners
1

Hi,

I have value error: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer.

I installed and updated sentencepiece (0.1.95) but still getting the same error. Can someone help?
Thank you!

1 Like
2

Hi @Katarina, what happens if you try installing transformers in a new environment with

pip install transformers[sentencepiece]

Does that solve the problem?

3 Likes
3

Hi, thanks on answer…I tried with this but still the same error

4

Can you please share the code you are running and the full stack trace / error message?

5

Acctually it is working after I restarted my enviroment…thank you!

2 Likes
6

This works:

pip install sentencepiece
1 Like
7

use this one-

pip install transformers[sentencepiece] datasets

8

this has to do with you shell actually. You are probably using zsh if you are on Mac, and you are getting zsh: no matches found: transformers[sentencepiece].

Solution: pip install "transformers[sentencepiece]"

7 Likes
9

Fix: pip install transformers[sentencepiece]
if it is installed and still getting error, restart the kernel

2 Likes
10

Kernel Restart solved this problem.

3 Likes
11

Thank you!

12

Use pip install transformers[sentencepiece]

And restart the kernel. It will work :slight_smile:

13

I also found that using conda install works. I had to do conda install sentencepiece and conda install protobuf to get it to work for me.

14

Besides conda install sentencepiece, I found that I can do conda install sentence-transformers which pull in sentencepiece also.

15

Yes, I met this issue either.
My platform is below:
MacOS 15.0.1
Python 3.12.6
pip 24.2

Then I executed this command:

pip3 install sentencepiece

And restarted Kernel, then it works

1 Like
16

For those who find this thread before others across the web, sentencepiece is currently failling install in python 3.13. Other threads suggest that installing 3.12 can resolve the issue and allow for the install to complete.

sentencepiece does not support python 3.13 yet.
Its latest version was released on Feb 19, 2024.
You need to downgrade to python 3.12
sentencepiece · PyPI

also, for those new to python, you can have more than one python version installed at the same time.

If you have multiple versions of python installed, don’t forget to use “py --list” to see which ones you got. As an example, after discovering that the issue is with python 3.13, I explicitly told it to use update 3.11 as follows: “py -3.11 -m pip install -r requirements.txt” This got through the sentencepiece part of my requirements file

quotes copied from here

1 Like