Errors with Tokenizers on Llama

🤗Tokenizers
1

I tried downloading llama3 on my server and train. I downloaded it from meta and I tried to convert the raw meta data. I have protobuf and sentencepiece downloaded still it says I have to download it.

ValueError: Failed to instantiate tokenizer. Please, make sure you have sentencepiece and protobuf installed.

1 Like
2

Assuming that PyTorch is installed. This should be fine, but there is a possibility that the actual error is occurring elsewhere.

pip install protobuf sentencepiece transformers