What to do when HuggingFace throws "Can't load tokenizer"

Models
9

You may encounter this error if the model is a gated model, and you need to accept the licensing terms before using the model.

Go to the Model Card in hugging face, accept the licensing term, generate key and add the key in your program

import os
os.environ['HF_TOKEN']='my-key'

this worked for me.

4 Likes