“OSError: Model name './XX' was not found in tokenizers model name list” - cannot load custom tokenizer in Transformers

🤗Tokenizers
9

Isn’t AlbertTokenizerFast only available in from transformers if tokenizers is installed? I don’t see where else it is imported in transformers.init.

https://github.com/huggingface/transformers/blob/e977ed2142a022aa969c03836340edcff4f479b2/src/transformers/init.py#L241-L242

@tlqnguyen Make sure the path is correct. You are using a relative path now, so make sure it is relative to your current script. If you can’t get it to work, try an absolute path.