ISSUE 401 Unauthorized with transformers, but curl authentication works

🤗Hub
1

Issue:
Authentication works with curl but fails with Python libraries like transformers and huggingface_hub, resulting in a 401 Unauthorized error, even for public models.

Details:

  • curl Success:
    The command curl -H “Authorization: Bearer ” https://huggingface.co//resolve/main/config.json successfully retrieves the config file.
  • Python Failure:
    • Using AutoTokenizer.from_pretrained(“”) or direct downloads with huggingface_hub fails with 401 Unauthorized.
    • This occurs for both gated and public models (e.g., bert-base-uncased).
  • Environment:
    • Anaconda, Python 3.12
    • transformers 4.50.0, huggingface_hub 0.29.3
    • Token set as HF_TOKEN environment variable
    • Git installed and in PATH
    • Cache cleared
    • PyTorch installed

Additional Notes:

  • The token is valid (works with curl), suggesting a configuration or handling issue in the Python libraries.

Request:
Looking for help to understand and fix why it works with curl but not with Python libraries.

Thank you!

1 Like
2

There don’t seem to be any major changes to the base of from_pretrained(), but the HF_TOKEN and environment variable-related parts have changed quite a bit…

You might be able to find out more by turning on HF_DEBUG and running it.

import os
os.environ["HF_DEBUG"] = "1"