I work inside a secure corporate VPN network, so I’m unable to download Huggingface models using from_pretrained commands. However, I can request the security team to whitelist certain URLs needed for my use-case.
The security team has already whitelisted the ‘huggingface.co’ and ‘cdn-lfs.huggingface.co’ URLs. I can now download the files from repo but the loading functions from_pretrained still don’t work.
I think it’s getting blocked while redirecting the requests internally. So, is there a way to know all (hop) URLs I can request to whitelist to make the load functions work?
Having the same issue. Is there a listing of URLs that we can whitelist? Also if there are any planned changes to URLs is there a roadmap so we can stay on top of it?
I’ll try to supply error logs next time I encounter it, but it has come up multiple times for me as well. When we try to call <model>.from_pretrained("repo") in our DataBricks environment, we get an SSL error about not having the proper certificate. We’ve also gotten a max_retries error but I can’t say for certain if that was due to the underlying whitelist request. There are ways around this, but if HF published a domain list that we could use to properly configure our environments, that would be very useful!
hi! any updates on this? or any alternatives to follow meanwhile? I am about to try downloading a model and going offline and then pushing it up to databricks. Yet, if you had a better idea, or tried this before, I’d like to hear.
I have same issue with download from different cdn name.
After our IT team added http://huggingface.co/ and http://cdn-lfs.huggingface.co/ in whitelist.
For example, it is work for download meta-llama/Llama-2-13b-chat.
But error when the cdn become cdn-lfs-us-1.huggingface.co or other regions.
Update? Same issue here. I’ve gotten around by using my home network to connect to the hf repo and download to my workstation cache. Then I reconnect to VPN into the corporate network and copy from my workstation to the server cache. This is painfully slow.
FWIW curl -IL test shows redirection (302 responses) from the repo when I am connected to the corporate network (fails to download). However on my home network there are no redirects (successful download). Is there an issue with general redirection handling?
Note that for security reasons, we recently updated the domain for our CDN; in order to be able to download files you also need to whitelist the following domains:
we have created exception for SSL inspection for FQDN listed by pierric plus these 2 ones:
But it is still does not work, always same error encountered SSL: CERTIFICATE_VERIFY_FAILED when trying to download sentence-transformers/all-MiniLM-L6-v2
Hey @sean-pai, sorry about that, indeed we recently started migrating some repos from LFS to Xet (checkout this blogpost if you want to learn more about Xet).
As a result (and as you found out), you need to add cas-bridge.xethub.hf.co for the download path (I updated my original reply above). We’ll communicate here when we enable the Xet upload path.
Hi @sean-pai, just a quick follow up, we’ve just released the Xet client which can be used to download these repos using the xet format directly. If you are interested in faster downloads of Xet enabled repos, follow these instructions here.
If you install the client and download the same content, you will also need to add two new endpoints, cas-server.xethub.hf.co and transfer.xethub.hf.co.