Uploading files larger than 5GB to model hub

I want to upload ctrl to model hub. I have followed the instructions from the documentation and it seems that they are applicable for smaller models (<5GB). Issues have been raised here and here but it still seems unresolved.

I followed the one shared by @julien-c (aws s3 cp …) but got the error Unable to locate credentials indicating that they are for HF staff.

I am getting the error could not push some refs as 5GB limit is crossed. Can you please suggest a workaround if that exists ?

@patrickvonplaten

Did you install git-lfs? git lfs install

You also have to install our custom transfer agent for files >5GB: transformers-cli lfs-enable-largefiles

Let me know if this helps!

Thanks for replying. I have git lfs and I tried custom transfer agent earlier. I tried it again just now, still gives me:

EOFoading LFS objects:  50% (1/2), 5.0 GB | 0 B/s                                                                                                                                                            
error: failed to push some refs to 'https://huggingface.co/prajjwal1/ctrl_discovery_1'

I manged to do it on the server. For some reason, it won’t work on local machine.

are you able to upload files to S3 from your local machine? Might be a proxy-related issue.

If you want more info, you can try GIT_TRACE=1 GIT_CURL_VERBOSE=1 git push

I have loaded many models with earlier version of transformers which used to not rely on lfs. I’m not sure whats the issue on local machine, but its working on the server. May use the flag if I would need a trace.

@prajjwal1 git-lfs does its own networking (upload) calls with libcurl so the behavior when behind a proxy might be different from python-requests.

Let me know if you get more info in the future, this is interesting!