Hello,
The new “Model sharing and uploading” using git is a great idea.
However, I tried to update one of my models “t5-3b” which is “11G”, and it is taking too much time for “git add --all” command.
I had to cancel it at the end.
This is because the nature of git for large files while calculating the diff.
Is there any solution for this problem or workaround ?
To have an idea of which steps take time, you can add the following env variables to your git commands:
GIT_CURL_VERBOSE=1 GIT_TRACE=1
Also make sure you actually have git-lfs activated: git lfs install. (We’re in the process of adding it everywhere in the documentation as it was easy to miss).
On the hashing overhead part: for me hashing (w/ sha256) a 42GB file takes 2’30sec (on a beefy computer) so the hashing overhead is not too huge, and of course gives the benefit of security/reproducibility.
And finally in your precise use case, uploading files larger than 5GB is currently not supported out of the box, though we’ll support it in the next 2-3 weeks. Issue to follow is at https://github.com/huggingface/transformers/issues/8480#issuecomment-726731046
Thanks @julien-c , the issei’s was indeed on “git lsf”. After reinstalling it, it worked much faster.