Git lfs enabled but git push throwing error file larger then 10MB

🤗Hub
1

I have done this multiple times in past successfully, however, as of last 2 days I am having issues, following exact same steps, which I am using on same machine, while uploading artifacts from SOME_REPO git throws me error file larger than 10MB. Here are the steps I am following:

python -m pip install huggingface_hub

huggingface-cli login

git clone https://huggingface.co/psmathur/SOME_REPO

cd SOME_REPO

git lfs install

huggingface-cli lfs-enable-largefiles .

cd ..

cp -r LOCAL_REPO/checkpoint SOME_REPO

cd SOME_REPO

git add .

git config --global user.email "MY_EMAIL@gmail.com"

git config --global user.name "MY_NAME"

git commit -m "Initial Commit"

git push

This throws me error error file larger then 10MB

Here is the huggingface-cli env output:

- huggingface_hub version: 0.16.4
- Platform: Linux-5.15.0-1042-azure-x86_64-with-glibc2.31
- Python version: 3.10.0
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/azureuser/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: psmathur
- Configured git credential helpers: store
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.0.1
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.25.2
- pydantic: N/A
- aiohttp: 3.8.5
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/azureuser/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/azureuser/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/azureuser/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

Any suggestion would be highly appreciated.

1 Like
2

I’m having the same issue.

❯ git lfs install --force
Updated Git hooks.
Git LFS initialized.

❯ git push --recurse-submodules=no hf main -f
Enumerating objects: 18207, done.
Counting objects: 100% (18207/18207), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5958/5958), done.
Writing objects: 100% (18207/18207), 72.05 MiB | 412.15 MiB/s, done.
Total 18207 (delta 13634), reused 16293 (delta 11969), pack-reused 0
remote: Resolving deltas: 100% (13634/13634), done.
remote: -------------------------------------------------------------------------
remote: Your push was rejected because it contains files larger than 10 MiB.
remote: Please use https://git-lfs.github.com/ to store large files.
remote: See also: https://hf.co/docs/hub/repositories-getting-started#terminal
remote: 
remote: Offending files:
remote:   - docs/examples/velodyne-lidar-scan/shelby-scene.ply (ref: refs/heads/main)
remote:   - docs/examples/velodyne-lidar-scan/shelby-scene.ply (ref: refs/tags/v0.3.0-alpha.10)
remote:   - docs/examples/velodyne-lidar-scan/shelby-scene.ply (ref: refs/tags/v0.3.0-alpha.11)
... etc ...

The .git/config shows that it is enabled (I think?):

[remote "hf"]
	url = https://trusktr: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@huggingface.co/spaces/lume-creative/lume
	fetch = +refs/heads/*:refs/remotes/hf/*
[lfs]
	repositoryformatversion = 0
[lfs "https://trusktr:<TOKEN>@huggingface.co/spaces/lume-creative/lume.git/info/lfs"]
	locksverify = false

but git push still fails.

Here’s my huggingface-cli env, which I don’t think really matters:

❯ huggingface-cli env
/Users/trusktr/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 0.26.5
- Platform: macOS-15.0.1-arm64-arm-64bit
- Python version: 3.9.6
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /Users/trusktr/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: trusktr
- Configured git credential helpers: osxkeychain
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /Users/trusktr/.cache/huggingface/hub
- HF_ASSETS_CACHE: /Users/trusktr/.cache/huggingface/assets
- HF_TOKEN_PATH: /Users/trusktr/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /Users/trusktr/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

Any idea why Git LFS is not working?

1 Like
3

From the directory structure, your environment is probably Windows. In the case of Windows, git-lfs will not work properly unless the latest version of git itself is also installed in addition to git-lfs.
Of course, there is also the possibility of more complex problems.

https://git-scm.com/downloads/win

4

I’m on macOS actually. It’s a MacBook Air M2.

I’m just learning about Git LFS, never used it before, but seems very cumbersome to set up. I wouldn’t want to re-write my whole history to do that.

1 Like
5

Oh… I mistook.:sweat_smile: Macs are almost all UNIX now, so they shouldn’t be that different from Linux…

6

Yeah, I think its just that migrating my history to Git LFS seems daunting. I’m a newb, so I didn’t realize that git lfs install was not enough, now I need to migrate my repo which seems like a big hassle.

It would be sweet if HuggingFace can match GitHub file size, so that at least migrating from GitHub will be super easy.

1 Like
7

You’ve solved it!:grinning: Requests for HF are being accepted in the post below.