Git push rejected

anjalyjayakrishnan · February 7, 2023, 3:09am

Hi,

I am trying to create an Audio dataset. Not able to push data to my repository. Getting error like as shown below.

Uploading LFS objects: 100% (23938/23938), 28 GB | 0 B/s, done.                                                                               
Enumerating objects: 25039, done.
Counting objects: 100% (25039/25039), done.
Delta compression using up to 32 threads
Compressing objects: 100% (24983/24983), done.
Writing objects: 100% (25038/25038), 11.29 MiB | 297.00 KiB/s, done.
Total 25038 (delta 67), reused 24963 (delta 46), pack-reused 0
remote: Resolving deltas: 100% (67/67), done.
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/my_dataset
 ! [remote rejected]   main -> main (pre-receive hook declined)

Audio files and csvs are within ‘data’ folder. Please help me to resolve this.

radames · February 7, 2023, 4:28am

hi @anjalyjayakrishnan ,

It looks like you’re trying to push to https://huggingface.co/datasets/my_dataset ?

Could you please share your .git/config inside your repo, if it does not contain any confidential tokens?
You should see something similar to this

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	ignorecase = true
	precomposeunicode = true
[remote "origin"]
	url = https://huggingface.co/datasets/radames/test
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
	remote = origin
	merge = refs/heads/main
[lfs]
	repositoryformatversion = 0
[lfs "https://huggingface.co/datasets/radames/test.git/info/lfs"]
	locksverify = false
	access = basic

anjalyjayakrishnan · February 7, 2023, 5:53am

Hi @radames ,

Thanks for your time!

Here is my .git/config content.

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://huggingface.co/datasets/bridgeconn/snow_mountain
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
        remote = origin
        merge = refs/heads/main
[lfs "customtransfer.multipart"]
        path = huggingface-cli
        args = lfs-multipart-upload
[credential]
        helper = store
[lfs "https://huggingface.co/datasets/bridgeconn/snow_mountain.git/info/lfs"]
        locksverify = false
        access = basic
[pull]
        rebase = false

radames · February 7, 2023, 6:14am

hi, it seems all right on your git/config. Do you have more context on the original error? maybe more lines about why your push was rejected?

Also noticed here .gitattributes · bridgeconn/snow_mountain at main
that csv files are not on your list, do you have large csv files > 10MB ? if that’s the case you have to use lfs to track them

git lfs track "*.csv"
git add *.csv
git commit

anjalyjayakrishnan · February 7, 2023, 7:11am

@radames ,

My csvs are of < 10MB only. My ‘data’ folder is of size 27GB. It has audios and csvs. This is what i gets while doing git push

git push origin main
Uploading LFS objects: 100% (23938/23938), 28 GB | 0 B/s, done.                                                                               
Enumerating objects: 25048, done.
Counting objects: 100% (25047/25047), done.
Delta compression using up to 32 threads
Compressing objects: 100% (24989/24989), done.
Writing objects: 100% (25044/25044), 11.23 MiB | 295.00 KiB/s, done.
Total 25044 (delta 72), reused 24963 (delta 46), pack-reused 0
remote: Resolving deltas: 100% (72/72), completed with 1 local object.
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/bridgeconn/snow_mountain
 ! [remote rejected]   main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://huggingface.co/datasets/bridgeconn/snow_mountain'

radames · February 7, 2023, 7:22am

can you try to track the *.csv via lfs? git lfs track "*.csv" you might need to restart the repo and force push

anjalyjayakrishnan · February 9, 2023, 5:40am

@radames ,

Sorry for the late reply. Tracked csv files… still facing the same issue. Do you have any ideas that I can try?

radames · February 9, 2023, 5:47am

hi @anjalyjayakrishnan sorry it’s still not working,
is this the repo you’re trying? I don’t see csv tracked on .gitattributes here

Maybe @mariosasko can help us?

anjalyjayakrishnan · February 9, 2023, 5:56am

Yes it is the repo. I tried tracking csvs and few more things. But I couldn’t push my files. Then started again from my initial commit (that’s why csv not in the .gitattributes) and tried to push few audios, csv, and loading script. It worked and you can see that in the repo. Not able to push full data.

radames · February 9, 2023, 6:12am

ok, I’ve cloned you repo and restart the commits, can you try cloning or duplicating radames/snow-mountain-test · Datasets at Hugging Face
and then add, commit and push? Or share a subset of your dataset?

anjalyjayakrishnan · February 9, 2023, 1:42pm

@radames ,

Couldn’t push data to radames/snow-mountain-test. I can share a part of my dataset. How can I share it?

anjalyjayakrishnan · February 10, 2023, 12:10am

@radames ,

Thanks a lot for your time!

I managed to push the data. I have sub-directories in my audio folder. Just tried to push by making them zip files. It worked.

stma · February 16, 2023, 1:20pm

This is happening to me too, don’t know how to fix it

Uploading LFS objects: 100% (10000/10000), 8.5 GB | 21 MB/s, done.
Enumerating objects: 10014, done.
Counting objects: 100% (10014/10014), done.
Delta compression using up to 16 threads
Compressing objects: 100% (10013/10013), done.
Writing objects: 100% (10014/10014), 1.37 MiB | 789.00 KiB/s, done.
Total 10014 (delta 0), reused 10014 (delta 0), pack-reused 0
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/stma/danbooru-metadata
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://huggingface.co/datasets/stma/danbooru-metadata'

stma · February 17, 2023, 6:03am

Gave up and tried disabling LFS, now I get this error when trying to push:

Enumerating objects: 24029, done.
Counting objects: 100% (24029/24029), done.
Delta compression using up to 16 threads
Compressing objects: 100% (24027/24027), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (24028/24028), 7.00 GiB | 20.13 MiB/s, done.
Total 24028 (delta 0), reused 24028 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly
Everything up-to-date

stma · February 17, 2023, 6:19am

Tried again, this time making a single commit for each directory of files. They’re 12 directories of 1000 zip files each. That worked. Not sure why it would not work if I committed them all at once. It’s 11999 files total

Sylvestre · February 17, 2023, 9:51am

Hello,

We have investigated the problem, and we have deployed a fix for it, let me know if happens again

Cheers

anjalyjayakrishnan · March 7, 2023, 7:11am

@stma ,

I just saw your previous messages. Sorry for this late reply. Hope your issue is fixed.

dkqkxx · April 6, 2023, 6:09pm

same errors

02:08:06.487430 trace git-lfs: xfer[huggingface-cli]: File “C:\CodeEnv\Python\lib\site-packages\requests\adapters.py”, line 559, in send
02:08:06.487939 trace git-lfs: xfer[huggingface-cli]: raise ProxyError(e, request=request)
02:08:06.487960 trace git-lfs: xfer[huggingface-cli]: requests.exceptions.ProxyError: HTTPSConnectionPool(host=‘s3.us-east-1.amazonaws.com’, port=443): Max retries exceeded with url: /lfs.huggingface.co/repos/9e/58/9e5880f0fd2c6de0bed7dce546ecd12063e3c075b291a60eca162833eb16b2d1/cd53101409717bd31a8c726d2e29208ceeb61525ee69254fb952a494308a9867?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGO27GPWFUO%2F20230406%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230406T180720Z&X-Amz-Expires=86400&X-Amz-Signature=b7101ccc55fb4e98c8ef51bc3f0da8b5891fc1350a0cd9d3beedf2d41d38cc9e&X-Amz-SignedHeaders=host&partNumber=2&uploadId=4gkaMP4WcOBNSRwowXog_.HQajNmo.QZdINHdbr2vhvff8dOXAj9GKYHW1Qz0axs8if761DAfrfMM8sn1CeDej99ROsogfiXkxi4eA_uzhXI9ye8xkKY48eAegYS8Az8&x-id=UploadPart (Caused by ProxyError(‘Cannot connect to proxy.’, ConnectionResetError(10054, ‘远程主机强迫关闭了一个现有的连接。’, None, 10054, None)))
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 0 finished job for “cd53101409717bd31a8c726d2e29208ceeb61525ee69254fb952a494308a9867”
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” End()
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 3 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 3
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 6 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 6
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 1 stopping
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 0 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 0
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 2 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 2
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 4 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 4
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 1
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 7 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 7
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 3 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: adapter “multipart” worker 5 stopping
02:08:06.511866 trace git-lfs: xfer: Shutting down adapter worker 5
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 0 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 5 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 7 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 6 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 4 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 2 sending message: {“event”:“terminate”}
02:08:06.511866 trace git-lfs: xfer: error finishing up custom transfer process “huggingface-cli” worker 0, aborting: write |1: The pipe is being closed.
02:08:06.511866 trace git-lfs: xfer: Aborting worker process: 0
02:08:06.511866 trace git-lfs: xfer: Custom adapter worker 1 sending message: {“event”:“terminate”}
02:08:06.642464 trace git-lfs: xfer: adapter “multipart” stopped
EOFoading LFS objects: 0% (0/1), 16 MB | 204 KB/s
error: failed to push some refs to ‘QingyiSi/Alpaca-CoT · Datasets at Hugging Face’

luizapzbn · April 14, 2023, 7:48pm

I was having a similar error. Even though I ran the git lfs track *.jsonl it was not working. I suspect it was because my files were in a folder inside the repo.

batch response: Too many password attempts B/s
Uploading LFS objects:   0% (0/2), 0 B | 0 B/s, done
error: failed to push some refs to 'git@hf.co:datasets......'

In my case, I solved it by using git lfs track FOLDER_NAME.

radames · April 14, 2023, 8:56pm

Yes you can also do git lfs track "*.jsonl" with quotes, so it tracks all the files on the repo and subfolders. You can check your .gitattributes to see the differences