Hi,
I am trying to create an Audio dataset. Not able to push data to my repository. Getting error like as shown below.
Uploading LFS objects: 100% (23938/23938), 28 GB | 0 B/s, done.
Enumerating objects: 25039, done.
Counting objects: 100% (25039/25039), done.
Delta compression using up to 32 threads
Compressing objects: 100% (24983/24983), done.
Writing objects: 100% (25038/25038), 11.29 MiB | 297.00 KiB/s, done.
Total 25038 (delta 67), reused 24963 (delta 46), pack-reused 0
remote: Resolving deltas: 100% (67/67), done.
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/my_dataset
! [remote rejected] main -> main (pre-receive hook declined)
Audio files and csvs are within ‘data’ folder. Please help me to resolve this.
hi @anjalyjayakrishnan ,
It looks like you’re trying to push to https://huggingface.co/datasets/my_dataset
?
Could you please share your .git/config
inside your repo, if it does not contain any confidential tokens?
You should see something similar to this
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = true
[remote "origin"]
url = https://huggingface.co/datasets/radames/test
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
remote = origin
merge = refs/heads/main
[lfs]
repositoryformatversion = 0
[lfs "https://huggingface.co/datasets/radames/test.git/info/lfs"]
locksverify = false
access = basic
Hi @radames ,
Thanks for your time!
Here is my .git/config content.
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://huggingface.co/datasets/bridgeconn/snow_mountain
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
remote = origin
merge = refs/heads/main
[lfs "customtransfer.multipart"]
path = huggingface-cli
args = lfs-multipart-upload
[credential]
helper = store
[lfs "https://huggingface.co/datasets/bridgeconn/snow_mountain.git/info/lfs"]
locksverify = false
access = basic
[pull]
rebase = false
hi, it seems all right on your git/config. Do you have more context on the original error? maybe more lines about why your push was rejected?
Also noticed here .gitattributes · bridgeconn/snow_mountain at main
that csv
files are not on your list, do you have large csv
files > 10MB ? if that’s the case you have to use lfs
to track them
git lfs track "*.csv"
git add *.csv
git commit
@radames ,
My csvs are of < 10MB only. My ‘data’ folder is of size 27GB. It has audios and csvs. This is what i gets while doing git push
git push origin main
Uploading LFS objects: 100% (23938/23938), 28 GB | 0 B/s, done.
Enumerating objects: 25048, done.
Counting objects: 100% (25047/25047), done.
Delta compression using up to 32 threads
Compressing objects: 100% (24989/24989), done.
Writing objects: 100% (25044/25044), 11.23 MiB | 295.00 KiB/s, done.
Total 25044 (delta 72), reused 24963 (delta 46), pack-reused 0
remote: Resolving deltas: 100% (72/72), completed with 1 local object.
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/bridgeconn/snow_mountain
! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://huggingface.co/datasets/bridgeconn/snow_mountain'
can you try to track the *.csv
via lfs? git lfs track "*.csv"
you might need to restart the repo and force push
@radames ,
Sorry for the late reply. Tracked csv files… still facing the same issue. Do you have any ideas that I can try?
hi @anjalyjayakrishnan sorry it’s still not working,
is this the repo you’re trying? I don’t see csv
tracked on .gitattributes
here
Maybe @mariosasko can help us?
Yes it is the repo. I tried tracking csvs and few more things. But I couldn’t push my files. Then started again from my initial commit (that’s why csv not in the .gitattributes
) and tried to push few audios, csv, and loading script. It worked and you can see that in the repo. Not able to push full data.
radames
February 9, 2023, 6:12am
10
ok, I’ve cloned you repo and restart the commits, can you try cloning or duplicating radames/snow-mountain-test · Datasets at Hugging Face
and then add, commit and push? Or share a subset of your dataset?
@radames ,
Couldn’t push data to radames/snow-mountain-test . I can share a part of my dataset. How can I share it?
@radames ,
Thanks a lot for your time!
I managed to push the data. I have sub-directories in my audio folder. Just tried to push by making them zip files. It worked.
stma
February 16, 2023, 1:20pm
13
This is happening to me too, don’t know how to fix it
Uploading LFS objects: 100% (10000/10000), 8.5 GB | 21 MB/s, done.
Enumerating objects: 10014, done.
Counting objects: 100% (10014/10014), done.
Delta compression using up to 16 threads
Compressing objects: 100% (10013/10013), done.
Writing objects: 100% (10014/10014), 1.37 MiB | 789.00 KiB/s, done.
Total 10014 (delta 0), reused 10014 (delta 0), pack-reused 0
remote: Scanning LFS files for validity...
remote: KeyError: 'data'
To https://huggingface.co/datasets/stma/danbooru-metadata
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://huggingface.co/datasets/stma/danbooru-metadata'
stma
February 17, 2023, 6:03am
14
Gave up and tried disabling LFS, now I get this error when trying to push:
Enumerating objects: 24029, done.
Counting objects: 100% (24029/24029), done.
Delta compression using up to 16 threads
Compressing objects: 100% (24027/24027), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (24028/24028), 7.00 GiB | 20.13 MiB/s, done.
Total 24028 (delta 0), reused 24028 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly
Everything up-to-date
stma
February 17, 2023, 6:19am
15
Tried again, this time making a single commit for each directory of files. They’re 12 directories of 1000 zip files each. That worked. Not sure why it would not work if I committed them all at once. It’s 11999 files total
Hello,
We have investigated the problem, and we have deployed a fix for it, let me know if happens again
Cheers
1 Like
@stma ,
I just saw your previous messages. Sorry for this late reply. Hope your issue is fixed.