Uploading large files (>5GB) to HF Spaces

Spaces
1

Hi everyone,

I need to use an 8.5 GB faiss index for my HF Spaces app. Although I’ve successfully uploaded large files to datasets and model hub before, I am now getting an error when pushing to HF Spaces repo. I read all the issues and troubleshooting others had with git lfs, but this one I really couldn’t solve after trying for some time.

Is this the recommended approach for large faiss files (pushing with git lfs)? I am using a cloud server so upload speed shouldn’t be an issue. It seems like the first 5 GB are transferred ok, and then towards the end of the transfer, I notice “400 Bad request” error logs. Here are my full git logs.

TIA, Vladimir

1 Like
2

i don’t remember, did we implement lfs-largefiles for Spaces or no? @cbensimon @pierric

1 Like
3

Yes, it should be implemented. It might be buggy though sometimes to upload very large files … not completely sure about this. Are you still facing this issue @vblagoje ?

4

Yes @cbensimon no luck. I still get:

(hf) paperspace@psnwsc5dm:~/lfqa$ git push
Uploading LFS objects: 0% (0/1), 5.0 GB | 13 MB/s EOFoading LFS objects: 0% (0/1), 10 GB | 23 MB/s
error: failed to push some refs to ‘https://huggingface.co/spaces/vblagoje/lfqa’

5

Hey guys, even an alternative approach of getting the faiss file using wget fails with: “No space left on device”. Perhaps there is a size limitation on the container running the HF Spaces app but then how is the runtime related to git? Any thoughts? Can anything be done here @cbensimon ?
Thanks,
Vladimir

1 Like
6

@vblagoje I am facing the exact same issue @cbensimon, please help us!

7

Hey there, I used huggingface hub and it worked! LMK if that worked for you!