Hey, I’d like to host a dataset on Hugging Face – The AI community building the future. Is there a size limit? My dataset is 106GB. I tried to add it but I can’t even add the files to the git repo (I’m getting fatal: Out of memory, realloc failed after git add), although I’m using git-lfs Bests, …

Hi @PaulLerner and @dynamicwebpaige ! As far as I know, we do have datasets with some Terabytes. As Paige suggested, you can store your dataset in alternate locations, but it is also possible (as far as I know) to upload datasets above 5GB with huggingface-cli lfs-enable-largefiles . This is simila…

Is there a size limit for dataset hosting

🤗Datasets

wendlerc July 5, 2023, 11:33am 7

No limit for individual files. Ok wow that sounds great!!! Does this mean that there is also no limit for all files combined?

Best,
Chris

1 Like

Topic		Replies	Views
Max individual file size for LFS files is 46.6GB 🤗Datasets	2	3199	May 19, 2022
Error while uploading files larger than 10Mb 🤗Datasets	1	913	October 10, 2022
Cannot push dataset of 100k images 🤗Hub	0	589	March 26, 2023
Can we upload datasets with a total size like the pile? 🤗Datasets	2	641	December 10, 2021
Recommendations on millions of files 🤗Datasets	0	123	June 18, 2024

Is there a size limit for dataset hosting

Related topics