Hey, I’d like to host a dataset on Hugging Face – The AI community building the future. Is there a size limit? My dataset is 106GB. I tried to add it but I can’t even add the files to the git repo (I’m getting fatal: Out of memory, realloc failed after git add), although I’m using git-lfs Bests, …

Hi @PaulLerner and @dynamicwebpaige ! As far as I know, we do have datasets with some Terabytes. As Paige suggested, you can store your dataset in alternate locations, but it is also possible (as far as I know) to upload datasets above 5GB with huggingface-cli lfs-enable-largefiles . This is simila…

Is there a size limit for dataset hosting

🤗Datasets

ProgramComputer August 23, 2023, 6:10pm 13

@julien-c The limit as of now is 50 GB for each LFS file on upload. I did not see this in the documentation.

Topic		Replies	Views
Request for Additional Storage Space for Dataset Repository 🤗Datasets	3	91	October 11, 2024
Failed to upload 5GB csv to huggingface dataset Beginners	3	499	March 10, 2022
Cannot push dataset of 100k images 🤗Hub	0	567	March 26, 2023
Recommendations on millions of files 🤗Datasets	0	115	June 18, 2024
Recommended max size of dataset? 🤗Datasets	5	49	March 11, 2025

Is there a size limit for dataset hosting

Related topics