Failed to upload 5GB csv to huggingface dataset

I was trying to upload a dataset of 2 million * 2 string to my dataset but keep getting an “internal error” when it almost complete. any thought why this might happen? Thanks for the advice!

Hi ! Can you try separating your csv file into several files and try again ? Maybe 1GB per file ?

Thanks for the quick reply! I managed to upload a 400MB file to the dataset, now my problem is how to upload 70 of them in bulk? doing it one by one is frustrating…

We’re working on letting users upload an entire folder on the website. In the meantime, you can use git to upload them all at once (more info on how to use git to upload a dataset here : Share)