I have a huge (100GB+) dataset of audio (.wav files) and its respective metadata I was able to easily load the dataset using load_dataset and uploaded it using push_to_hub which converts it to a parquet file what is the best way to upload such large dataset (particularly images and audio) I want to …

Standard way to upload huge dataset

severo April 12, 2024, 8:04am 2

Topic		Replies	Views
Stream Audio Dataset that Can't be moved to Hub 🤗Datasets	7	494	March 17, 2023
Unable to upload large audio dataset using push_to_hub 🤗Datasets	5	871	November 17, 2023
Allow streaming of large datasets with image/audio 🤗Datasets	18	3980	May 30, 2022
How to convert dir-with-images properly? 🤗Datasets	2	440	June 11, 2024
Image dataset best practices? 🤗Datasets	9	17567	January 15, 2023