Hi all, What I am trying to do is to push the dataset I created locally (which is around 1.2TB) to huggingface datasets.It contains large images and some other textual data paired with them. What I follow: Pass a generator to Dataset.from_generator(), which reads image files (bytes with the help…

Good news, we improved how push_to_hub works : it does only one commit every 50 files instead of every file. The improvement is available if you install datasets from source, and it will be available in 2.15 as well Feel free to try it out

Any workaround for push_to_hub() limits?

🤗Datasets

zeio December 10, 2023, 8:03pm 7

Solved for now by setting max shard size to 700 MB:

batch.push_to_hub('zeio/auto-batch', config_name = 'spoken', max_shard_size = '700MB')

Topic		Replies	Views
Problem "Bad request" when using datasets.Dataset.push_to_hub() 🤗Datasets	6	596	October 28, 2024
Cannot push dataset of 100k images 🤗Hub	0	611	March 26, 2023
Unable to upload large audio dataset using push_to_hub 🤗Datasets	5	907	November 17, 2023
Datasetdict push_to_hub failing with payload to large 🤗Datasets	6	145	February 11, 2025
Exceeded our hourly quotas for action while loading dataset to HF Hub 🤗Datasets	9	1507	November 7, 2023

Any workaround for push_to_hub() limits?

Related topics