Hello, I have been working on recreating the OPT pre-training corpus to upload to the Huggingface dataset hub. I have been processing the Pile dataset into chunks and filtering the subsets to only include the specified sections such as CommonCrawl, DM Mathematics, etc. in the paper. I am working w…

Hi! We’ve recently added support for resuming an upload in push_to_hub to address this issue, so now you can just rerun the push_to_hub line after an error to resume the upload.

Push to Hub - HTTPS Connection Pool( host = ‘huggingface.co’, port = 443 )

🤗Datasets

conceptofmind June 30, 2022, 10:31pm 6

Hi @mariosasko ,

I appreciate the help.

I will try uploading again now.

Thank you,

Enrico

Topic		Replies	Views
HTTP 400 on push_to_hub with datasets 🤗Hub	2	1089	October 11, 2022
Unable to upload large audio dataset using push_to_hub 🤗Datasets	5	872	November 17, 2023
Problem pushing dataset to huggingface 🤗Datasets	11	3649	June 26, 2023
RuntimeError: Error while uploading 'data/train-00040-of-00157-15109dabc9b3967a.parquet' to the Hub 🤗Datasets	2	394	November 28, 2024
Cannot push dataset of 100k images 🤗Hub	0	589	March 26, 2023

Push to Hub - HTTPS Connection Pool( host = ‘huggingface.co’, port = 443 )

Related topics