Hi, I am trying to upload about 300GB of text. First I would like to do some processing on the text and then upload to the dataset for e.g I open a text file and make it clean and then upload it and then do this with the next file. How do i do this iteratively? To use .push_to_hub you need to have …

Uploading large datasets iteratively

316usman October 30, 2023, 7:20pm 5

Thank you so much I’ll try and get back to you

Topic		Replies	Views
Uploading a dataset that doesn't fit in memory to the HF hub 🤗Datasets	5	81	October 24, 2024
Streaming in dataset uploads 🤗Datasets	2	69	March 31, 2025
I had collected data for a language text for translation How can I add it up into datsets 🤗Datasets	7	1582	August 23, 2021
Incrementally adding processed examples to a dataset 🤗Datasets	4	1426	June 23, 2022
Unable to upload large audio dataset using push_to_hub 🤗Datasets	5	865	November 17, 2023