How to Add New Data to an Existing Parquet Dataset?

John6666 · February 7, 2025, 7:58am

Features like incremental upload may still be in the works…

Incremental dataset (e.g. `.push_to_hub(..., append=True)`)

opened 03:18PM - 10 Oct 23 UTC

enhancement

### Feature request Have the possibility to do `ds.push_to_hub(..., append=True…)`. ### Motivation Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P). ### Your contribution What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`: 1. list files 2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005 3. delete files like parquet-0001-of-0004 4. generate + add last parquet file parquet-0005-of-0005 => make a single commit with all commit operations at once I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)

Topic		Replies	Views
Adding more data to the dataset uploaded on HF 🤗Datasets	4	341	January 15, 2024
Incrementally adding processed examples to a dataset 🤗Datasets	4	1281	June 23, 2022
Save `DatasetDict` to HuggingFace Hub 🤗Datasets	12	7080	October 20, 2023
Darshan Hiranandani : What methods are available for updating a saved dataset with new rows? 🤗Datasets	1	73	August 17, 2024
How to add a new column using only streaming dataset from remote? 🤗Datasets	3	21	March 6, 2025

How to Add New Data to an Existing Parquet Dataset?

Related topics