Load Dataset and Save as Parquet

🤗Datasets
1

I am curious whether we can directly save a dataset as parquet as in the original dataset at HF.

For example fineweb-edu/sample/10BT has 13 parquet files

# download the dataset
fw = load_dataset(
    "HuggingFaceFW/fineweb-edu",
    name="sample-10BT,
    split="train",
    num_proc=10,
    cache_dir=cache_dir,
)


fw.to_parquet(parquet_dir)

Loads these files as arrow although they are originally parquet, and when saving with to_parquet() it converts arrow to a single parquet file.

So my questions are:
How can i download/load and save them with the same structure in the repository?
How can i create multiple parquet files instead of single big parquet file with to_parquet?

2

Hi ! You can download a repository (and specify a subset of files) using the huggingface-cli :slight_smile:

Also right now the datasets lib doesn’t support writing to multiple Parquet files yet, though it would be amazing to contribute this feature

Maybe something like this ?

fw.to_parquet("path/to/parquet/", max_shard_size="500MB")
1 Like
3

By any chance was this made into a feature request or perhaps even implemented? I agree that this would be a great feature

1 Like
4

there is an open issue here: Save Dataset as Sharded Parquet · Issue #7047 · huggingface/datasets · GitHub

1 Like