Parquet-bot converted a parquet file into a bigger parquet chunk

馃Datasets
1

Here is my dataset: BaiqingL/pokemon-showdown-rag-prompts 路 Datasets at Hugging Face
To keep the size small, I used brotil compression and ended up with a file blob of around 242MB.

Parquet converter bot seems to have made 2 new branches, with each branch the size significantly larger than the file I originally had.

What is causing this behavior? Seems quite odd.

2

cc @lhoestq

I think it鈥檚 due to the row group size:

When the dataset is already in Parquet format, the data are not converted and the files in refs/convert/parquet are links to the original files. This rule suffers an exception to ensure the dataset viewer API to stay fast: if the row group size of the original Parquet files is too big, new Parquet files are generated.

3

I see, guessing the behavior here is to render it in chunks so it鈥檚 easier on the frontend.