Hello,
I plan to release a dataset (Metacreation/GigaMIDI, currently gated) which needs to be loaded from a script.
For compatibility reasons, I want to the main branch to contain the data as shards of tar files (webdatasets) along with json files for the metadata that will be loaded by the script, like the voxpopuli dataset.
I can’t get the parquet converter bot to automatically create the parquet version, and the parquet conversion script from the cli doesn’t do what I want.
I tried to do this manually by converting the splits to parquet files and uploading them to the specific branch, but it’s impossible to create the refs/convert/parquet branch.
What can I do to make this possible? There is nothing in the docs to cover this use case.
How did you manage to create the parquet branches for the voxpopuli or mozilla-foundation/common_voice_11_0 datasets?