Sharing ArrowDataset with subfolders

[samirchar]

Hello everyone!

I want to share multiple datasets in the same repo <my_username>/<my_repo_name>, each in its own folder. The datasets in each folder are already in sharded Arrow format (for best performance) and contain different splits, as usual. To read any of these datasets with load_dataset I would need a loading script to tell HF how to read from the folders, right? If so, should I use the ArrowBasedBuilder and how? I only see tutorials for GeneratorBaseBuilder!

Thanks!

[John6666]

If it’s already been converted to a Dataset class, is datasets.concatenate_dataset sufficient…? @lhoestq

Issue concatenating datasets

I am trying to concatenate two datasets from datasets import load_dataset, concatenate_datasets movie = load_dataset("movie_rationales") imdb = load_dataset("imdb") imdb = imdb['train'] Then I adapt the movie dataset movie_imdb_format = movie['train'].map( lambda x: {"text": x["review"]} ) movie_imdb_format = movie_imdb_format.remove_columns(["review", "evidences"]) and aim to concatenate them dataset_cc = concatenate_datasets([imdb, movie_imdb_format]) These both datasets output Dat…

ArrowBasedBuilder versus GeneratorDBasedBuilder

Could you please enumerate pros and cons for both these dataset builder classes. I couldn’t find anything in the documentation. When would I prefer one over the other. Is ArrowBasedBuilder more performant for large datasets? Thank you!

[samirchar]

@John6666 no because i dont want to concateneate the datasets! Each folder is a different dataset with different features. So do i need the arrow builder to tell HF how to load the different datasets from the subfolder?

[John6666]

Hmm…
In that case, I thought that it would be easier for Hugging Face, which is based on one model per repo (and dataset), to work properly if the datasets with different structures were kept separate.
However, I think there was a way to merge datasets with different structures. Let’s wait for lhonestq.

[samirchar]

Yeah, maybe. I’m hesitating to separate into different repos because the datasets are related. It’s not completely separate projects. Think of it as GLUE, which is a set of multiple datasets but they are all related to one objective or project, like shown here Create a dataset loading script

[lhoestq]

You can configure the subsets present in your dataset repository in YAML see the docs at Manual Configuration

See the GLUE dataset for example: nyu-mll/glue at main

[John6666]

Thank you!

[samirchar]

This is amazing! Thank you very much.