Sharing ArrowDataset with subfolders

[John6666]

If it’s already been converted to a Dataset class, is datasets.concatenate_dataset sufficient…? @lhoestq

Issue concatenating datasets

I am trying to concatenate two datasets from datasets import load_dataset, concatenate_datasets movie = load_dataset("movie_rationales") imdb = load_dataset("imdb") imdb = imdb['train'] Then I adapt the movie dataset movie_imdb_format = movie['train'].map( lambda x: {"text": x["review"]} ) movie_imdb_format = movie_imdb_format.remove_columns(["review", "evidences"]) and aim to concatenate them dataset_cc = concatenate_datasets([imdb, movie_imdb_format]) These both datasets output Dat…

ArrowBasedBuilder versus GeneratorDBasedBuilder

Could you please enumerate pros and cons for both these dataset builder classes. I couldn’t find anything in the documentation. When would I prefer one over the other. Is ArrowBasedBuilder more performant for large datasets? Thank you!