Load Pascal VOC with different configurations from s3

🤗Datasets
1

Hello everybody,

I would like to use s3 as storage for a dataset with different configurations. A good example is Pascal VOC dataset here fuliucansheng/pascal_voc at main
The underlying data of all configurations is the same and main difference is how _generate_examples function is executed. Local caching works seems to work well and reuse the images under different configs.

Now I would like to be able to specify dataset + configuration and load it directly from s3. What is the best way to do that?

I tried

dataset.save_to_disk("s3:/xxx/xxx", storage_options=storage_options)
dataset = load_from_disk("s3:/xxx/xxx", storage_options=storage_options)

but that works for a single configuration apparently.

2

Hmm it seems that what I am asking for is basically what hf hub offers and could be achieved by creating a version of the class here datasets/src/datasets/load.py at main · huggingface/datasets · GitHub but for s3.