Semantic Correspondence Dataset

🤗Datasets
1

Hi! I would like to convert the semantic correspondence dataset SPair (SPair-71k: A Large-scale Benchmark for Semantic Correspondence) into a huggingface dataset. Each of the 71k rows would contain two images and metadata. However, an image can occur in more than one row, so naively saving two images per row would be a huge waste of disk space. How can I make sure the images are only saved once in the dataset but can be referenced multiple times?

Edit:
It seems like a custom dataset loading script might do the trick, which, however, requires users to run untrusted code. Is this also possible without a custom loading script, or is such a script the way to go?

2

Can confirm that this is possible with a custom dataset loading script, as I managed to create and upload the dataset: 0jl/SPair-71k · Datasets at Hugging Face

However, I’m still interested in whether or not this is possible without a custom loading script