Hi all, I’m working on releasing a dataset of around 1M images of galaxies labelled by volunteers ( www.galaxyzoo.org ). I’m trying to understand the pros and cons of: Storing the dataset on the hub as either webdataset .tar or HF ‘standard’ .parquet shards Loading the data with either the webdatas…

Loading webdatasets across multiple nodes

Alejandro98 April 21, 2025, 10:04pm 4

Really interesting thread. However, the question ‘Loading WebDatasets across multiple nodes’ isn’t fully answered. Is there an example that can be used as a reference?

1 Like

Topic		Replies	Views
Use load dataset to load a sample of the dataset 🤗Datasets	3	1277	May 24, 2021
Using Webdatasets to stream data 🤗Datasets	6	1923	February 19, 2024
Allow streaming of large datasets with image/audio 🤗Datasets	18	3979	May 30, 2022
Roadmap/timeline for dataset streaming 🤗Datasets	9	2280	July 5, 2021
Stream Audio Dataset that Can't be moved to Hub 🤗Datasets	7	494	March 17, 2023

Loading webdatasets across multiple nodes

Related topics