I would like to use multiple workers and interleave datasets with iterable datasets. What is the best way to do this? I gather from this PR Implement sharding on merged iterable datasets by Hubert-Bonisseur · Pull Request #5735 · huggingface/datasets · GitHub that the individual datasets should be …

Sharding depends on the dataset, one shard corresponds to one file. Sharding is quite important to enable the use of num_workers in in data loaders, so feel free to use datasets that are already sharded or shard a dataset your self, e.g. using: from datasets import load_dataset ds = load_dataset(…

Correct way to use multiple workers with interleave_datasets for iterable datasets

🤗Datasets

system Closed July 4, 2024, 1:08pm 3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Interleaving Iterable Dataset with num_workers > 0 🤗Datasets	3	1599	April 11, 2023
Creating sharded IterableDataset from a list of IterableDatasets? Intermediate	2	590	July 2, 2024
Num_worker with IterableDataset 🤗Datasets	4	2841	November 16, 2023
Homogeneous batches from list of IterableDatasets 🤗Datasets	6	74	October 23, 2024
Making an infinite IterableDataset 🤗Datasets	6	137	March 19, 2025

Correct way to use multiple workers with interleave_datasets for iterable datasets

Related topics