If I have multiple serialized datasets then is it a good practice to use multiprocessing along with datasets.load_dataset()? Something like so: import multiprocessing import datasets import gcsfs BUCKET_NAME = "my-bucket" GCS_FS = gcsfs.GCSFileSystem() def load_ds(ds_path): return datasets…

Loading multiple serialized datasets with `multiprocessing`

carted-ml April 2, 2022, 5:30am 3

By "does not execute as expected”, I meant it does not execute at all.

Topic		Replies	Views
Load_dataset hangs with local files 🤗Datasets	6	4347	January 3, 2023
Multiprocessing and sharding when creating dataset from scratch using loading script 🤗Datasets	2	1637	November 4, 2022
DatasetDict save_to_disk with num_proc > 1 seems to hang with error 🤗Datasets	2	379	March 4, 2024
Streaming dataset freezes with multi-gpu 🤗Transformers	2	1695	December 8, 2022
Load shards as one dataset 🤗Datasets	0	186	February 16, 2024