I apply the tokenizer to my custom dataset using the datasets.Dataset.map() method as done in the run_mlm.py example. FYI, I am using multiprocessing by setting num_proc parameter of map(). I have a multi-GPU system, and doing the above usually takes about ~10 minutes. But once I use DeepSpeed (d…

Slow processing with map when using deepspeed or fairscale

sgugger June 24, 2021, 12:12pm 2

Maybe @stas will have an idea?

Topic		Replies	Views
Dataset map function takes forever to run! 🤗Datasets	16	6866	August 15, 2024
How does `datasets.Dataset.map` parallelize data? Beginners	3	3164	August 5, 2024
Multiprocessing map taking too much memory footprint 🤗Datasets	17	5900	April 5, 2024
Datasets mapping slow down in the end 🤗Datasets	0	33	January 27, 2025
Using num_proc>1 in Dataset.map hangs 🤗Datasets	8	4098	August 19, 2024