I apply the tokenizer to my custom dataset using the datasets.Dataset.map() method as done in the run_mlm.py example. FYI, I am using multiprocessing by setting num_proc parameter of map(). I have a multi-GPU system, and doing the above usually takes about ~10 minutes. But once I use DeepSpeed (d…

Slow processing with map when using deepspeed or fairscale

jasonyoun June 25, 2021, 6:13am 11

I was also able to reproduce the result. Thanks for the prompt support @stas and @sgugger.

2 Likes

Caching a dataset with map() when loaded with from_dict()

Topic		Replies	Views
Dataset map function takes forever to run! 🤗Datasets	16	6611	August 15, 2024
When using Dataset.map to tokenize a dataset, the speed slows down as the progress approaches 100% 🤗Datasets	3	885	December 23, 2024
Tokenizer dataset is very slow 🤗Tokenizers	3	4305	March 2, 2024
Datasets mapping slow down in the end 🤗Datasets	0	26	January 27, 2025
Processing Large Dataset for Training GPT2 model 🤗Datasets	4	1146	April 12, 2023