I apply the tokenizer to my custom dataset using the datasets.Dataset.map() method as done in the run_mlm.py example. FYI, I am using multiprocessing by setting num_proc parameter of map(). I have a multi-GPU system, and doing the above usually takes about ~10 minutes. But once I use DeepSpeed (d…

Slow processing with map when using deepspeed or fairscale

jasonyoun June 25, 2021, 6:13am 11

I was also able to reproduce the result. Thanks for the prompt support @stas and @sgugger.

2 Likes

Caching a dataset with map() when loaded with from_dict()

Topic		Replies	Views
Clm repeats tokenization when distributed Intermediate	5	1162	July 15, 2022
Multiprocessing map taking too much memory footprint 🤗Datasets	17	3514	April 5, 2024
About the DeepSpeed category DeepSpeed	1	700	October 30, 2021
More processes than GPUs with DeepSpeed launcher DeepSpeed	0	114	January 25, 2024
Infrence time increase when using multi-GPU DeepSpeed	1	758	November 28, 2023