Hugging Face Forums
Slow processing with map when using deepspeed or fairscale
🤗Datasets
jasonyoun
June 25, 2021, 6:13am
11
I was also able to reproduce the result. Thanks for the prompt support
@stas
and
@sgugger
.
2 Likes
Caching a dataset with map() when loaded with from_dict()
show post in topic
Related topics
Topic
Replies
Views
Activity
Dataset map function takes forever to run!
🤗Datasets
16
6714
August 15, 2024
How does `datasets.Dataset.map` parallelize data?
Beginners
3
3114
August 5, 2024
Multiprocessing map taking too much memory footprint
🤗Datasets
17
5817
April 5, 2024
Datasets mapping slow down in the end
🤗Datasets
0
28
January 27, 2025
Using num_proc>1 in Dataset.map hangs
🤗Datasets
8
4011
August 19, 2024