I’m trying to process audio data faster (librispeech) using multi-processing. Serially everything works, but when I try num_proc>1 in the .map function the progress bar just sits at 0. Reproducer: from datasets import load_dataset from transformers import AutoFeatureExtractor feature_extractor = …

Tried in main and it still hangs. However I found the the real reason for the hang seems to be torch threads. Setting torch.set_num_threads(1) made it work. :smiling_face:

Using num_proc>1 in Dataset.map hangs

🤗Datasets

jimypbr June 23, 2023, 12:37pm 2

Could related to this: Load_dataset hangs with local files - #4 by lhoestq
I’m doing this inside an IPython session…

Topic		Replies	Views
Datasets map keeps hanging Beginners	0	79	April 7, 2024
Progress bar of dataset.map with num_proc>1 hangs 🤗Datasets	2	414	December 6, 2023
Num_proc is not working with map Beginners	5	912	April 15, 2024
Dataset.map hangs on tokenization (relatively small dataset) 🤗Datasets	2	1617	April 22, 2022
Dataset map function takes forever to run! 🤗Datasets	11	3011	January 29, 2024

Using num_proc>1 in Dataset.map hangs

Related Topics