Hi all, I am trying to use filter/map on a dataset in a script, and I am finding that the script hangs upon completion of the filter/map operations (the tqdm progress bar always goes up to 100%) I am calling filter like my_dataset = my_dataset.filter(lambda example: example['image_id'] not in some_…

Datasets filter/map hangs when multithreading

dlrudwo1269 April 18, 2023, 1:30pm 2

I would also like to add that this is not reproducible 100% of the time… (it worked in the first few runs which made me believe there was no issue)

Topic		Replies	Views
Using num_proc>1 in Dataset.map hangs 🤗Datasets	8	4094	August 19, 2024
Progress bar of dataset.map with num_proc>1 hangs 🤗Datasets	2	1313	December 6, 2023
Dataset.map hangs on tokenization (relatively small dataset) 🤗Datasets	2	1998	April 22, 2022
Debugging parallel Datasets transformations 🤗Datasets	3	2089	December 17, 2022
Dataset map function takes forever to run! 🤗Datasets	16	6861	August 15, 2024