Hi All, I have been struggling to make the map tokenization parallel, however, I couldn’t make it. I request, could you please suggest me in this regard. Here is the example code. training_dataset = dataset.map( lambda example, idx: tokenize( example, idx, vocab, in_df.columns, decoder_dat…

Num_proc is not working with map

G999n August 23, 2025, 3:15pm 7

Thanks a lot bud! I was stuck in it for days. I reset password and logged in just to thank you. Haha

1 Like

Topic		Replies	Views
Tokenizer dataset is very slow 🤗Tokenizers	3	4408	March 2, 2024
Dataset map function takes forever to run! 🤗Datasets	16	6881	August 15, 2024
AutoTokenizer.encode with multiThread and mutliProcess 🤗Tokenizers	2	340	October 9, 2024
When using Dataset.map to tokenize a dataset, the speed slows down as the progress approaches 100% 🤗Datasets	3	952	December 23, 2024
Dataset.map hangs on tokenization (relatively small dataset) 🤗Datasets	2	1998	April 22, 2022