I often use the batch mode of dataset.map to process large amounts of data. Since there may be some format problems in the data, some batches may fail in the map (while most batches are OK). Is there some way to ignore the failed batches and return the successful batches?

For example, how about just use Python Exception? [image] Saving outcomes if Error while applying map function on dataset 🤗Datasets I use an API (like huggingface_hub) to let a language model answer questions from my dataset. Since I want to send every single example to t…

Dataset.map Ignore failed batches

🤗Datasets

John6666 June 11, 2025, 11:39am 2

For example, how about just use Python Exception?

Topic		Replies	Views
Saving outcomes if Error while applying map function on dataset 🤗Datasets	2	1146	February 14, 2023
Huggingface Dataset.map shows red progress bar when batched=True 🤗Datasets	1	1175	October 24, 2022
Problems while filtering large datasets using `map` 🤗Datasets	2	660	July 21, 2021
Map function skipping rows (only 8k out of 1.6M rows) 🤗Datasets	1	197	December 25, 2023
Map() function doesn't process 🤗Datasets	2	1083	April 21, 2022

Dataset.map Ignore failed batches

Related topics