Dataset.map Ignore failed batches

[whh]

I often use the batch mode of dataset.map to process large amounts of data. Since there may be some format problems in the data, some batches may fail in the map (while most batches are OK).

Is there some way to ignore the failed batches and return the successful batches?

[John6666]

For example, how about just use Python Exception?

Saving outcomes if Error while applying map function on dataset

I use an API (like huggingface_hub) to let a language model answer questions from my dataset. Since I want to send every single example to the language model, I wrote a function that does that and then use the map function to map this API call to every example of my dataset. My issue is: If there is an Error at any point (e.g. the API throws an Error after one hour, because sth happend) I loose all the information. Lets say the map worked for the first 100 examples and then the API throws an E…

[whh]

Thanks, It’s helpful !