Dataset.map Ignore failed batches

[John6666]

For example, how about just use Python Exception?

Saving outcomes if Error while applying map function on dataset

I use an API (like huggingface_hub) to let a language model answer questions from my dataset. Since I want to send every single example to the language model, I wrote a function that does that and then use the map function to map this API call to every example of my dataset. My issue is: If there is an Error at any point (e.g. the API throws an Error after one hour, because sth happend) I loose all the information. Lets say the map worked for the first 100 examples and then the API throws an E…