I’m using the datasets library to load gigant/oldbookillustrations . It’s stored in parquet files, with a couple Image fields plus a handful of other short text fields with assorted metadata. It loads okay, and many records seem to work just fine, but when I try to filter or map over the whole thing…

You can disable decoding and apply your own transform to decode the images: def decode_images(batch): batch["rawscan"] = [decode_image(raw_data) for raw_data in batch["rawscan"]] return batch ds = ds.cast_column("rawscan", Image(decode=False) ds = ds.with_transform(decode_images) ds[0]["raw…

Handling decoding errors such as UnidentifiedImageError

🤗Datasets

John6666 February 5, 2025, 2:44pm 10

I found a similar case.

Topic		Replies	Views
Handle errors when loading images (404, corrupted, etc) 🤗Datasets	4	825	August 17, 2023
PIL.UnidentifiedImageError: cannot identify image file 🤗Datasets	4	8402	March 3, 2023
Issues in loading image from dataset Beginners	3	1167	January 22, 2024
Handling non-existing url in image dataset while cast_column 🤗Datasets	2	420	January 16, 2024
Unable to load images 🤗Datasets	2	152	December 31, 2024

Handling decoding errors such as UnidentifiedImageError

Related topics