Datasets.map is not consistent with IterableDataset?

[John6666]

I think that’s probably normal…
https://stackoverflow.com/questions/74251282/huggingface-datasets-storing-and-loading-image-data

Turn of automatic Pil image generation in load_dataset

when loading a dataset, I have the following arrow format - label: int, image: struct<bytes:binary, path:string>. when using load_dataset() method, the image is automatically converted to Pil image format, and the path is lost. is there a way to avoid that behavior? In my example - dataset = load_dataset("chronopt-research/cropped-vggface2-224") for i in range(0, len(dataset['train']), batch_size): batch = dataset['train'][i:i + batch_size] images = batch['image'] # Original 224x224 i…

How to return PIL.Image from dataset generator?

I have a custom GeneratorBasedBuilder subclass that I’m trying to get running. It seems like I’m almost there, but my trainer is failing with a type error when trying to access a generated sample—to be more specific, I’m working with the train_text_to_image.py script and it’s failing at the preprocess_train(examples) function… From what I understand, this is because the actual data is a PIL.Image, but my generator’s features dict lists the features as "image": datasets.Value("string"), which is…