Imagefolder is flipping some of my images by 90 degrees

I am creating an ImageFolder dataset and setting it’s format to be later ‘torch’

I have noticed that after doing that some of my images are being rotated by 90 degrees to the left. I tried to add transform to fix that, but that transform is being override later by the format

def transform_batch(example):
    example["image"] = [ImageOps.exif_transpose(i.convert("RGB")) for i in example["image"]]
    return example

dataset = load_dataset(
            'imagefolder',
            data_dir=cfg.data.dataset_name_or_path)

# now the images comes up flipped 
dataset.set_transform(transform=transform_batch, output_all_columns=True)
## now the transform rotates them back

dataset.set_format('torch', columns=[...])


## Now getting back some of the tensors flipped

Do you maybe have an idea why the format flippes some of the images?

Thanks!