So when I tried with your dataset I did the same, checked if was missing any steps, but IDK why I got the same error.
in case of the segments dataset:
pixel_values looks like this : <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1920x1080 at 0x7FADD7C17F90>
label looks like: <PIL.PngImagePlugin.PngImageFile image mode=L size=1920x1080 at 0x7F54823057D0>
and my dataset looks like
pixel_values: <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=305x305 at 0x7FADEE32EF90>
label : <PIL.PngImagePlugin.PngImageFile image mode=L size=305x305 at 0x7FADEE0E9B90>
I am genuinely lost at what is going wrong, I have tried both declaring as a dataset dict as well as using a datasetloader
the dataset looks same hence I am getting the same error