Whether it’s a bug or intended behavior, when loading models from the pipeline class, adding use_safetensors=True sometimes causes problems. The reason this doesn’t happen with Unet2DModel is that the search range for files is slightly narrower.
You can simply delete use_safetensors=True, but updating Diffusers and related libraries to the latest versions may also resolve the issue.
pip install -U diffusers huggingface_hub[hf_xet]