Overflow error: cannot fit 'int' into an index-sized integer For SDXL

I am trying to Run SDXL based on the code given in the model card:

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")

# if using torch < 2.0
# pipe.enable_xformers_memory_efficient_attention()

prompt = "An astronaut riding a green horse"

images = pipe(prompt=prompt).images[0]

But I am getting the following Error:

encoded_inputs["attention_mask"] = encoded_inputs["attention_mask"] * [0] * difference

Overflow error: cannot fit  'int' into an index-sized integer

System Info:

Python:3.10
GPU: A100
Torch:2.3.1 + cu12.1