Refiner SD-XL-1.0 is degraded latent of base model

Intermediate
1

The below code is related to controlnet SD-XL-1.0-refiner+base:
It’s generates image, but refiner model is degraded the latent of base model, why?

ontrolnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0",
    torch_dtype=torch.float16
)

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    torch_dtype=torch.float16,
)
pipe.enable_xformers_memory_efficient_attention()
pipe.to("cuda")

refiner = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    vae=pipe.vae,
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)

refiner.enable_xformers_memory_efficient_attention()
refiner.to("cuda")

prompt = "some things"

negative_prompt = """ some things"""

image = load_image()

image = np.array(image)
image = cv2.Canny(image, 75, 150)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
image_org = Image.fromarray(image)

n_steps = 50
high_noise_frac = 0.95
controlnet_conditioning_scale = 0.5  # recommended for good generalization


for i in range(10):
     image = pipe(
     prompt, negative_prompt=negative_prompt, image=image_org, controlnet_conditioning_scale=controlnet_conditioning_scale,
     num_inference_steps=n_steps, 
     output_type="latent"
     ).images

     image = refiner(
     prompt=prompt,
     negative_prompt=negative_prompt,
     num_inference_steps=n_steps,
     denoising_start=high_noise_frac,
     # generator = torch.Generator(device="cuda").manual_seed(12345),
     image=image,
     ).images

     image[0].save()
2

I am also seeing this, did you figure it out?