SD XL Multi ControlNet Inpainting in diffusers

🧨 Diffusers
1

Our current pipeline uses multi controlnet with canny and inpaint and use the controlnetinpaint pipeline

Is the inpaint control net checkpoint available for SD XL?

Reference Code:

        controlnet_inpaint_model = ControlNetModel.from_pretrained(
            CONTROLNET_INPAINT_MODEL_ID, torch_dtype=torch.float16, cache_dir=DIFFUSION_CHECKPOINTS_PATH).to(
            MODEL_DEVICE)
        controlnet_hed_model = ControlNetModel.from_pretrained(
            CONTROLNET_HED_MODEL_ID, torch_dtype=torch.float16, cache_dir=DIFFUSION_CHECKPOINTS_PATH).to(
            MODEL_DEVICE)
        controlnet_canny_model = ControlNetModel.from_pretrained(
            CONTROLNET_CANNY_MODEL_ID, torch_dtype=torch.float16, cache_dir=DIFFUSION_CHECKPOINTS_PATH).to(
            MODEL_DEVICE)
        controlnet_depth_model = ControlNetModel.from_pretrained(
            CONTROLNET_DEPTH_MODEL_ID, torch_dtype=torch.float16, cache_dir=DIFFUSION_CHECKPOINTS_PATH).to(
            MODEL_DEVICE)

        self.inpaint_pipeline = StableDiffusionControlNetInpaintPipeline.from_pretrained(
            CUSTOM_SD_MODEL,
            controlnet=[controlnet_inpaint_model, controlnet_hed_model],
            torch_dtype=torch.float16).to(MODEL_DEVICE)
2 Likes