Multi_controlnet + inpaint

🧨 Diffusers
1

Hello dear devs. I am using the stable_diffusion_controlnet_inpaint.py (from community examples, main version) to generate a defective product with using initial image, masked image of the defect area, and two controlnet conditioning images. When I generated an image with same parameters as in SD Webui (enable_controlnetmodel_1, enable_controlnetmodel_2), the generated area of defect was totally different. My question is:

  1. Are these assignments (as per examples from website and github)
    controlnet = [ControlNetModel_1, ControlNetModel_2]
    pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(INPT_MODEL, controlnet=controlnet, ...)
    …
    pipe(... , controlnet_conditioning_image=[image_for_model1, image_for_model2]).images[0]
    are correct?

  2. I could not find more parameter setting when creating the pipeline such as, mask blur or inpaint area (whole picture or only masked), are these possible to bet set or they have not been implemented?

How could I make the script generated image similar to SD Webui’s? If you have any idea what might be the issue, would really appreciate your help!
P.S. I have read many implementation’s of controlnet + inpaint pipelines in github, I came here as a last resort.

1 Like
2

Hello @Novruz97, were you able to train/finetune this custom pipeline for your use-case?

3

@MeherShashwat In the new version of diffusers, this problem was solved. They included new pipeline in controlnet, you can have a check!

1 Like
4

Thanks @Novruz97, I saw the mentioned pipelines. I was asking about a training script that helps you finetune an inpainting stable diffusion model. The training scripts in the example folder of diffusers library doesn’t have any script that does the same (correct me if I am wrong!)
If you were able to finetune a inpainting stable diffusion model kindly share the script :slight_smile:

5

@MeherShashwat Unfortunately, I was given other project. I haven’t implemented that.

6

Hi, do you have any update/suggestion to share on that? I’m trying to do something alike what you described