Object Removal in the the new Inpainting SDXL model?

🧨 Diffusers
1

Hi Everyone!

I just started using it and the results are great! However, it has one big issue: It’s hard to remove a big object out of the input. Let’s say I want to empty a room, instead, it will keep throwing random asterisks into the input.

Any ideas how to implement or setting the parameters to achieve this?

EditPreview

2

I suppose it’s because a totally empty room might not be the most realistic thing, according to SD.

Here are two solutions I would try :

  • Do it iteratively. Remove big objects, select newly created small objects (maybe using Segment Anything) inpainting again, rinse and repeat
  • In case the problem is specific to SDXL, do the first inpainting with SD2, and do the second inpainting on the result with SDXL but with a strength slightly lower than 1 (let’s say 0.99)

The second solution worked well for me, it’s probably a temporary patch, because SDXL inpainting apparently has some problems with strength set to 1 (see diffusers/stable-diffusion-xl-1.0-inpainting-0.1 · Hugging Face ) which would indicate more a bug soon to be solved than a real limitation.