Hi, everyone.
Hope you are doing well.
Now I wanna make project that similar to https://renovateai.app/.
If users upload house image and select material, change material according to the part.
But I’m not sure how should i do for that… which model should i use or what is best approach?
If anyone know about this, please let me know.
Thanks.
There are two main approaches: using a standard diffusion model or using a diffusion model designed for replacement, such as the VTON series. (In the case below, the target is clothing…)
When using a standard diffusion model, techniques such as Inpainting, IP-Adapter, or ControlNet would likely be employed.
Thanks, @John6666
But could you explain in more detail?
The overall process is the same for all cases. First, select the area of the object you want to replace. This can be done manually by the user. If you want to automate this step, use some kind of object detection model (there are many existing options such as YOLO or SAM).
Based on the selected area and the room image, create a mask image (a black-and-white image specifying the area to be redrawn) using image processing with Python’s OpenCV or similar tools.
When the room image and mask image are passed to the Diffusion model’s Inpaint pipeline, an image is output where only the masked area is redrawn as naturally as possible.
This is the basic flow.
This workflow assumes the use of a standard image generation model, so it is inefficient for the specific purpose, but it has the advantage of not requiring a dedicated model.
If a dedicated redrawing model for rooms exists, using that would be the simplest solution, but I am unsure if such a model exists.
I found an explanation by Diffusers’s developer.