Create a system to convert color, textures of furnitures

Beginners
1

As the title mentioned, I would like to create a system which could convert color/textures of furniture by

  1. text prompts.
  2. image with specific color/texture

For task 1, I guess Pix2Pix tutorial could help me achieve the goal. Or I should try another solution?

For task 2, this is quite similar to virtual try on, you give an original image and a target “cloth”, then ask the models to swap them. Do any training tool exist in the diffusers library for this task?

I’m still a novice in this field, so I hope you guys can give me some pointers. Please suggest me which method is worth trying, thank you

2

By the way, is it practical to create algorithm like fashion mirror doing? How many resource I need?

I check some papers, hard to say there exist defacto, standard solution yet, the methods keep evolving. Is it practical to train a diffusion model by diffusers for virtual try on? I have a naive idea, concatenate mask of the target cloth, original image, mask of the cloth and the pose of the human(something by openpose), then feed them into the diffusion network to finetune a model suit for virtual try on, problem is the network only accept 3 channels image I guess.

TryOnDiffusion looks cool, but no codes/model release, everything need to retrain from scratch + I don’t have 4 millions data to train.

Thanks, please forgive me if I asked something stupid.

3

any update on this?