🧵 Need help creating flat texture image from button photos (AI or OpenCV approach)

Beginners
1

Hi everyone :waving_hand:

I’m working on a project where I need to generate a flat image from a button photo — basically an image with the same color and pattern as the button, but without any holes, lighting, or shadows.

Take a photo of a button (which has holes and curved reflections), and create a flat version of it where the area inside the holes is intelligently filled with matching pattern and color.

:bullseye: Goal

Take an image like this :backhand_index_pointing_down:

images (1)
images (1)225×225 4.7 KB

png-transparent-black-cloth-button-button-t-shirt-clothing-button-tshirt-cloth-necktie-thumbnail
png-transparent-black-cloth-button-button-t-shirt-clothing-button-tshirt-cloth-necktie-thumbnail360×360 31.8 KB

images
images271×186 4.77 KB

Basically — remove the lighting, fill holes intelligently, and produce a seamless, flat surface with the same color/pattern as the original button.

:brain: What I’ve tried so far

  • OpenCV Inpainting:
    Works for simple fills but gives blurry results and wrong color tones.

  • Hugging Face Stable Diffusion Inpainting:
    Tried the runwayml/stable-diffusion-inpainting pipeline with mask, but it generated random textures or background instead of the real pattern.

  • Cropping + Tiling:
    Works only if the button color is solid, but fails when there’s a gradient or marble-like pattern.

:light_bulb: What I’m looking for

  1. Is there any AI model or method that can intelligently generate a flat texture from a photo (preserving the same color/pattern)?

  2. Should I use Diffusers, ControlNet, or Texture Synthesis models for this?

  3. Any tips, examples, or notebooks that can help me achieve better results?

:puzzle_piece: My Setup

  • Language: Python

  • Libraries: OpenCV, Diffusers

  • System: macOS M1

  • Torch Backend: MPS (Metal)

Example Code

Here’s a simplified version of my inpainting attempt:

from diffusers import StableDiffusionInpaintPipeline
from PIL import Image
import torch

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("mps")

result = pipe(
    prompt="fill the button hole with same color and make it flat texture, realistic fabric color, consistent pattern",
    image=Image.open("button.png"),
    mask_image=Image.open("mask.png")
).images[0]

result.save("flat_button.png")

:framed_picture: Images

I’ve attached a couple of real button photos for reference.
They have small holes and lighting variations, which is making it tricky to create a clean flat texture.

:heart: Any guidance appreciated

If there’s a specific model, prompt strategy, or approach I should test, I’d love to hear your thoughts.
This is part of a real-world project where consistent textures are crucial for product visualization.

Thanks in advance for your time and help! :folded_hands:

1 Like
2

LaMa and Intrinsic Image Decomposition seem promising options in this case, but getting either to run with the current mainstream Python and library versions looks like a lot of work. If I find a better approach, I’ll write about it again.

3

Thank you for the response, John. I would appreciate if there is any code is available, any language would work.

1 Like
4

Marigold Computer Vision architecture recently made available for Diffusers seems capable of performing albedo extraction. The sample image I’m attaching isn’t very good yet, but well… it does seem to be extracting…?

albedo
albedo225×225 33.6 KB

5

Thanks, John.

Seems to be extracting the image color.

I will test the code in different cases.

patterned buttons images and will keep posted the result. In additional to that If you get any other way of doing kindly keep posting here.

Any paid services would work as well for the best result. I am exploring Vertex AI: Imagen 3, Image Generation, Editing (In-painting) from Google service provide as well.

This is very critical part of the solution where visual is very important part of user experience. I am exploring them and lets see which one can provide the best result for the end users.

1 Like
6

With Marigold or similar models, it seems high-frequency components like wood grain patterns inevitably get stripped out. While there are methods to extract and re-inject them via image processing, if you want to do it within a single pipeline, this is about the only solution I’ve found…

7

We can try this solution and see the results, John. I tested the previous code with different cases and found inconsistent results.

I updated the ort_session function to force CPUExecutionProvider instead of CoreML to avoid permission errors on macOS (M1). After testing, it runs successfully without CoreML issues.

The source images are in test-results (see source images.txt for links used), and the expected results from Gimini are in the expectations folder (sample button images prefixed with expectation_).

Note that CPU usage was quite high and result was inconsistent. I’m continuing to research to get more consistent results or explore alternative solutions.

Currently I am experimenting Google Inpainting in Vertex AI. Any other AI based solution would be welcome as well.

1 Like