Loading an image to SAG pipeline

🧨 Diffusers
1

I realize that this is a deprecated pipeline, but I still want to give it a try in refining an existing image. Does anyone know how I can pass an image to StableDiffusionSAGPipeline?

I can tell you what I’ve tried, but I am keeping the question simple.

How can I pass an existing image (latent or minimally processed) to StableDiffusionSAGPipeline?

1 Like
2

Using an IP adapter is relatively easy, but otherwise I think you will need to modify the pipeline.

# pip install -U diffusers transformers accelerate torch torchvision safetensors
import torch
from diffusers import StableDiffusionSAGPipeline
from diffusers.utils import load_image

device = "cuda" if torch.cuda.is_available() else "cpu"

# SD 1.5 + SAG
pipe = StableDiffusionSAGPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    use_safetensors=True,
).to(device)

# Load IP-Adapter for SD 1.5 (repo subfolder = "models")
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="models",
    weight_name="ip-adapter_sd15.bin", 
)
pipe.set_ip_adapter_scale(0.6)  # 0=text only, 1=image only

# URL reference image
ref_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/load_neg_embed.png"
ref_img = load_image(ref_url)

# Generate
g = torch.Generator(device="cpu").manual_seed(42)
image = pipe(
    prompt="bear eats pizza, photorealistic, detailed",
    negative_prompt="low quality, watermark, text",
    ip_adapter_image=ref_img,   # image conditioning
    guidance_scale=1.0,         # low CFG pairs well with SAG
    sag_scale=0.75,             # SAG strength
    num_inference_steps=40,
    generator=g,
).images[0]

image.save("out_sag_ip_adapter.png")
print("saved out_sag_ip_adapter.png")
3

You are a Boss!

I got something from the model, not what I wanted, but something. Most importantly, I learned a new line. Thanks!

1 Like