FLUX.1-dev image generation on M1 Mac using GPU

Beginners
1

I want to make use of GPU on my M1 Mac to generate images through prompt using black-forest-labs/FLUX.1-dev. Adding the code below.

I’ve specified mps for torch.Generator but the script fails stating with error latents = torch.randn(shape, generator=generator, device=rand_device, dtype=dtype, layout=layout).to(device) RuntimeError: Placeholder storage has not been allocated on MPS device!

How can I make use of GPU to generate image?

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator("mps").manual_seed(0)
).images[0]
image.save("flux-dev.png")
1 Like
2

How about this?

import torch
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
prompt = "A cat holding a sign that says hello world"
pipe.to(device)
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator(device).manual_seed(0)
).images[0]
image.save("flux-dev.png")
3

Although now it picks GPU but still fails adding error below. I’ve added pipe.enable_model_cpu_offload() but still fails. Also I’ve executed PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 still fails

RuntimeError: MPS backend out of memory (MPS allocated: 36.21 GB, other allocations: 384.00 KB, max allowed: 36.27 GB). Tried to allocate 144.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
4

How about this?

pip install -U accelerate

import torch
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16, device_map="auto") # reweite it
pipe.enable_model_cpu_offload()
prompt = "A cat holding a sign that says hello world"
#pipe.to(device) # it causes OOM
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=1,
    max_sequence_length=512,
    generator=torch.Generator(device).manual_seed(0)
).images[0]
image.save("flux-dev.png")
5

there is syntax error in this code. pipe is referenced before assignment

6

Sorry, I failed to copy and paste.