I want to use 2 ControlNets with SD1.5 on AWS Inferentia2 / Neuron. It seems to be supported but at runtime it fails with:
Loading only U-Net into both Neuron Cores...
Traceback (most recent call last):
File "test-controlnet.py", line 40, in <module>
image = pipe(
^^^^^
File .../optimum/neuron/pipelines/diffusers/pipeline_controlnet.py", line 352, in __call__
else controlnet.nets[0].config.global_pool_conditions
^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DataParallel' object has no attribute 'config'
Segmentation fault (core dumped)
I tried to fix that line to controlnet.config[0].global_pool_conditions but it still failed later on in the same module.
Any idea how to use 2 controlnets at the same time?
This is how I compiled the model for Neuron:
optimum-cli export neuron \
--inline-weights-neff \
--model jyoung105/stable-diffusion-v1-5 \
--task stable-diffusion \
--auto_cast=matmul --auto_cast_type=bf16 \
--batch_size 1 --num_images_per_prompt 1 \
--controlnet_ids \
lllyasviel/control_v11p_sd15_openpose \
lllyasviel/control_v11f1p_sd15_depth \
--height 512 --width 512 \
sd15-512x512-bf16-openpose+depth.neuron
And this is my test code:
from optimum.neuron import NeuronStableDiffusionControlNetPipeline
from diffusers import UniPCMultistepScheduler
from diffusers.utils import load_image
# OpenPose+Depth ControlNet
MODEL = "sd15-512x512-bf16-openpose+depth.neuron"
OPENPOSE_IMAGE = "openpose-512x512.png"
DEPTH_IMAGE = "depth-512x512.png"
# Load ControlNet images
images = []
for image in [OPENPOSE_IMAGE, DEPTH_IMAGE]:
print(f"Loading control image: {image}")
images.append(load_image(image))
pipe = NeuronStableDiffusionControlNetPipeline.from_pretrained(MODEL)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
prompt = "futuristic looking female"
image = pipe(prompt=prompt, image=images).images[0]
Am I doing something wrong or is the multi-controlnet support broken?
