Issue when trying to switch between different LoRa adapters

🧨 Diffusers
1

I am trying to create a StableDiffusion XL pipeline and I am trying to provide different LoRa adapters and allow users to switch between them with a button. I am using the commands pipe.set_adapters() to change the active LoRa (as explained in this article), but it behaves strangely. Sometimes it works and sometimes it doesn’t and the previous adapter remains active. Any idea on how could I allow to consistently switch between LoRas? Thanks!
Here is the code: app.py · ar0551/ArchitecturalRendering_SD-XL-1.0 at main

1 Like
2

LoRA has some quirks. In particular, there were some problems with the previous version of Diffusers when using multiple LoRA.
There are a few points you should generally be aware of.

  • The behavior changes depending on whether low_cpu_mem_usage=True/False is set.
  • The behavior may change depending on whether LoRA is applied on the GPU or CPU.
  • Quantization often prevents LoRA from being applied successfully.
  • fuse_lora() will definitely apply LoRA, but unfuse_lora() may does not undo the changes, so it is not recommended for your use case.
  • Don’t forget to unload the previous LoRA before loading a new one.
  • If you are applying multiple LoRA, try to apply them all at once.
  • Monitor the currently active Adapter frequently.
  • In the case of web services used by multiple people, such as a shared space, the pipeline or model itself may be shared, so consider the risk of multiple users’ LoRA being mixed together. Basically, it should be safe to unload everything and reapply everything for each request.
1 Like