In diffusers documentation there are several ways to load and use Lora models with SDXL but it starts to get quite confusing what is the correct way to do it
I have trained a lora model on SDXL model using Kohya_ss
Now I want to load it form the .safetensors file in the pipeline and generate an image with lora scale=1.
There are these 2 links
What is confusing is when we load lora in the pipeline is :
- do I need to use PEFT and set the adapters?
- do I need to fuse_lora() in the pipeline and specify the lora_scale?
- are these mutually exclusive or doing both won’t affect the result?
What is confusing when generating the image is:
- do I need to format the prompt as in auto1111 adding
<lora:model_name-step:lora_scale>to it or just use the token+class ? - do I need to pass
cross_attention_kwargs={"scale": lora_scale}when generating the image?
Anyone can clarify what are the correct steps?
thank you