[LoRA SD Fine-tuning] Should I be expecting this loss curve?

[neoshai]

Hi there,

I’m currently fine-tuning SD1.5 on my machine using the diffusers LoRA script (detailed here) on the Pokemon BLIP dataset.

However, my (smoothed) loss curve look like this:

Are these kinds of loss curves (which do not seem to decrease) par for the course with LoRA fine-tuning? What kind of loss curves should we expect?

Thank you in advance for the help!