Training controlNet on 8gb GPU

Hi, I’m trying to train a controlNet on the basic fill50k dataset (the controlnet example on the diffusers repo).
Using all the requirements provided in the example results in my model not converging.
Has anyone been able to train with those configurations?

accelerate launch
–validation_image “./conditioning_image_1.png” “./conditioning_image_2.png”
–validation_prompt “red circle with blue background” “cyan circle with brown floral background”
–mixed_precision fp16

Here are some results after 12000 steps:

Additionally, can anyone share a profile of the error as a function of the steps for controlNet?
Is the error gradually reducing or suddenly reducing?