Just wondering; does the learning rate decay in the train_text_to_image.py script? I’m resuming from a checkpoint quite a long way into training (360k steps) and I’m still seeing learning_rate=0.0001 in the progress indicator.
I always assumed that was just showing the initial learning rate, but is it supposed to reflect the actual learning rate at the current step (or epoch)? I’m only wondering because improvement in my outputs is extremely slow. I expected it to be slow, but it seems almost conspicuously slow, and if the learning rate isn’t decreasing (and is generally too high) that might explain it… it’s potentially bouncing around the minima.
The launch process does indicate: “All scheduler states loaded successfully”, so I’m assuming it’s resuming correctly from my checkpoint.
I’ve just started another run, for another 20 epochs, and added --learning_rate=1e-6 to my command, but the progress still indicates:
UPDATE: Digging in a bit more, comet_ml is showing the learning rate as 1e-6, which is what I set… so who’s right? It does also show lr_scheduler as “constant”, which surprised me. Does diffusion use a constant learning rate? Is that the best approach?
You can use other schedulers if you want and see if that helps. I believe that a decreasing learning rate is usually applied when training huge models with lots of data and steps.
Regarding you setting the learning rate to 1e-6 and not seeing it in the log, I’m not sure what might be going on. Do you think the script could be scaling it? (diffusers/train_text_to_image.py at main · huggingface/diffusers · GitHub). Otherwise, it could be a bug, please let us know and feel free to open an issue in GitHub
I haven’t run it again on the standard script since starting LoRA, so I’m not sure. I do know that I didn’t have --scale_lr set, so it shouldn’t have been scaling.
I can see about reproducing it when my GPUs free up again.
Just to be clear, it was only when resuming from a checkpoint that I seemed unable to change the learning rate, which I’m guessing might be due to the “constant” lr schedule. If I start from the pretrained model I can set the learning rate as expected.
Okay, just confirming: I cannot change the learning rate when resuming from a checkpoint. I don’t suspect this is actually a “bug”, but just a fact of using resuming with the constant scheduler. It does make sense that the “schedule” is basically fixed when you start a new run, so I’m not suggesting that this should be editable… It would just have been convenient in my case!
I’m going to save the pipeline and start again with the new learning rate, just to see how it progresses.
