Hello,
While fine-tuning my network, I would like to set up a polynomial learning rate scheduler by setting lr_scheduler_type="polynomial" and learning_rate=0.00005.
However, when I visualize the learning rate on the wandb dashboard, I’m observing a linear decrease of the learning rate instead of polynomial. (Screenshot below)
What might be causing the issue? I’ve tested without setting the learning_rate and the behavior was exactly the same.
Polynomial scheduler is not really supported via this argument, as it requires an additional power keyword argument that defaults to 1 and which can’t be set via this API. You should thus set the scheduler directly in the Trainer.
@sgugger I previously tested with lr_scheduler_type=“polynomial” and lr_scheduler_type="linear" to compare the outcome and I ended up getting different accuracy results, which makes me wonder - if the polynomial scheduler is not supported via this argument, shouldn’t I get exactly the same results?
Thank you for the info, I was able to set the scheduler directly in the Trainer.
Did you use the exact same seed as well?
Yes, the seed was set using set_seed(42).
