Linear learning rate despite lr_scheduler_type="polynomial"


While fine-tuning my network, I would like to set up a polynomial learning rate scheduler by setting lr_scheduler_type="polynomial" and learning_rate=0.00005.

However, when I visualize the learning rate on the wandb dashboard, I’m observing a linear decrease of the learning rate instead of polynomial. (Screenshot below)

What might be causing the issue? I’ve tested without setting the learning_rate and the behavior was exactly the same.

Polynomial scheduler is not really supported via this argument, as it requires an additional power keyword argument that defaults to 1 and which can’t be set via this API. You should thus set the scheduler directly in the Trainer.

@sgugger I previously tested with lr_scheduler_type=“polynomial” and lr_scheduler_type="linear" to compare the outcome and I ended up getting different accuracy results, which makes me wonder - if the polynomial scheduler is not supported via this argument, shouldn’t I get exactly the same results?

Thank you for the info, I was able to set the scheduler directly in the Trainer.

Did you use the exact same seed as well?

Yes, the seed was set using set_seed(42).