Hyperparameters for lr_scheduler_type in Trainer Arguments

Beginners
1

My understanding is that I can pass the following string values to use the corresponding learning rate schedulers in Training Arguments:

  • “linear” → transformers.get_linear_schedule_with_warmup
  • “cosine” → transformers.get_cosine_schedule_with_warmup
  • “cosine_with_restarts” -->transformers.get_cosine_with_hard_restarts_schedule_with_warmup
  • “polynomial” → transformers.get_polynomial_decay_schedule_with_warmup
  • “constant” → transformers.get_constant_schedule
  • “constant_with_warmup” → transformers.get_constant_schedule_with_warmup
  • “inverse_sqrt” → transformers.get_inverse_sqrt_schedule

However, some of the schedulers have specific hyperparameters. For example, transformers.get_cosine_schedule_with_warmup has num_cycels as a parameter. How can we change the default values?

1 Like
2

Hi, there is a another argument called lr_scheduler_kwargs, you can change the hyperparameters in this with dict.

3

which version have the argument called lr_scheduler_kwargs. i used transformers 4.28.0