How to use lr_scheduler_kwargs param in TrainingArguments?

luohuashijieyoufengj · June 24, 2025, 9:12am

The params of consine scheduler are as below in the documetation:

and I use params in TrainingArguments as below:

But it shows an error:

So here are my questions:

Why this error occurs?
If I need assign optimizer param in get_cosine_schedule_with_warmup?

John6666 · June 24, 2025, 2:10pm

1

This error often occurs when the contents of kwargs (or any valid name. a simple Python dictionary that stores keyword arguments to be passed later) conflict with arguments that are passed directly. It can often be avoided by simply omitting the argument specification.

https://stackoverflow.com/questions/28336270/how-to-fix-got-multiple-values-for-argument-error-for-args-and-kwargs

2

Maybe yes.

luohuashijieyoufengj · June 25, 2025, 4:25am

If I omit the argument, how can I assign it to my prefer value?

John6666 · June 25, 2025, 5:00am

You may need to specify it with a command line argument…

github.com/huggingface/transformers

examples/pytorch/translation/run_translation_no_trainer.py

f2fbe4475


      
              help="Number of updates steps to accumulate before performing a backward/update pass.",
          )
          parser.add_argument(
              "--lr_scheduler_type",
              type=SchedulerType,
              default="linear",
              help="The scheduler type to use.",
              choices=["linear", "cosine", "cosine_with_restarts", "polynomial", "constant", "constant_with_warmup"],
          )
          parser.add_argument(
              "--num_warmup_steps", type=int, default=0, help="Number of steps for the warmup in the lr scheduler."
          )
          parser.add_argument("--output_dir", type=str, default=None, help="Where to store the final model.")
          parser.add_argument("--seed", type=int, default=None, help="A seed for reproducible training.")
          parser.add_argument(
              "--model_type",
              type=str,
              default=None,
              help="Model type to use if training from scratch.",
              choices=MODEL_TYPES,
          )

github.com/huggingface/transformers

examples/pytorch/translation/run_translation_no_trainer.py

f2fbe4475


      
          ]
          optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate)
          
          # Scheduler and math around the number of training steps.
          overrode_max_train_steps = False
          num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
          if args.max_train_steps is None:
              args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
              overrode_max_train_steps = True
          
          lr_scheduler = get_scheduler(
              name=args.lr_scheduler_type,
              optimizer=optimizer,
              num_warmup_steps=args.num_warmup_steps,
              num_training_steps=args.max_train_steps,
          )
          
          # Prepare everything with our `accelerator`.
          model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
              model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
          )

luohuashijieyoufengj · June 25, 2025, 7:31am

But I use jupyter…

luohuashijieyoufengj · June 25, 2025, 7:37am

I think it is because TrainingArguments has another argument warmup_steps…

John6666 · June 25, 2025, 9:00am

warmup_steps

Oh. Let’s use it.

Topic		Replies	Views
Hyperparameters for lr_scheduler_type in Trainer Arguments Beginners	2	13590	March 5, 2024
How do use lr_scheduler Beginners	11	15137	January 23, 2024
Using Cosine LR scheduler via TrainingArguments in Trainer Beginners	10	12234	June 3, 2024
Trainer lr scheduler does not get kwargs Beginners	0	137	July 3, 2024
How to ignore attributes of TrainingArguments? Intermediate	4	982	July 30, 2021

How to use lr_scheduler_kwargs param in TrainingArguments?

Related topics