This is my config, the learning rate is set to auto and is supposed to be initialized by training_args { "zero_optimization": { "stage": 1, "allgather_partitions": true, "allgather_bucket_size": 1e9, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 1e9, …

How to use different learning rates when deepspeed enabled

John6666 June 14, 2025, 8:11am 2

In cases more complicated than that example, you may need to create your own scheduler.

1 Like

Topic		Replies	Views
How to set different learning rates for different parameters in the model? Beginners	7	412	December 17, 2024
Learning rate with deepspeed is fixed despite lr set to auto DeepSpeed	2	2202	September 6, 2023
Which is actually used to configure scheduler in deepspeed and TrainingArguments? 🤗Transformers	0	93	May 17, 2024
Learning rate setting 🤗Transformers	1	2033	November 16, 2020
HF accelerate DeepSpeed plugin does not use custom optimizer or scheduler 🤗Accelerate	2	37	March 1, 2025