How to freeze layers while fine-tuning?

John6666 · May 15, 2025, 2:19pm

Maybe requires_grad=False?

Exclude the parameters with `requires_grad=False` in the `Trainer` optimizer.

opened 07:46PM - 18 Jan 23 UTC

closed 02:28PM - 23 Jan 23 UTC

### Feature request Attempt to optimize the training for models with weights/pa…rameters that are set to `requires_grad=False`. This is done by excluding these parameters in the optimizer. ### Motivation I am building a Seq2Seq model where I use a pre-trained model for the encoder. I freeze all the parameters of the encoder by setting `requires_grad=False`. I expected the training to speed up compared to a model where both the encoder and decoder weights are trainable. However, I found that there's no difference in speed and also memory. I investigated a bit and found that all the model parameters, regardless of whether gradients are required to be computed, are included in the optimizer https://github.com/huggingface/transformers/blob/00ba7cadd812437708b380ab078a3cfe8cfaff31/src/transformers/trainer.py#L1021-L1030 I tested an idea and subclassed the `Seq2SeqTrainer`. So, I updated the above snippet with this: ```Python optimizer_grouped_parameters = [ { # Add here the `p.requires_grad` condition "params": [p for n, p in opt_model.named_parameters() if (n in decay_parameters and p.requires_grad)], "weight_decay": self.args.weight_decay, }, { # Add here the `p.requires_grad` condition "params": [p for n, p in opt_model.named_parameters() if (n not in decay_parameters and p.requires_grad)], "weight_decay": 0.0, }, ] ``` Doing this actually improved both the speed and the memory during the training. I was wondering if this is something we can add to the codebase. If not, I am curious as to why we shouldn't exclude the parameters that are intended not to be trainable in the optimizer. ### Your contribution I can make the PR if this is an acceptable change. 🤗

Topic		Replies	Views
How to freeze layers using trainer? Beginners	11	32471	May 26, 2024
Gradual Unfreezing support for Fine tuning models 🤗Transformers	3	4001	August 26, 2020
Gradual Layer Freezing with huggingface model 🤗Transformers	1	898	February 10, 2021
Freeze Lower Layers with Auto Classification Model 🤗Transformers	6	18560	May 25, 2023
How to correctly freeze some of the Wav2Vec2-Bert's layers? 🤗Transformers	0	116	July 24, 2024

How to freeze layers while fine-tuning?

Related topics