When creating a default optimizer the Trainer class creates two parameter groups based on whether weight decay should be applied or not and it does that based on the parameter name (does it contain “LayerNorm” or “bias” in it or not). Problem is, not all models’ parameters are named the same way; G…

Parameter groups and GPT2 LayerNorm

salti March 9, 2021, 1:14pm 4

Thanks!
That was very quick

Topic	Replies	Views
Bert model on Acceptability Judgement Task \|\| Optimizer Grouped Parameters Beginners	558	September 11, 2021
Pre/Post Normalization-layers 🤗Transformers	90	June 3, 2024
Unable to train a good model after using exclude_from_weight_decay Intermediate	408	October 19, 2021
Why get_parameter_names()? Beginners	481	March 8, 2024
AdamW Pytorch vs Huggingface 🤗Transformers	1400	January 27, 2023