L^2-SP Regularization

Dr-James July 31, 2022, 6:43am 1

Has anyone by chance implemented L^2-SP regularization for the Adam optimizer?

I want to avoid reinventing the wheel, but I believe this would require a custom version of the AdamW class.

Topic		Replies	Views
Learning rate, LR scheduler and optimiser choice for fine-tuning GPT2 Beginners	1	6611	September 3, 2020
If there are adamw optimizer in pytorch version, while there aren't have a same one in tensorflow version? 🤗Transformers	0	211	July 23, 2022
There is a adamw optimizer in pytorch version.Is there a adamw in tensorflow2 version 🤗Transformers	1	282	July 26, 2022
FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead Beginners	2	4610	July 9, 2023
AdamW implementation Beginners	4	1002	July 17, 2021