Gradient clipping on Transformers

Saugatkafley December 20, 2023, 5:54am 1

Gradients and optimizers are taking too much space on GPU , thus How to perform gradient clipping during transformers training?

Topic		Replies	Views
Setting requires_grad=False seems not saving GPU memory usage 🤗Transformers	0	329	January 18, 2024
How can I restrict the GPU usage in this case? 🤗Transformers	0	206	January 19, 2024
Why is grad norm clipping done during training by default? 🤗Transformers	3	12872	February 17, 2025
Trainer() and required_grad=false 🤗Transformers	1	284	January 18, 2024
Training Transformer doesn't reach full GPU usage 🤗Transformers	0	540	February 10, 2023