Gradient clipping on Transformers

Saugatkafley December 20, 2023, 5:54am 1

Gradients and optimizers are taking too much space on GPU , thus How to perform gradient clipping during transformers training?

Topic		Replies	Views
Finetuning and single-GPU utilization 🤗Transformers	0	405	August 19, 2021
Why is there no cross-gpu negative sample gathering for CLIP model in multiple-gpu training? Models	2	89	March 18, 2024
Are the performance tricks from v4.18.0 relocated in the main branch site? Site Feedback	3	540	November 1, 2022
Model Parallelism, how to parallelize transformer? Beginners	3	10429	June 18, 2021
OutOfMemoryError when trying Transformers "Training on one GPU" Tutorial Beginners	1	201	July 16, 2023