For example, I use batch size=64, max steps=1000, log, evaluate and save steps=100. Question 1: I want to confirm that if I use gradient accumulation (batch size=32, gradient accumulation steps=2), I don’t need to change step args (1000 → 2000, 100 → 200). Will transformers take care of gradient_ac…

Questions about steps with gradient accumulation

logvinata July 19, 2023, 11:47am 2

Hi, @shelf! Did you find the answers?

Topic		Replies	Views
Gradient accumulation: should I duplicate data? 🤗Transformers	7	1017	February 1, 2021
Using gradient_accumulation_steps does not give the same results 🤗Accelerate	0	519	February 18, 2023
Any incompatibility of gradient_accumulation with the streaming data? 🤗Transformers	0	251	July 10, 2023
Batch size, gradient accumulation steps for Linear schedule Models	0	719	May 1, 2021
Bug in gradient accumulation training_step in huggingface Trainer? 🤗Transformers	3	942	November 2, 2024