Hi,
I have a basic theoretical question. Which one is better for the model and GPU usage?
First option:
--per_device_train_batch_size 8
--gradient_accumulation_steps 2
Second option:
--per_device_train_batch_size 16
2 Likes
If the second one does not OOM, you should have better performance with it. The first is a way to get around the memory error the second would give you.
Both commands are completely equivalent in terms of training done otherwise.
11 Likes
Hi @sgugger
It is better if you can supply some references on this problem.
2 Likes
A source is not necessary for this, I think. The goal of gradient accumulation is exactly to overcome memory constraints of the hardware.
3 Likes
So where is the difference in performance between using GA and without GA as @sgugger mentioned in his answer?
I am not sure that it just involves hardware only.
1 Like
Using gradient accumulation loops over your forward and backward pass (the number of steps in the loop being the number of gradient accumulation steps). A for loop over the model is less efficient than feeding more data to the model, as you’re not taking advantage of the parallelization your hardware can offer.
The only reason to use gradient accumulation steps is when your whole batch size does not fit on one GPU, so you pay a price in terms of speed to overcome a memory issue.
20 Likes
As far as I am aware, the common rule of thumb to select the batch size is “as big as your hardware can support”. For example, the most recent leaks concerning GPT-4’s training suggest that a staggering batch size of 60M is used. This makes me wonder how an engineer should balance the batch size and gradient accumulation steps hyperparameters. For example, at what point do the potential drawback of increasing gradient accumulation steps outweigh the benefits that are attained by using large batch sizes (I guess this particular question would specifically pertain to the clear performance benefits of large batch sizes vs the possible convergence benefits)?
2 Likes
such a beautiful answer bro, it just clicked
1 Like
Isnt it the opposite? Using batched input results in higher memory usage and not gradient accumulation. If gradient accumulation is giving an OOM (Out Of Memory Error), it is guranteed that the first one will also give the same error
1 Like
- Thanks to @ozcangundes for this question. It helps me to understand the interaction of batch_size and gradient_accumulation_steps, and to calculate the number of chunks/rows processed in each step (basically the product of the two), and the number of steps (# of chunks/# of chunks processed in each step).
- Although theoretically, First option and Second option should be the essentially the same (round error notwithstanding), be aware of a bug in small batch size that got fixed just recently.
- Here is the link:
Fine-tuning with small batch sizes and gradient accumulation poorly perform if you use Transformers (TRL)!. - Basically, batch_size=1 and gradient_accumulation_steps=32 is much worse than batch_size= 32 and gradient_accumulation_steps=1 while they are mathematically equivalent! From this article, batch_size of 8 seems to be just good enough.
- Here is one of the fix (and HF has also a fix subsequently): LLM training bug fixes - Gradient accumulation was wrong
Bugs in LLM Training - Gradient Accumulation Fix
Transformers Release Notes v4.46.3 and v4.46.0 - This bug seems to have pretty widespread consequence, and could affect many models.
4 Likes