Hi, I have a basic theoretical question. Which one is better for the model and GPU usage? First option: --per_device_train_batch_size 8 --gradient_accumulation_steps 2 Second option: --per_device_train_batch_size 16

Using gradient accumulation loops over your forward and backward pass (the number of steps in the loop being the number of gradient accumulation steps). A for loop over the model is less efficient than feeding more data to the model, as you’re not taking advantage of the parallelization your hardwar…

Batch size vs gradient accumulation

Beginners

bui December 21, 2021, 12:30pm 5

So where is the difference in performance between using GA and without GA as @sgugger mentioned in his answer?

I am not sure that it just involves hardware only.

1 Like

Topic		Replies	Views
Selecting batch_size and gradient_accumulation_steps when fine-tuning Models	1	2044	December 31, 2023
Switch batch size and gradient accumulation step values mid training Beginners	0	216	February 28, 2024
Question about Gradient Accumulation step in Trainer 🤗Transformers	2	2501	September 10, 2021
What is the limit of grad accumulation? Intermediate	2	2839	May 4, 2021
Batch size, gradient accumulation steps for Linear schedule Models	0	700	May 1, 2021

Batch size vs gradient accumulation

Related topics