Bug in gradient accumulation training_step in huggingface Trainer?

John6666 · November 2, 2024, 1:14am

The github version has been slightly changed…
I don’t know if it’s fixed or not.

huggingface/transformers/blob/main/src/transformers/trainer.py#L3645


      
              if self.use_apex:
                  with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                      scaled_loss.backward()
              else:
                  if num_items_in_batch is not None:
                      if self.compute_loss_func or self.model_accepts_loss_kwargs:
                          loss *= self.args.gradient_accumulation_steps
                      # Average tokens across devices is orthogonal to gradient accumulation
                      if self.args.average_tokens_across_devices:
                          loss *= self.args.world_size
                  self.accelerator.backward(loss, **kwargs)
          
              return loss.detach() / self.args.gradient_accumulation_steps
          
          def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):
              """
              How the loss is computed by Trainer. By default, all models return the loss in the first element.
          
              Subclass and override for custom behavior.
              """
              if (self.label_smoother is not None or self.compute_loss_func is not None) and "labels" in inputs:

I don’t think the part in question has changed…
I wonder what is going right…

Topic		Replies	Views
Is there a way to backpropagate through multiple steps while using Trainer API 🤗Transformers	1	254	July 9, 2021
Gradient accumulation averages over gradient 🤗Transformers	2	2070	November 12, 2020
Using gradient_accumulation_steps does not give the same results 🤗Accelerate	0	520	February 18, 2023
Gradient accumulation: should I duplicate data? 🤗Transformers	7	1020	February 1, 2021
Question about training_step Function in Class Trainer 🤗Transformers	0	41	September 23, 2024

Bug in gradient accumulation training_step in huggingface Trainer?

Related topics