Accelerator.backward freeze

[John6666]

I think this is a similar problem, but it might be more reliable to raise an issue on the github page for the accelerate library…

How to collect the accuracy when running multi GPU model with accelerate?

Dear all, When using accelerate for speeding up my training I have the problem, that the calculated accuracy far away from what I get when running the same model on one graphic card is. network, optimizer, trainData, validationData, scheduler = accelerator.prepare(network, optimizer, trainData, validationData, None) for epoch in range(args[“epochs”]): # Set the model in training mode network.train() trainRunningLoss = 0.0 trainRunningCorrect = 0 for i, (images, labels) in enumerate(trainD…

Question about calculating training loss of multi-GPU with Accelerate

I am currently training a model in Kaggle with Accelerate (2 T4 GPUs), and I’m confused about how to calculate or log the training loss correctly. According to some discussions (https://github.com/huggingface/accelerate/issues/2109, https://github.com/huggingface/accelerate/issues/226) and some official examples (complete_nlp_example), it appears that I may only need to log the training loss on the main process. I do some test in Kaggle with the following code: from accelerate import Acceler…