Accelerator.backward freeze

🤗Transformers DeepSpeed
1 
        if image_list is not None:
            self.accelerator.backward(loss_ce+loss_diff)
            loss = loss_ce.detach()+loss_diff.detach()
        else:
            self.accelerator.backward(loss_ce)
            loss = loss_ce.detach()

I’m using multi-GPU training, and the code above causes the backward computation graphs on different GPUs to be different, leading to deadlocks. Is there any way to solve this issue?

1 Like
2

I think this is a similar problem, but it might be more reliable to raise an issue on the github page for the accelerate library…