Hi all! I have a 12B model that is distributed on 4 GPUs and a 2.8B model that is also distributed. I’m performing inference on 12B followed by training 2.8B. However, after a first few thousand steps of training, I’m getting the NCCL timeout error. I even tried increasing the default timeout to 90 …

NCCL Watchdog Timeout error while using Deepspeed and accelerate

jaydeepb July 16, 2025, 3:52am 4

thanks! let me try that and see if it helps.

1 Like