I am using accelerate with deepspeed stage 1. I get the following error while saving the model: NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(Seq Num=913, OpType=ALLREDUCE, NumelIn=499133440, NumelOut=499133440, Timeout(ms)=600000) …

Accelerator.save_state errors out due to timeout. Unable to increase timeout through kwargs_handlers

JiaxuZ February 24, 2025, 1:47am 4

I meet the same issue. Have you solved it?

1 Like