Hi everyone, For quite long time I’m struggling with some weird issue regards distributed train/eval. I’m running a slightly modified version of run_clm.py script with vary number of A100 GPUs (4-8) on 1 node, and keep getting the ChildFailedError right after the training/evaluation ends. I’m run…

Hey guys, I’m glad to announce I solved the issue on my side. As can be seen I use multiple GPUs, which have sufficient memory for the use case. HOWEVER! My issue was due to not enough CPU memory. That’s why my runs crashed and without any trace of the reason. Once I allocated enough cpu (on my c…

torch.distributed.elastic.multiprocessing.errors.ChildFailedError

🤗Transformers

AnustupOCR August 2, 2023, 9:02am 13

Yeah, my issue got resolved after specifying the docker memory and properly specifying the device id’s with CUDA_VISIBLE_DEVICES=…

2 Likes

Topic		Replies	Views
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 🤗Accelerate	1	549	August 15, 2024
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 3 (pid: 10561) of binary 🤗Accelerate	4	4750	January 24, 2024
Errors when training on multi node single gpu 🤗Transformers	1	1729	February 25, 2022
Multi-GPU Distributed Training using Accelerate on Windows 🤗Accelerate	0	1522	August 9, 2023
RAM memory issues while training with torch.distributed.launch 🤗Transformers	1	1013	October 19, 2022

torch.distributed.elastic.multiprocessing.errors.ChildFailedError

Related topics