Hi everyone, For quite long time I’m struggling with some weird issue regards distributed train/eval. I’m running a slightly modified version of run_clm.py script with vary number of A100 GPUs (4-8) on 1 node, and keep getting the ChildFailedError right after the training/evaluation ends. I’m run…

Hey guys, I’m glad to announce I solved the issue on my side. As can be seen I use multiple GPUs, which have sufficient memory for the use case. HOWEVER! My issue was due to not enough CPU memory. That’s why my runs crashed and without any trace of the reason. Once I allocated enough cpu (on my c…

torch.distributed.elastic.multiprocessing.errors.ChildFailedError

🤗Transformers

christineac January 22, 2025, 11:06am 20

@shirubei what’s your version of transformers?

1 Like

Topic		Replies	Views
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 🤗Accelerate	1	622	August 15, 2024
Error when fine-tuning on multi-gpu 🤗Transformers	1	636	February 17, 2025
Generate keeps increasing memory usage on ubuntu 🤗Transformers	6	56	May 25, 2025
Multi-GPU Training sometimes working with 2GPU, but never more than 2 🤗Accelerate	5	3016	August 8, 2024
Run crash with all GPU's and success with less 🤗Transformers	0	421	December 12, 2022

torch.distributed.elastic.multiprocessing.errors.ChildFailedError

Related topics