It seems that the hugging face implementation still uses nn.DataParallel for one node multi-gpu training. In the pytorch documentation page, it clearly states that " It is recommended to use DistributedDataParallel instead of DataParallel to do multi-GPU training, even if there is only a single nod…

Multi gpu training

Lee1990 April 24, 2022, 9:04am 3

how to find the example?

Topic		Replies	Views
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	18028	September 6, 2023
Which method is use HF Trainer with multiple GPU? 🤗Transformers	4	1564	June 19, 2023
Running a Trainer in DistributedDataParallel mode 🤗Transformers	1	1450	October 24, 2020
How to run single-node, multi-GPU training with HF Trainer? 🤗Transformers	5	15292	October 16, 2024
Boilerplate for Trainer using torch.distributed Beginners	4	2055	January 11, 2022