It seems that the hugging face implementation still uses nn.DataParallel for one node multi-gpu training. In the pytorch documentation page, it clearly states that " It is recommended to use DistributedDataParallel instead of DataParallel to do multi-GPU training, even if there is only a single nod…

Multi gpu training

sgugger February 25, 2021, 11:51pm 2

Both are supported by the Hugging Face Trainer. You just have to use the pytorch launcher to use DistributedDataParallel, see an example here.

Topic		Replies	Views
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	17935	September 6, 2023
Which method is use HF Trainer with multiple GPU? 🤗Transformers	4	1564	June 19, 2023
Running a Trainer in DistributedDataParallel mode 🤗Transformers	1	1449	October 24, 2020
How to run single-node, multi-GPU training with HF Trainer? 🤗Transformers	5	15224	October 16, 2024
Boilerplate for Trainer using torch.distributed Beginners	4	2050	January 11, 2022