Hi, I want to train Trainer scripts on single-node, multi-GPU setting. Do I need to launch HF with a torch launcher (torch.distributed, torchX, torchrun, Ray Train, PTL etc) or can the HF Trainer alone use multiple GPUs without being launched by a third-party distributed launcher?

How to run single-node, multi-GPU training with HF Trainer?

brando August 17, 2022, 3:38pm 3

4 Likes

Topic		Replies	Views
Distributed Training w/ Trainer 🤗Transformers	8	7884	May 31, 2022
Which method is use HF Trainer with multiple GPU? 🤗Transformers	4	1470	June 19, 2023
Single Node Multi GPU FlanT5 fine-tuning using HF Dataset and HF Trainer 🤗Transformers	4	1991	July 5, 2023
Does the HF Trainer class support multi-node training? Beginners	4	2364	January 9, 2023
Training using multiple GPUs Beginners	20	19695	February 25, 2024