Hello. I try to train RoBERTa from scratch. I have several V100 GPUs. I already know that huggingface’s transformers automatically detect multi-gpu. And I checked it for myself in training log. But, there is something I couldn’t understand. There is no improvement performance between using single a…

How can I get advantage using multi-GPUs

valhalla January 22, 2021, 7:02am 2

for multi-GPU training you need to launch the script using torch.distributed.launch, have a look at this section of example docs

Topic		Replies	Views
Multiple GPUs do not speed up the training 🤗Accelerate	1	3493	January 26, 2022
Training using multiple GPUs Beginners	20	20271	February 25, 2024
Mutli GPU freezes on Roberta Pretraining Beginners	6	2086	July 11, 2022
Multiple gpu training 🤗Transformers	1	3009	August 10, 2024
How can I use trainer.train() in runpod's multi gpu? 🤗Transformers	0	430	August 4, 2023