I have a machine with 3 3090s and have been using accelerate with lm_eval to speed up inference and seeing sensible results. I wrote some custom training scripts using accelerate but noticed about a 3x slowdown vs the single GPU case. While debugging, I decided to try the nlp_example.py and I’m get…

If anyone else runs into this issue: it turned out to be a BIOS level change that was needed in order to fix the communication overhead. Specifically changing the link speed on the PCI ports from Gen 1 to Gen 4 resulted in seeing speedups using multiple GPUs for fine-tuning!

Multi-GPU is slower than single GPU when running examples

🤗Accelerate

system Closed July 25, 2024, 7:33am 3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Single GPU is faster than multiple GPUs 🤗Accelerate	3	2022	January 31, 2024
What does "--multi_gpu" do under the hood? (and how to use it) 🤗Accelerate	7	6641	May 31, 2023
Multiple GPUs do not speed up the training 🤗Accelerate	1	3464	January 26, 2022
Multi-gpu training does not optimize as expected Beginners	1	458	February 26, 2024
Multi-GPU Training sometimes working with 2GPU, but never more than 2 🤗Accelerate	5	3047	August 8, 2024

Multi-GPU is slower than single GPU when running examples

Related topics