Hey folks, I’m trying to minimize my inference time when using XLNet for text classification. I’ve used Deepspeed and it’s integration with Huggingface pipeline. Right now the issue is it takes more time on 4 GPUs than a single GPU. I’m researching for couple of days but didn’t find anything to add…

Infrence time increase when using multi-GPU

nuxee November 28, 2023, 5:48am 2

Have you solved this problem?

Topic		Replies	Views
I have a question about multi-GPU inference DeepSpeed	0	1521	March 9, 2023
How to train on multiple GPUs the Informer model for time series forecasting? 🤗Accelerate	7	2838	August 18, 2023
Multiple gpu not properly parallelized during model.generate() 🤗Transformers	4	1639	October 9, 2022
Timeout Issue with DeepSpeed on Multiple GPUs DeepSpeed	2	617	July 21, 2025
What does "--multi_gpu" do under the hood? (and how to use it) 🤗Accelerate	7	6640	May 31, 2023