Hello I’m building a chatbot using a transformer model (e.g., GPT 2 or BlenderBot) and I would like to let it run on a server (Windows or Linux). The server has one 11GB GPU. If there is only one inference of the chatbot model at the same time there is no problem. But if there are several concurren…

Concurrent inference on a single GPU

Eichhof November 22, 2021, 12:08am 2

Does somebody have any suggestions? I’m happy about every input.

1 Like

Topic		Replies	Views
When I try to inference on multiple GPUs using multiple processes, the time for model. generate() becomes very long 🤗Transformers	0	480	June 12, 2023
Having issues with running parallel, independent inferences on multiple GPUs Beginners	0	259	September 10, 2024
Multiple threads of Stable diffusion Inpainting slows down the inference on same GPU 🧨 Diffusers	4	2637	March 14, 2025
GPU inference slows down if done in a loop 🤗Transformers	1	1576	July 20, 2020
API Rest with several models loaded using GPU but not at same time Beginners	1	405	June 10, 2021