Multiple threads of Stable diffusion Inpainting slows down the inference on same GPU

🧨 Diffusers
1

I am using Stable diffusion inpainting pipeline to generate some inference results on a A100 (40 GB) GPU. For a 512X512 image it is taking approx 3 s per image and takes about 5 GB of space on the GPU.

In order to have faster inference, I am trying to run 2 threads (2 inference scripts). However, as soon as I start them simultaneously. The inference time decreases to ~6 sec per thread with an effective time of ~3 s per image.

I am unable to understand why this is so. I still have a lot of space available (about 35 GB) on GPU and quite a big CPU ram of 32 GB.

Can someone help me in this regard?

2

Hi @garg-aayush, I think that’s to be expected to some extent, at the end of the day your GPU needs to run double as many computations.

Instead of threading, I’d recommend you accumulate your requests in a batch and pass the whole batch for inference. This should scale a bit better, I think.

1 Like
3

Actually, I never faced this issue while inferring using other models (segmentation or classification). However, it seems the case here.

I was looking at the number of SMs cores being utilised in Stable diffusion pipeline case. It shows SMs being utilised at max capacity:
Screenshot 2022-12-06 at 6.24.45 PM

Maybe, this is the reason why multiple threads are not working. Maybe, this is what you meant by double the computations.

Thanks

2 Likes
4

im trying to figure this issue out as well. i found that there are two ways to perform batching with Stable diffusion using diffusers. the first way is to use the variable num_images_per_prompt, but what’s faster is to provide prompts in a list variable. like prompt = [“prompt 1”, “prompt 2”], it will be faster.

but to perform multi inference using diffusion, i am also struggling with this and looking for answers

1 Like
5

There used to be a problem with multi-threading in Diffusers. If you have 40GB of VRAM, I think the fastest way is to use multi-processes, even though it’s inefficient with VRAM…