Loading a HF Model in Multiple GPUs and Run Inferences in those GPUs

🤗Accelerate
1

Hi,
Is there any way to load a Hugging Face model in multi GPUs and use those GPUs for inferences as well?

Like, there is this model which can be loaded on a single GPU (default cuda:0) and run for inference as below:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/LLaMA-2-7B-32K")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/LLaMA-2-7B-32K", torch_dtype=torch.float16)

input_context= "Your text here"
input_ids = tokenizer.encode(input_context, return_tensors="pt").to(model.device)
output = model.generate(input_ids, max_length=256, temperature=0.7)
output_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(output_text)

How should I load and run this model for inference on two or more GPUs using Accelerate or DeepSpeed?

Any guidance/help would be highly appreciated, thanks in anticipation!

2

Could you find a solution of this problem?
It didn’t work for me but did you try this on terminal? $accelerate launch model.py

3

Hi, this will not work since I have not used accelerator related stuffs in the code.

4

Oh okay. I have a code which in accelerator is used but I couldn’t run on that 2 gpu. I’m searching for a solution for that.

5

Can you share it with me? I will run it.

6

I’m triying to run demo.py file on 2 gpu because when I use 1 gpu it gives out of memory error. But I couldn’t run on multi gpu.

7

I found a solution and have posted it here.

1 Like
8

It’ll spin up PyTorch properly to use DDP, so you can prepare the model that way if you want. Otherwise there’s a tutorial on distributed inference with accelerate here: Distributed Inference with 🤗 Accelerate

9

will this solution work for me? Should I change the accelerator to deep speed?

10

This works for both, you can use accelerate launch model.py instead of the deepspeed command.

11

Hi, I’m having a similar problem but I could not apply the tutorial you’re referencing here because I’m working with Diffusers pipelines and I do not get the model isolated to use “prepare_pippy” as the tutorial says. Do you know if there is any possibility to isolate the model from the pipeline so I can follow that tutorial or should I use another approach?

Thanks in advance.

1 Like