How to do distributed Inference for large models with multiprocess?

NobelHu · March 21, 2024, 8:45am

I have a model with 20B and 4 A100 GPU with 40G gpu memory. I want to create two processes and each process own 2 gpus, then I can inference with fp16. So how can I do that with accelerate?

NobelHu · March 25, 2024, 6:52am

I have solved this problem by using DeepSpeed inference.

marcsun13 · April 15, 2024, 1:34pm

Hi @NobelHu, glad that you managed to make it work ! Would you mind sharing your solution for the community ? We are also thinking about linking an deepspeed inference in our docs so that everyone benefits from it.

Topic		Replies	Views
Data Parallel Multi GPU Inference 🤗Accelerate	9	2841	September 15, 2023
Data Parallelism for multi-GPUs Inference Intermediate	0	349	October 26, 2022
When I try to inference on multiple GPUs using multiple processes, the time for model. generate() becomes very long 🤗Transformers	0	281	June 12, 2023
Any good code/tutorial that is shows how to do inference with Llama 2 70b on multiple GPUs with accelerate? 🤗Accelerate	1	856	November 27, 2023
Proper way to gather output from accelerate multi-gpu inference Beginners	1	521	November 7, 2023

How to do distributed Inference for large models with multiprocess?

Related Topics