Hi, I wanted to play with the LLaMA 7B model recently released. With the command below I got OOM error on a T4 16GB GPU. How much GPU do I need to run the 7B model? In the Meta FAIR version of the model, we can adjust the max batch size to make it work on a single T4. What should be done here to ma…

To run the 7B model in full precision, you need 7 * 4 = 28GB of GPU RAM. You should add torch_dtype=torch.float16 to use half the memory and fit the model on a T4.

LLaMA 7B GPU Memory Requirement

🤗Transformers

Atharva May 10, 2023, 12:29am 3

How much would 13B take, 13*4 = 52 GB?

We are getting a CUDA OOM error while finetuning a 13B Llama model on a 4xA100 cluster, what may we be doing wrong

Topic		Replies	Views
LLaMA2 7B uses > 128 GB of GPU Ram and fails with OOM or Loss Scale Minimum 🤗Transformers	3	4303	August 17, 2023
Running into cuda out of memory when running llama2-13b-chat model on multi-gpu machine Intermediate	5	7760	December 21, 2023
OOM error with multi-GPU training of Llama2-70B using QLora 🤗Accelerate	2	1420	October 17, 2023
Multi-GPU inference with accelerate Beginners	0	1168	October 19, 2023
Increasing VRAM Usage with Transformers Trainer Leads to OOM on GPUs 🤗Transformers	2	181	March 29, 2024

LLaMA 7B GPU Memory Requirement

Related Topics