Hi, I wanted to play with the LLaMA 7B model recently released. With the command below I got OOM error on a T4 16GB GPU. How much GPU do I need to run the 7B model? In the Meta FAIR version of the model, we can adjust the max batch size to make it work on a single T4. What should be done here to ma…

To run the 7B model in full precision, you need 7 * 4 = 28GB of GPU RAM. You should add torch_dtype=torch.float16 to use half the memory and fit the model on a T4.

LLaMA 7B GPU Memory Requirement

🤗Transformers

Atharva May 10, 2023, 12:29am 3

How much would 13B take, 13*4 = 52 GB?

We are getting a CUDA OOM error while finetuning a 13B Llama model on a 4xA100 cluster, what may we be doing wrong

Topic		Replies	Views
LLaMA2 7B uses > 128 GB of GPU Ram and fails with OOM or Loss Scale Minimum 🤗Transformers	3	5156	August 17, 2023
Llama 3.1 70-B run on 32 GB Vram? 🤗Transformers	5	513	September 20, 2024
Fine tune Meta-Llama-3.1-8B OOM error after the 1st training step Models	0	51	September 6, 2024
Running into cuda out of memory when running llama2-13b-chat model on multi-gpu machine Intermediate	5	9878	December 21, 2023
OOM error with multi-GPU training of Llama2-70B using QLora 🤗Accelerate	2	2021	October 17, 2023

LLaMA 7B GPU Memory Requirement

Related Topics