Hi, I wanted to play with the LLaMA 7B model recently released. With the command below I got OOM error on a T4 16GB GPU. How much GPU do I need to run the 7B model? In the Meta FAIR version of the model, we can adjust the max batch size to make it work on a single T4. What should be done here to ma…

To run the 7B model in full precision, you need 7 * 4 = 28GB of GPU RAM. You should add torch_dtype=torch.float16 to use half the memory and fit the model on a T4.

LLaMA 7B GPU Memory Requirement

🤗Transformers

awacke1 August 2, 2023, 5:10pm 9

Thanks much. This is very useful! I’m curious to learn more about bitsandbytes - e.g. AdamW 8bit to get it working w 14GB. Does anyone have the model on HF by using the last optimizer you mention?
–Aaron

Topic		Replies	Views
Llama2 70b - Cuda out of memory exceptions 🤗Transformers	0	164	February 28, 2024
Requirements Llama2 Intermediate	0	291	April 13, 2024
How to ensure that while running with llama2-70B, we use parallelism? 🤗Optimum	11	1593	August 22, 2023
LLaMA2 7B uses > 128 GB of GPU Ram and fails with OOM or Loss Scale Minimum 🤗Transformers	3	5597	August 17, 2023
Colab CUDA OOM using Llama-2-7b-chat-hf even with 40GPU RAM 🤗Transformers	0	912	December 29, 2023

LLaMA 7B GPU Memory Requirement

Related topics