GPU memory calculator

🤗Accelerate
1

I am looking for a calculator tool that can estimate the amount of memory a GPU machine instance will use in advance. Specifically, I’m interested in the following configuration:

  • FSDP Zero 2

  • Model: e.g., LLama3 8B

-Lora

  • Max tokens: 8K

  • 16-bit precision

  • Gradient checkpointing

  • No quantization

Any guidance or recommendations would be greatly appreciated! Thank you.

Edit: I know about hf tool (Understanding how big of a model can fit on your machine) but this is rudimentary and lacks many of parameters such as max sequence token, lora, …

3 Likes
2

This one has a bit more than the HF one: Tech 42. There’s an option to add a custom model.

1 Like
3

You can try the Python code for the calculations. Use the formulas with all parameters:

total_memory=(model_memory+activation_memory+lora_memory)Xcheckpointing_factorXfsdp_factor