When i do model = AutoModelForCausalLM.from_pretrained( "EleutherAI/gpt-neo-125m", low_cpu_mem_usage=True ) I expect that since this is a fp32 model with 0.125 billion parameters, the amount of vRAM the model should occupy in GPU should be: 4 bytes per parameter x 0.125 billion parameters …

[image] clam004: so i guess what youre saving is that my GPU is “reserving” twice the model’s parameters in memory. Nope that’s not what I’m saying at all. There’s certain overhead CUDA needs when doing things under the hood with all their drivers. It is far from 2x otherwise it’d be impossib…

GPU memory usage is twice (2x) what I calculated based on number of parameters and floating point precision

Intermediate

system Closed May 19, 2024, 9:44am 6

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Loading of a model takes much RAM, passing to CUDA doesn't free RAM 🤗Transformers	0	774	August 8, 2021
Memory overhead/usage calculation Intermediate	3	53	June 20, 2025
Missmatch between memory-estimate and Trainer-API Beginners	0	183	January 23, 2024
Loading model directly to GPU omitting RAM Beginners	6	84	March 28, 2025
GPU memory calculator 🤗Accelerate	2	1870	July 5, 2024

GPU memory usage is twice (2x) what I calculated based on number of parameters and floating point precision

Related topics