I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious a…

CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags

hanguo September 1, 2023, 2:44pm 5

Got it, thanks again!

Topic		Replies	Views
LLaMA-2: CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags 🤗Transformers	0	3292	September 1, 2023
Double expected memory usage Beginners	1	1416	August 17, 2022
The CPU memory usage becomes very small during model inference 🤗Transformers	0	49	November 30, 2024
Loading model directly to GPU omitting RAM Beginners	6	80	March 28, 2025
GPU memory usage is twice (2x) what I calculated based on number of parameters and floating point precision Intermediate	5	451	May 18, 2024