I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious a…

CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags

hanguo September 1, 2023, 2:44pm 5

Got it, thanks again!

Topic		Replies	Views
LLaMA-2: CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags 🤗Transformers	0	3104	September 1, 2023
Llama 3.1 8b Instruct - Memory Usage More than Reported Models	5	209	February 18, 2025
CPU generate is only using 15% cpu (LLaMA 13B) 🤗Transformers	0	1312	April 9, 2023
Device_map="auto" in MIG Instance 🤗Transformers	0	511	January 23, 2024
Load_checkpoint_and_dispatch without heavy system memory usage 🤗Accelerate	1	3016	April 10, 2023