I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious a…

CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags

hanguo September 1, 2023, 2:44pm 5

Got it, thanks again!

Topic		Replies	Views
LLaMA-2: CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags 🤗Transformers	0	3360	September 1, 2023
Double expected memory usage Beginners	1	1431	August 17, 2022
Question about memory usage Beginners	0	984	May 15, 2023
Loading of a model takes much RAM, passing to CUDA doesn't free RAM 🤗Transformers	0	783	August 8, 2021
On cpu, how to save memory when inferencing? 🤗Transformers	1	639	July 13, 2023