I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious a…

CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags

hanguo September 1, 2023, 2:44pm 5

Got it, thanks again!

Topic		Replies	Views
LLaMA-2: CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags 🤗Transformers	0	3426	September 1, 2023
Double expected memory usage Beginners	1	1441	August 17, 2022
Question about memory usage Beginners	0	1022	May 15, 2023
Loading of a model takes much RAM, passing to CUDA doesn't free RAM 🤗Transformers	0	794	August 8, 2021
On cpu, how to save memory when inferencing? 🤗Transformers	1	651	July 13, 2023