I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious a…

CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags

hanguo September 1, 2023, 2:44pm 5

Got it, thanks again!

Topic		Replies	Views
LLaMA-2: CPU Memory Usage with ‘low_cpu_mem_usage=True’ and ‘torch_dtype=“auto”’ flags 🤗Transformers	0	2138	September 1, 2023
CPU generate is only using 15% cpu (LLaMA 13B) 🤗Transformers	0	1178	April 9, 2023
Device_map="auto" in MIG Instance 🤗Transformers	0	245	January 23, 2024
Load_checkpoint_and_dispatch without heavy system memory usage 🤗Accelerate	1	2236	April 10, 2023
Code makes inference with "Llama 3 70b instruct" model on CPU but has problem with inference with GPUs Beginners	0	441	April 28, 2024