Hello, I’m exploring methods to manage CUDA Out of Memory (OOM) errors during the inference of 70 billion parameter models without resorting to quantization. Specifically, I’m interested in leveraging CPU/disk offloading. Does the Accelerate library offer solutions for this? I’ve examined the curre…

[image] hellov: Does the Accelerate library offer solutions for this? Yes. Just set device_map="auto" in your call and it’ll do this automatically

Big Model Inference: CPU/Disk Offloading for Transformers Using from_pretrained

muellerzr February 28, 2024, 3:56pm 2

Yes. Just set device_map="auto" in your call and it’ll do this automatically

2 Likes

Topic		Replies	Views
Llama 3.1 70-B run on 32 GB Vram? 🤗Transformers	5	4227	September 20, 2024
Accelerate use of memory 🤗Transformers	1	170	February 7, 2025
Offloading LLM models to CPU uses only single core 🤗Transformers	1	4077	June 3, 2024
Runtime error: CUDA out of memory, not sure if accelerate offloading is working Beginners	0	928	October 2, 2023
Any good code/tutorial that is shows how to do inference with Llama 2 70b on multiple GPUs with accelerate? 🤗Accelerate	1	2835	November 27, 2023