Llama 3.1 70-B run on 32 GB Vram?

[John6666]

To do this, it seems to be sufficient to specify device_map=“auto”.

Big Model Inference: CPU/Disk Offloading for Transformers Using from_pretrained

Hello, I’m exploring methods to manage CUDA Out of Memory (OOM) errors during the inference of 70 billion parameter models without resorting to quantization. Specifically, I’m interested in leveraging CPU/disk offloading. Does the Accelerate library offer solutions for this? I’ve examined the current documentation, which appears to focus on custom NLP models rather than facilitating the use of pretrained large models like LLaMA 2 70B. Here’s the guide I found: Hugging Face’s Accelerate Big Mode…

However, HF libraries in general are still not very good at dealing with quantised files, so it would not be surprising if there were errors.