Hi everyone I was following these two blogs Handling big models and How :hugs: Accelerate runs very large models thanks to PyTorch and I wanted to use it for nllb-200-3.3B on CPU. Here is my script from accelerate import init_empty_weights, load_checkpoint_and_dispatch from transformers import …

Using loaded model with accelerate for inference

sgugger November 3, 2022, 3:04pm 2

You can’t use disk offload on CPU, this is only supporter on GPU for now.

2 Likes

Topic		Replies	Views
How to load part of the model weight to inference? 🤗Accelerate	0	358	June 28, 2023
Accelerate not spreading on multiple CPUs 🤗Accelerate	1	1837	August 1, 2023
Meta device error while instantiating model 🤗Accelerate	5	7155	April 1, 2025
Big Model Inference: CPU/Disk Offloading for Transformers Using from_pretrained 🤗Accelerate	2	5131	February 28, 2024
NLP Pretrained model model doesn’t use GPU when making inference 🤗Transformers	11	10162	March 11, 2022