Hi everyone I was following these two blogs Handling big models and How :hugs: Accelerate runs very large models thanks to PyTorch and I wanted to use it for nllb-200-3.3B on CPU. Here is my script from accelerate import init_empty_weights, load_checkpoint_and_dispatch from transformers import …

Using loaded model with accelerate for inference

sgugger November 3, 2022, 3:04pm 2

You can’t use disk offload on CPU, this is only supporter on GPU for now.

2 Likes

Topic		Replies	Views
Meta device error while instantiating model 🤗Accelerate	5	7063	April 1, 2025
How to load part of the model weight to inference? 🤗Accelerate	0	356	June 28, 2023
Accelerate use of memory 🤗Transformers	1	129	February 7, 2025
Big Model Inference: CPU/Disk Offloading for Transformers Using from_pretrained 🤗Accelerate	2	4966	February 28, 2024
HuggingFacePipeline Llama2 load_in_4bit from_model_id the model has been loaded with `accelerate` and therefore cannot be moved to a specific device 🤗Accelerate	2	7176	October 9, 2024