I’ve recently been trying to work with large models and discovered that using device_map="auto" is an easy way to delegate loading large models onto GPU devices without me having to use .to with a lot of other configuration. I want to learn a bit more about how to design device maps but it seems li…

[image] Handling big models for inference We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Anywhere where I can read more about the `device_map` kwarg in `from_pretrained`?

JoPmt January 5, 2024, 7:44am 2

1 Like

Topic		Replies	Views
Device_map="auto" Beginners	5	19458	September 25, 2024
Using device_map='auto' for training 🤗Accelerate	5	35561	January 24, 2025
Why am I out of GPU memory despite using device_map="auto"? 🤗Accelerate	3	17483	March 18, 2024
Automatically cast input to model's device map Beginners	0	306	March 11, 2024
Infer_auto_device_map returns empty 🤗Accelerate	2	3215	March 15, 2023