Optimizing Model Loading with a CPU Bottleneck

[John6666]

Accelerate Utilities

I think that’s the correct workaround. If you want to treat VRAM, RAM, and disk as a single entity, you should use the Accelerate library.

Environments where RAM is less than VRAM are not commonly expected, so this issue is not often reported, but excessive RAM consumption during model loading can occasionally become an issue.

• infer_auto_device_map inefficiently allocates GPU memory for models with imbalanced module sizes
• device_map=‘auto’ causes memory to not be freed with torch.cuda.empty_cache()
• How to avoid the peak RAM memory usage of a model when I want to load to GPU