While analysing the GPU memory footprint of a few models ‘prepared’ via accelerator.prepare(), I found the models occupying more than twice the GPU memory compared to normally loaded models on device. What can be the reason for this inflated memory footprint? I am using a single V100 32 GB GPU and …

Inflated GPU memory footprint of model prepared via accelerate

varadhbhatnagar September 15, 2023, 4:40am 5

1 Like

Topic		Replies	Views
`Accelerator.prepare` utilize only one GPU instead of all the 8 available GPUs and raises "CUDA out of memory" 🤗Accelerate	2	1638	July 13, 2023
Data Parallel Multi GPU Inference 🤗Accelerate	9	2902	September 15, 2023
Accelerator OOM 🤗Accelerate	2	769	July 5, 2023
FSDP accelerate.prepare gives OOM. How to load model into single GPU, then distribute shards? 🤗Accelerate	2	501	January 24, 2024
How to load large model with multiple GPU cards? Beginners	8	22026	October 25, 2023