GPU memory usage is twice (2x) what I calculated based on number of parameters and floating point precision

Nope that’s not what I’m saying at all. There’s certain overhead CUDA needs when doing things under the hood with all their drivers. It is far from 2x otherwise it’d be impossible to train some models :slight_smile: (And it’s all usable memory that’s available, just might not be “in use”)

This is more indicitive, however in general if you get cuda OOM that just means that again, you ran out of cuda memory. Looking at either or for hints won’t really per-se do much.

After you’ve gone through the initial parts (so like a step or two in) then you can eyeball the output on nvidia-smi (or GPU memory allocated % when looking at like W&B)

1 Like