When I load a checkpoint from my 40 thousandth step, the model loads and continues training fine. But when I tried to resume training from m 120 thousandth step’s checkpoint, I get Runtime Error: Cuda out of memory.
Why must that be happening?
When I load a checkpoint from my 40 thousandth step, the model loads and continues training fine. But when I tried to resume training from m 120 thousandth step’s checkpoint, I get Runtime Error: Cuda out of memory.
Why must that be happening?