Thanks @muellerzr, could you also take a look at a related problem I have ZeRO uses more RAM than DDP??