A custom model is nn.Module with self.lm(large language model, CLM). I’m trying to train the model with deepspeed zero3. I’m using A100 x8 gpus. I made ds_config.json file and ran sh file like this. deepspeed finetune/run.py --num_gpus=8 --deepspeed ds_config.json ... Also, I changed Trainer’s met…

Saving checkpoint is too slow with deepspeed

berkayberabi September 11, 2023, 4:44pm 2

Hi, I have the same problem with saving codet5-6b with zero3. The logs says pytorch_model.bin is saved but it is not there and the process hangs. Did you find a solution?

Topic		Replies	Views
Saving checkpoints when using DeepSpeed is taking abnormally long DeepSpeed	0	192	July 22, 2024
Trainer option to disable saving DeepSpeed checkpoints 🤗Transformers	8	6648	May 23, 2023
Checkpoint breaks with deepspeed 🤗Transformers	6	3476	March 20, 2021
Rewrite trainer's save_model method get unexpected pytorch_model.bin file DeepSpeed	0	410	January 8, 2024
Questions about deepspeed resume training 🤗Accelerate	2	2107	October 21, 2022

Saving checkpoint is too slow with deepspeed

Related topics