i want to resume training unconditional image generation from latest checkpoint. But after loaded checkpoints, doesn’t continue training. Here is my code and output:
!accelerate launch --multi_gpu train_unconditional.py
–dataset_name=“-------”
–resolution=64 --center_crop --random_flip
–output_dir=“--------”
–train_batch_size=16
–num_epochs=1000
–gradient_accumulation_steps=1
–use_ema
–learning_rate=1e-4
–lr_warmup_steps=500
–mixed_precision=no
–push_to_hub
–resume_from_checkpoint=“checkpoint-4000”
Output:
Resuming from checkpoint checkpoint-4000
04/08/2024 08:51:40 - INFO - accelerate.accelerator - Loading states from checkpoint-4000
The config attributes {‘decay’: 0.9999, ‘inv_gamma’: 1.0, ‘min_decay’: 0.0, ‘optimization_step’: 4000, ‘power’: 0.75, ‘update_after_step’: 0, ‘use_ema_warmup’: True} were passed to UNet2DModel, but are not expected and will be ignored. Please verify your config.json configuration file.
04/08/2024 08:51:41 - INFO - accelerate.checkpointing - All model weights loaded successfully
04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All optimizer states loaded successfully
04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All scheduler states loaded successfully
04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All dataloader sampler states loaded successfully
04/08/2024 08:51:42 - INFO - accelerate.checkpointing - All random states loaded successfully
04/08/2024 08:51:42 - INFO - accelerate.accelerator - Loading in 0 custom states
this is output and stopped running
Help me pls ))