I have implemented DDPM based on The Annotated Diffusion Model. The model is training, however, the loss during training does not decrease from epoch 10. It oscillates around some value. The results from sampling are ugly (not sharp, and no apparent shape, just color like mess).
I have tried flowers102 dataset and my sampling looks like this (last three steps):
Epoch 30:
This is after training for 30 epochs,
I cannot find any problem with my model.
I have 1000 timesteps, resolution for images 96x96, using linear_beta_schedule, MSE loss and Adam optimizer with 1e-3.
Any idea what can be wrong or how to debug this? Thank you