Hi guys, I was training my model, which was decreasing the loss consistently for several steps until, for no reason, it degraded to very high values in both training and validation as you can see in the dark green line. I just stopped the training and restarted from the best checkpoint (best loss) that is marked by the vertical on the chart and it started to fall as you can see in the light green line.
If I restart the training again it behaves the same way as the light green line. It seems that after a few weeks of training (dark green line) something internal went wrong and the model started to deviate from the expected reducing loss course. When I analyzed the gradients for the dark green line, they showed slight variations and no sign of gradient explosion.
Can anyone explain the reason for the difference or have you been in a similar situation?