Hello,
I am training the SpeechT5 model for TTS, and the training loss is stuck at 0.41, and the loss for the evaluation set is at 0.36. No matter what I do, the loss isn’t going down. I tried to change the following values:
- Learning rate: 1e-6
- Weight decay: 1e-6
- Adam beta 1: 0.8
- Adam beta 1: 0.96
- Adam epsilon: 1e-8
None of these changes helped. The training batch size is 32 with gradient accumulation steps at 4.
Has anyone been able to successfully train this model to lower losses, and if so, what’s the secret?