Fine-tuning Wav2v2.0: Loss increasing, WER decreasing

Models
1

I’m fine-tuning the XLS-R large on my own dataset. After 20 epochs, the validation loss keeps increasing while validation WER keeps decreasing? Is this a sign of overfitting?

2

@ngoquanghuy I’m facing the opposite issue where the loss keeps decreasing whereas the WER keeps increasing (to the point where it’s 2x worse than a model not fine-tuned at all). Did you figure it out?

3

@iamgroot42 Ignore the WER. Loss is the most truthful factor.