Hello, I’m fine-tuning a BERT model with MatryoshkaLoss and noticed that the eval_loss suddenly spikes. At the same time, the Spearman and Pearson correlation values drop sharply and then turn to NaN. After the spike, both eval_loss and train_loss remain constant throughout training. Do you have any idea why this might be happening?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Loss behaviour for bert fine-tuning on QNLI | 3 | 4292 | October 15, 2021 | |
| Eval Loss spike Seq2seq Trainer Resume from Checkpoint | 0 | 516 | June 22, 2021 | |
| Training Loss Sudden Spike After 8 Hours of pre-training a BERT Model | 0 | 1085 | September 13, 2023 | |
| HuggingFace Trainer - Eval loss abruptly goes up at the last step of training | 1 | 1957 | November 8, 2022 | |
| `nan` training loss but eval loss does improve over time | 5 | 3949 | October 10, 2022 |