How to load the best model based on loss *and* eval_loss

🤗Transformers
1

Is there any way to load the best model at the end of training based on training loss and eval loss? I’m asking since there may be some scenarios where after a couple of epochs, we end up having a training loss that is getting larger but an eval loss that is getting smaller. For example,

Epoch|train_loss|eval_loss
|1   |1.420300  |1.265727|
|2   |1.237900  |1.162285|
|3   |1.184300  |1.076674|
|4   |0.982200  |1.068642|
|5   |1.055500  |1.035673|
|6   |0.889900  |0.997194|
|7   |0.830300  |0.999240|
|8   |0.799100  |0.957392|
|9   |0.963000  |0.870068|
|10  |0.798400  |0.994886|

So in this case, even though at epoch 9 we have the best eval loss, the train loss is actually getting larger than previous epochs. As far as I understand, when metric_for_best_model is set to eval_loss, the best model will be loaded solely based on eval loss. So is there any way that the best model can be loaded based on the best combination of the train and eval loss? Or does it make sense at all?

1 Like