Token classification example script metrics improve despite overfit

馃Transformers
1

Hello everyone,

I鈥檇 like to compare different models on various NER tasks. I found an example script for token classification here (notebooks/token_classification.ipynb at main 路 huggingface/notebooks 路 GitHub)

This script, however, appears to compute precision/recall/F1 using the training dataset rather than the validation dataset.

Starting with epoch 3, the validation loss increases (overfit?), but the scores continue to improve. The bert-base-cased model trained on the conll2003 dataset looks like this:
BERT-Base-Overfit-CONLL-metrics

Trainer.evaluate() also appears to select the training dataset rather than the evaluation dataset. How can I adjust the settings to calculate the metrics on the evaluation dataset?

Best,
Oliver