Is there a clean way to perform evaluation on a model without using the Trainer?
I have scripts which train and then evaluation my model. I want to perform further evaluation on the trained model - for now I’ve hacked my training script so I create a Trainer but don’t call train() - is there a better way of doing this?
you’ll need to set up the evaluation dataset / dataloader and write your own evaluation loop to compute the metrics.
you can find examples of this in the examples/legacy directory of the transformers repo, e.g. here is the evaluation for question-answering on squad: transformers/run_squad.py at 6e1ee47b361f9dc1b6da0104d77b38297042efae · huggingface/transformers · GitHub
You can also have a look at our new examples that don’t use a Trainer (all the run_xxx_no_trainer in the task subfolders of the examples folder of the repo).