How to define the compute_metrics() function in Trainer?

🤗Transformers
1

Hello,

Coming from tensorflow I am a bit confused as to how to properly define the compute_metrics() in Trainer. For instance, I see in the notebooks various possibilities

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

My question may seem stupid (maybe it is) but how can I know how to compute the metrics if I cannot see what eval_pred looks like in Trainer? It is as if I had to guess what the output will be before actually training the model. Am I missing something here?

Thanks!

2 Likes
2

Hello, I’m a bit confused about your question. Are you trying to implement this on Trainer or TensorFlow?

3

hello @merve, thanks. I am trying to write the correct Trainer (in pytorch). The point is that I can follow the tutorial and copy-paste the compute_metrics functions there. However, if I want to modify them I need to understand and play a little bit with the eval_pred output. How can I get eval_pred from a trainer so that I can see what predictions and labels look like?

4

Just run trainer.predict on your eval/test dataset.

6 Likes