Different classification label produced by 'predictions' and 'label_ids' from Trainer.predict()

🤗Transformers
1

What does predictions and label_ids actually mean from Trainer.predict()?

I trained a multilabel classification model and tested it on a test dataset. The Trainer.predict.metrics gave me the output below:

image

I thought label_ids should be the predicted label so I did a confusion matrix between label_ids and my testing data. The result shows a perfect prediction with accuracy = 1, recall =1, precision = 1 etc.

I realized something was wrong so I computed the label myself with the logit values produced by Trainer.predict.predictions:
compute_labels = tf.round(tf.nn.sigmoid(test_prediction.predictions))

Running confusion matrix with the compute_labels and the test data, I am able to get a reasonable prediction results that replicated the output of Trainer.predict.metrics (i.e., above image).

My question is: What does Trainer.predict.label_ids mean? Why the output I got from this argument produced a perfect prediction results which was obviously wrong?

Thank you in advance

2

Hello, I too naïvely thought that Trainer.predict.label_ids were the predicted labels. But they are the actual labels, as we can read in this tutorial. You need to actually retrieve the predicted labels by yourself using np.argmax(test_prediction.predictions, axis=-1). As we can read later on in the tutorial, metrics are computed like so:

metric = load_metric("glue", "mrpc")
metric.compute(predictions=preds, references=predictions.label_ids)

Which confirms that label_ids are the actual labels indeed. I also find this very confusing.