Top-5 (k) Accuracy Score in Multi Class Single Label

🤗Evaluate
1

I have a multi class single label text classification problem (i.e. lots of classes, only 1 is correct). The following code snippet calculates the “top-1 accuracy” score. However, I require “top-5 accuracy” (or in deed k) as well to be reported. Since the only input is eval_pred I am not sure how I can add this. Can someone please assist or point me toward the docs that show this (have looked cannot find)

def compute_metrics(eval_pred):
    metric = evaluate.load("accuracy")
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)
2

bump bump still waiting for assistance

3

You would either need to add a new metric to evaluate or you could use Scikit-learn’s top_k_accuracy.

4

you can use this compute_metrics func instead:

top_k = 5
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    preds = np.argsort(-predictions)[:,0:top_k]
    acc_at_k = sum([l in p for l, p in zip(labels, preds)])/len(labels)
    return {'acc_at_k': acc_at_k}