Custom metrics with extra data?

Beginners
#1

I have a multi class model that I want to evaluate (during training on the eval set using additional data that is contained in the dataset. The extra data is a column that provides a grouping - a document id.

These are options I am considering.

  • The compute_metric function you can pass into the Trainer is only passed in an EvalPrediction which contains the predictions and labels, not the extra data, so this is not possible.
  • I can subclass the Trainer and then override the evaluation_loop. It would be a very much a copy paste effort with overriding the call to compute_metrics to pass in extra data. I don’t know if this would also affect the training time?
  • I can utilise a callback at the end of epoch and calculate the metrics I want outside of the evaluation loop. I’m not sure whether I will have access to all data here though.

Any other options that I am missing? What is the best way of accessing extra data in the evaluation metrics?

#2

cc @lvwerra do you have an idea ?

#3

Thanks for calling attention to it, I ended up choosing the last option, implement an end of epoch override. Worked well, unfortunately, it also meant I had to implement saving the best model myself too as it’s the same “metric” being used.

#4

An alternative way might be to either define a global variable or create something like a compute_metric factory:

def compute_metric_with_extra(extra_data):
    def compute_metric(preds, refs):
        # do something with preds, refs and extra_data
        return score
    return compute_metric

compute_metric = compute_metric_with_extra(extra_data)

You should be able to pass this function to the trainer. Sorry I was too late :slight_smile: