Is it possible to include sample weights in the compute_metrics function used by Trainer?
For simpler cases, I can just encode the gold labels for some samples as -1 and give them a weight of 0 when computing the metrics, but in other cases it might be trickier.
I don’t think it’s possible. For elaborate evaluations you can pick on your predictions after using trainer.predict then do something more sophisticated on them.