I was wondering if it is possible to pass additional labels into metric evaluation function. In more details, I want to calculate metrics using not all predicted tokens but only a small subset, and to do this I need to pass a tensor of indices of that tokens into the metric evaluation function. But I don’t need this tensor for anything else, e.g. for calculating the loss.
I tried to add this tensor into evaluation dataset, using
remove_unused_columns = False options. But in this case the model (T5 encoder-decoder) sees unrecognized keywords and fails. Default
remove_unused_columns = True just deletes the tensor from the batch.
I am using Transfomers version 4.7.0
Any help is much appreciated! Thanks.