Hi, all. I’m new to Transformers. I design my own compute metric for Trainer. Due to large eval dataset, I cannot load evalaute the dataset for one batch. I want to collect the all right prediction num of my eval dataset. But it seems that Trainer use a mean aggregate function for all batches. Could I change it to sum aggregation?
1 Like
You can choose when to execute the evaluation: every time, every epoch, or not at all.
If you do it every epoch, it should be manageable.
Alternatively, you could use a less elegant method of keeping the count in a class variable or global variable, and only executing the evaluation and resetting the counter to 0 when the count reaches 10.