Running generate while evaluating test set?

I’m trying to finetune a T5 model using Huggingface Trainer class which does simple evaluation and calculates the loss.

I want to run generations upto certain length at end of each evaluation on test set and store it in a file. I think I would need to override trainer to create a CustomTrainer , but I’m not sure how can I do it.

Could anyone help?