I am training a model while performing hyperparameter optimization with optuna, therefore I can’t afford saving a lot of checkpoints in each trial because my machine would run out of disk. For this reason in each trial I am setting save_total_limit=2. This means that the two checkpoints with less validation loss will be saved for each trial. How can I do if I want to save the two checkpoints which have best f1 instead of the least validation loss? Is there any parameter or I have to inherit Trainer and make my own code?
I don’t know if there is a way to do that but there is load_best_model_at_end=True. But I’m not quite sure if it is something you want. Maybe see here: Save only best model in Trainer - #5 by sgugger