I am training a model and am hoping to get the log of the per-evaluation-step performance while training. I currently set --evaluation_strategy steps and I could see logs got printed. However, is there a way I could get the table as an object (json or pd.DataFrame) in Python?
Out of curiosity, how does the load_best_model_at_end parameter work? If it loads the best model based on the x times evaluation over training, then there must be something like a best_step stored somewhere?
Hi @mralexis, as described in the docs, load_best_model_at_end works in conjunction with metric_for_best_model.
Under the hood, the Trainer keeps track of the best checkpoint seen to date via the Trainer.state.best_model_checkpoint attribute (see here) which is then used by load_best_model_at_endhere.
@lewtun Just to be sure, so the Trainer object does not keep track of the metrics such that we as end-users can access them as well? Writing a callback for this feels a bit cumbersome, as all we want is already displayed every step/epoch by the Trainer object.
I do not know how hugging face stores these metrics internally, but to me it seems the most logical that it keeps the metrics in memory somewhere and re-displays the entire dataframe after each step. But I could be wrong here.
