Clarification on push_to_hub, best model, and model card

🤗Hub
1

Hi,

I’m trying to solve a doubt that I saw a few other people have asked about, but couldn’t find an answer.

The question is quite simple: when using trainer.push_to_hub() together with the load_best_model_at_end argument, is the trainer pushing the last or the best model?

My doubt is raised by the fact that the automatically-created model card reports the selected score metric for the last epoch, instead of the one obtained at the best one.

I am reasonably sure is not an issue of loading the best model, since I’ve tested via trainer.evaluate() that after training the trainer is, in fact using the best-obtained model.

Thank you so very much for any help/clarification, and the great work you’re all doing!

4 Likes
2

Support your question!
I’ve checked my model after pushing it to Hub and it is true that the trainer pushes the best model.
However, the automatically-created model card is indeed misleading, reporting the wrong (last) score and number of training steps.

I think this issue should be addressed by the team!

3 Likes
3

Would be nice to have this fixed, now that several courses offered by HF are relying on the metrics reported on the automatically created model cards

2 Likes
4

I’ve found a workaround to push the trainer with an updated model card version, evaluating the trainer before pushing it:

trainer.evaluate()
trainer.push_to_hub()

Here is the detail of what I’ve found:

Summary

The trainer.push_to_hub method calls the trainer.create_model_card which itself calls the classmethod TrainingSummary.from_trainer on trainer. This one instanciate a TrainingSummary which eval_results attribute comes from the last element returned by the modelcard.parse_log_history method on trainer.state.log_history.

This attribute is updated once in the code with the method trainer.log which is called only with trainer._inner_training_loop and trainer.evaluate (there is another one that is only used in the former).

As its name indicates it, the first method is called during the execution of trainer.train so the last evaluation results written in the model card would be the evaluation results from the last epoch. Updating those results can be done by appending a log to the trainer.state.log_history calling the method trainer.evaluate.

1 Like