Is it possible to push_to_hub at every checkpoint?

Lookup the hub_strategy=checkpoint setting.
This will save only the latest checkpoint, and also in the end.

There is also the all_checkpoints option to store all checkpoints