@sgugger Thank you for the reply! Actually, what I’ve found is that push_to_hub=True launches a background git push. I think the problem is that sometimes the git push process hangs. Is there any way to debug that git process, to see why it does not finish pushing?
Also, just for context, I can sometimes go into the directory and use git push with success. But also, I’ve found that only the first checkpoint produces a git commit. While trying to debug the background git push, I noticed that sometimes there can be 3 or 4 checkpoints created, but no commits happening for them. So only the first checkpoint gets pushed.
Let me know if there’s any extra info I can provide. I might try and copy one of the run_*.py scripts and writing some custom Python code to solve this problem.