Maybe push_to_hub() is writing some temporary file(s) to /tmp, which runs out of space?
Try changing the location for the temporary files by putting TMPDIR=/mnt/tmp in front of the command you are running (it seemed that you are working in /mnt, which has much space, but some other location could be better for the temp files). See TMPDIR - Wikipedia
Edit: Or actually is the partition that is used by HFH cache filling up? In that case it could be relocated with HF_HOME or HF_HUB_CACHE envs: Manage huggingface_hub cache-system