I’ve been getting this today (200GB dataset, push in python), just love to raise this is still a problem (and it’d be so great to be able to resume pushes since they can take a really long time)
Creating parquet from Arrow format: 100%|██████████████████████████████████████████████████| 48/48 [00:00<00:00, 329.12ba/s]
Uploading the dataset shards: 46%|██████████████████████▊ | 192/420 [6:08:02<7:17:03, 115.01s/it]
Traceback (most recent call last):
File “/Users/dmackparty/dev/presto-py/app/pipelines/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_http.py”, line 409, in hf_raise_for_status
response.raise_for_status()
File “/Users/dmackparty/dev/presto-py/app/pipelines/.venv/lib/python3.12/site-packages/requests/models.py”, line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/complete_multipart?uploadId=ak3yajyFaNpfIpmsLljywyUVb1sNay5D5GUGKAjJcM1h2GjHSHWLUuXNQ2.rxlPK9Mydu5w.5iCJ5P9SRIw4tbO2Gk9bXPRqPshRZlskHKa.tpGImodDyUjcU1yH92La&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fbe%2Feb%2Fbeebea985423ba0ccb6c7ef0c6925225a4dcd28c85e8c4db44607f457a00ce2d&expiration=Tue%2C+04+Mar+2025+06%3A40%3A19+GMT&signature=3fe1b5f59f4c97236b73dc288be78fd3d1802a8fe42fc10db5d3342a8854fd50