I am trying to push a large dataset with help of
dataset.push_to_hub()
While pushing, it gives HTTP error :504
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/name/repo-name/upload/main/data/train2-00041-of-00064.parquet
How to avoid this whenever pushing large datasets
Hi ! As discussed on GitHub, feel free to try again (the server might have had some issues).
We’re also adding a retry mechanism to workaround 504 errors: Retry HfApi call inside push_to_hub when 504 error by albertvillanova · Pull Request #3886 · huggingface/datasets · GitHub
We’ll do a new release of datasets
soon to include this 
1 Like