504 Gateway Time-out when uploading large dataset to enterprise organization

🤗Datasets
1

Hi

We recently bought an enterprise subscription in order to increase our data usage limits.

However, it is no longer possible for me to push large datasets to the hub (500+ GB), as it eventually crashes with a “504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/{organization}/{dataset}/preupload/main”

I don’t know if this have anything to do with your recent change in data usage limits policy. I see that you write that you have to contact you in order to upload datasets larger than 300 GB, but I assumed that was for public datasets uploaded to free accounts/organizations.

I currently cannot use the main functionality which made me buy the enterprise subscription.

I’m pushing the dataset using DatasetDict.push_to_hub, so it automatically splits the dataset into reasonable parquet files and uploads them one at a time. However, since it doesn’t complete I have to start over instead of being able to resume the upload.

1 Like
2

@meganariley Paid service issue.

3

Any update?

If this isn’t the place for priority support which I’m paying for, please tell me.

1 Like
4

The official route is basically email. In some cases, HF Discord or the forum may be faster, but this is the exception.
website@huggingface.co

1 Like