How to get around rate limits?

John6666 · April 23, 2025, 1:16pm

That’s strange. I don’t think the numbers are high enough to cause an error… @Wauplin

github.com/huggingface/huggingface_hub

429 when uploading many files

opened 10:04PM - 01 Oct 24 UTC

closed 08:19AM - 15 Oct 24 UTC

fakerybakery

bug

### Describe the bug Hi, I'm using the experimental `upload-large-folder` feat…ure. I'm trying to upload ~2000 files, but I'm running into a 429 error. Would it be possible to increase the rate limit when using the `upload-large-folder` feature? Thanks! ### Reproduction _No response_ ### Logs _No response_ ### System info ```shell - huggingface_hub version: 0.25.1 - Python version: 3.12.5 - Running in iPython ?: No - Running in notebook ?: No - Running in Google Colab ?: No - Running in Google Colab Enterprise ?: No - Has saved token ?: True - Configured git credential helpers: store - FastAI: N/A - Tensorflow: N/A - Torch: 2.4.1 - Jinja2: 3.1.4 - Graphviz: N/A - keras: N/A - Pydot: N/A - Pillow: 10.4.0 - hf_transfer: 0.1.8 - gradio: N/A - tensorboard: N/A - numpy: 1.26.4 - pydantic: N/A - aiohttp: 3.10.8 - ENDPOINT: https://huggingface.co - HF_HUB_OFFLINE: False - HF_HUB_DISABLE_TELEMETRY: False - HF_HUB_DISABLE_PROGRESS_BARS: None - HF_HUB_DISABLE_SYMLINKS_WARNING: False - HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False - HF_HUB_DISABLE_IMPLICIT_TOKEN: False - HF_HUB_ENABLE_HF_TRANSFER: False - HF_HUB_ETAG_TIMEOUT: 10 - HF_HUB_DOWNLOAD_TIMEOUT: 10 ```

github.com/huggingface/huggingface_hub

upload_folder() fails to upload large datasets

opened 01:00PM - 15 Feb 24 UTC

WesleyHsieh0806

### Describe the bug I tried to use upload_folder to upload my dataset with ~36…0 GB image frames, but kept failing with the following error: `huggingface_hub.utils._errors.HfHubHTTPError: 413 Client Error: Payload Too Large for url` ### Steps to reproduce the bug The code for reference: For details regarding the `frames` folder structure, you can check [TAO](https://motchallenge.net/tao_download.php) ```python from huggingface_hub import upload_folder from huggingface_hub import login login() upload_folder( folder_path="TAO/frames", repo_id="chengyenhsieh/TAO-Amodal", path_in_repo="frames", repo_type="dataset", multi_commits=True, multi_commits_verbose=True, ) ``` ### Expected behavior Image frames are successfully updated to my Huggingface Dataset [Repo](https://huggingface.co/datasets/chengyenhsieh/TAO-Amodal) ### Environment info datasets 2.17.0 python 3.8

Topic		Replies	Views
Upload dataset rate limit 🤗Datasets	2	928	May 31, 2024
Too many requests for URL 🤗Hub	5	5087	May 25, 2025
429 client error when uploading large models (65B) Models	0	476	March 5, 2024
Failed to commit 504 Server Error Gateway Time-out for url Beginners	1	83	December 26, 2024
Error while uploading files larger than 10Mb 🤗Datasets	1	939	October 10, 2022

How to get around rate limits?

Related topics