The problem is not the total size but the number of files (around 70k+ in total?). upload-large-folder was not meant for that at first (my bad, I designed it mostly to upload folders with hundreds of large files instead of folders with tens of thousands of small files). Result is that we are committing them by chunk of 50 which is making hundreds of commits which triggers the rate limit. Fix dynamic commit size by maximizemaxwell · Pull Request #3016 · huggingface/huggingface_hub · GitHub is meant as a good workaroudn for that but it’s not finished yet.
In the meantime I don’t have many suggestions except making the upload more manual (i.e. running huggingface-cli upload on subparts of the repo)