How to get around rate limits?

Beginners
16

It seems like it failed even faster when using ‘upload’ rather than ‘upload-large-folder’

When you say run it on parts of the repo one by one… OK but how do I ensure it’s uploaded to the right path?

If the repository looks like this:

├── 2017/              # 2017 release
│   ├── part_1/        # 2017 part 1 
│   ├── part_2/        # 2017 part 2
│   ├── part_3/        # 2017 part 3
│   ├── part_4/        # 2017 part 4
│   └── part_5/        # 2017 part 5 (originally labeled "additional")
├── 2018/              # 2018 release
│   ├── part_1/        # 2018 part 1
│   └── part_2/        # 2018 part 2

Do use huggingface-cli like this if I want to start with the 2017 subfolder?

huggingface-cli upload mysocratesnote/jfk-files-text/2017 ~/Desktop/extracted_text/releases/2017 --repo-type=dataset

Thanks.

1 Like