Hi ! Amazing to have an image-text datasets like that, so you plan to share it with the community ?
This exceeds the rate limit for github commits.
Do you mean that you got an error while uploading ? Could you share the error message ?
If I push it like:
dataset.push_to_hub(dataset_id, num_shards=5)
It throws an error:
Any idea how I can oversome this?
You can try increasing the max shard size instead:
dataset.push_to_hub(dataset_id, max_shard_size="2GB")
This should upload less files than by default and still have shards of reasonable size