It seems that the datasets
library does not perform parallelization when uploading datasets.
Would setting the environment variable %env HF_HUB_ENABLE_HF_TRANSFER=0
help speed up datasets
’s push_to_hub
?
1 Like
There don’t seem to be any other options that could be used to speed things up…
upload_large_folder requires a local folder to be present first.