Does hf_transfer Help Speed Up the datasets Library?

It seems that the datasets library does not perform parallelization when uploading datasets.
Would setting the environment variable %env HF_HUB_ENABLE_HF_TRANSFER=0 help speed up datasets’s push_to_hub?

1 Like

There don’t seem to be any other options that could be used to speed things up…
upload_large_folder requires a local folder to be present first.