Does hf_transfer Help Speed Up the datasets Library?

ryan-minato · February 12, 2025, 8:57am

It seems that the datasets library does not perform parallelization when uploading datasets.
Would setting the environment variable %env HF_HUB_ENABLE_HF_TRANSFER=0 help speed up datasets’s push_to_hub?

John6666 · February 12, 2025, 11:39am

There don’t seem to be any other options that could be used to speed things up…
upload_large_folder requires a local folder to be present first.

github.com/huggingface/datasets

src/datasets/arrow_dataset.py

3.2.0


      
                      repo_id=repo_id,
                      additions=[shard_addition],
                      repo_type="dataset",
                      revision=revision,
                      create_pr=create_pr,
                  )
                  additions.append(shard_addition)
          
              return additions, uploaded_size, dataset_nbytes
          
          def push_to_hub(
              self,
              repo_id: str,
              config_name: str = "default",
              set_default: Optional[bool] = None,
              split: Optional[str] = None,
              data_dir: Optional[str] = None,
              commit_message: Optional[str] = None,
              commit_description: Optional[str] = None,
              private: Optional[bool] = None,
              token: Optional[str] = None,

Topic		Replies	Views
Uploading a dataset that doesn't fit in memory to the HF hub 🤗Datasets	5	60	October 24, 2024
Incrementally adding processed examples to a dataset 🤗Datasets	4	1288	June 23, 2022
How to convert dir-with-images properly? 🤗Datasets	2	386	June 11, 2024
Datasetdict push_to_hub failing with payload to large 🤗Datasets	6	44	February 11, 2025
Any workaround for push_to_hub() limits? 🤗Datasets	9	1972	May 2, 2024

Does hf_transfer Help Speed Up the datasets Library?

Related topics