Upload_large_folder() issue with uploading to spaces

DF38 · December 5, 2024, 4:16am

I am trying to use upload_large_folder() with HfApi to upload some large data to a space however I keep getting an error related to not specifying the sdk when calling create_repo(), which happens under the hood.

Any insight?

John6666 · December 5, 2024, 5:48am

It is optional, but if you don’t specify it, an error will occur. Let’s specify it.

github.com

huggingface/huggingface_hub/blob/v0.26.3/src/huggingface_hub/hf_api.py#L3478


      
          
          path = f"{self.endpoint}/api/repos/create"
          
          if repo_type not in constants.REPO_TYPES:
              raise ValueError("Invalid repo type")
          
          json: Dict[str, Any] = {"name": name, "organization": organization, "private": private}
          if repo_type is not None:
              json["type"] = repo_type
          if repo_type == "space":
              if space_sdk is None:
                  raise ValueError(
                      "No space_sdk provided. `create_repo` expects space_sdk to be one"
                      f" of {constants.SPACES_SDK_TYPES} when repo_type is 'space'`"
                  )
              if space_sdk not in constants.SPACES_SDK_TYPES:
                  raise ValueError(f"Invalid space_sdk. Please choose one of {constants.SPACES_SDK_TYPES}.")
              json["sdk"] = space_sdk
          
          if space_sdk is not None and repo_type != "space":
              warnings.warn("Ignoring provided space_sdk because repo_type is not 'space'.")

DF38 · December 5, 2024, 6:00am

Yes! Could you explain how I could specify it? Thanks for the response!

John6666 · December 5, 2024, 6:14am

There doesn’t seem to be a solution. The workaround would be to create_repo first. I’ll raise an issue if not exists.

John6666 · December 5, 2024, 6:35am

I raised an issue.

github.com/huggingface/huggingface_hub

upload_large_folder() issue with uploading to new spaces (create_repo requires space_sdk)

opened 06:33AM - 05 Dec 24 UTC

John6666cat

bug

### Describe the bug from HF Forum: https://discuss.huggingface.co/t/upload-…large-folder-issue-with-uploading-to-spaces/129326 If you run `upload_large_folder()` with a non-existent `space`, you will get an error from `create_repo()`. This is because [`create_repo`()'s `space_sdk` is `optional`, but if it is not there, an error will occur](https://github.com/huggingface/huggingface_hub/blob/v0.26.3/src/huggingface_hub/hf_api.py#L3478). https://github.com/huggingface/huggingface_hub/blob/v0.26.3/src/huggingface_hub/_upload_large_folder.py#L92 ### Reproduction HF_TOKEN = "hf_*********" from huggingface_hub import HfApi api = HfApi(token=HF_TOKEN) api.upload_large_folder("John6666/lftest", folder_path="test_folder", repo_type="space", private=True) ### Logs ```shell File "w:\TEMP\test\upload_large_folder_test.py", line 4, in <module> api.upload_large_folder("John6666/lftest", folder_path="test_folder", repo_type="space", private=True) File "c:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\huggingface_hub\hf_api.py", line 5473, in upload_large_folder return upload_large_folder_internal( File "c:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\huggingface_hub\_upload_large_folder.py", line 92, in upload_large_folder_internal repo_url = api.create_repo(repo_id=repo_id, repo_type=repo_type, private=private, exist_ok=True) File "c:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "c:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\huggingface_hub\hf_api.py", line 3479, in create_repo raise ValueError( ValueError: No space_sdk provided. `create_repo` expects space_sdk to be one of ['gradio', 'streamlit', 'docker', 'static'] when repo_type is 'space'` ``` ### System info ```shell - huggingface_hub version: 0.26.2 - Platform: Windows-10-10.0.19045-SP0 - Python version: 3.9.13 - Running in iPython ?: No - Running in notebook ?: No - Running in Google Colab ?: No - Running in Google Colab Enterprise ?: No - Token path ?: w:\hf\misc\token - Has saved token ?: False - Configured git credential helpers: manager - FastAI: N/A - Tensorflow: N/A - Torch: 2.4.0+cu124 - Jinja2: 3.1.4 - Graphviz: N/A - keras: N/A - Pydot: N/A - Pillow: 11.0.0 - hf_transfer: N/A - gradio: 4.44.1 - tensorboard: 2.6.2.2 - numpy: 1.23.5 - pydantic: 2.8.2 - aiohttp: 3.9.5 - ENDPOINT: https://huggingface.co - HF_HUB_CACHE: w:\hf\misc\hub - HF_ASSETS_CACHE: w:\hf\misc\assets - HF_TOKEN_PATH: w:\hf\misc\token - HF_STORED_TOKENS_PATH: w:\hf\misc\stored_tokens - HF_HUB_OFFLINE: False - HF_HUB_DISABLE_TELEMETRY: False - HF_HUB_DISABLE_PROGRESS_BARS: None - HF_HUB_DISABLE_SYMLINKS_WARNING: False - HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False - HF_HUB_DISABLE_IMPLICIT_TOKEN: False - HF_HUB_ENABLE_HF_TRANSFER: False - HF_HUB_ETAG_TIMEOUT: 10 - HF_HUB_DOWNLOAD_TIMEOUT: 10 ```

Topic		Replies	Views
Upload Error when api.upload_file Spaces	2	745	January 25, 2023
Api.upload_folder() 🤗Hub	0	256	February 18, 2024
Cannot upload by python api Spaces	0	401	August 21, 2023
HfApi uploading fail Beginners	0	106	April 14, 2024
Cannot upload large file to huggingface Beginners	1	929	June 14, 2023

Upload_large_folder() issue with uploading to spaces

Related topics