Huggingface_hub.client giving error on list_deployed_models()

John6666 · March 2, 2025, 8:38pm

I think the URL for the reference is no longer valid. Perhaps the library hasn’t been updated with the new URL?

There doesn’t seem to be an issue, but it might be fixed eventually.

github.com/huggingface/huggingface_hub

Add `list_deployed_models` to list models deployed on InferenceAPI

opened 03:04PM - 11 Jul 23 UTC

closed 09:37AM - 08 Sep 23 UTC

Wauplin

good first issue

The new `InferenceClient` allows to run inference on the cloud. For each impleme…nted task, new users that don't know which model to use can use the recommended model on the (free) Inference API (implemented in https://github.com/huggingface/huggingface_hub/pull/1510). Since the recommended model might not be suitable for all use cases, it is also possible to specify any model as input to the `InferenceClient`. While most models on the HuggingFace Hub is compatible with Inference API, waiting for the models to load can take a lot of time. Also, for large models the Inference API just cannot load them. To ease user experience, it would be nice to have a utility `list_deployed_models` that would return a list of already deployed models for the user to try. ```py class InferenceClient: ... def list_deployed_models(self, *, token: Optional[str] = None) -> Dict[str, List[str]]: ... return {"audio-to-audio": ["microsoft/speecht5_vc", ...], "text-generation": [...], ...} ``` This helper should help with discoverability. --- Implementation-wise, this list of deployed models can be retrieved using the `"https://api-inference.huggingface.co/framework/{framework_name}"` where "framework_name" is a framework compatible with the Inference API. For example: - https://api-inference.huggingface.co/framework/transformers - https://api-inference.huggingface.co/framework/diffusers - https://api-inference.huggingface.co/framework/text-generation-inference - https://api-inference.huggingface.co/framework/sentence-transformers Around ~30 frameworks are available but I think we should first focus on the main frameworks. Each url returns a list of dictionaries. Each item is a deployed model: `{"compute_type":"cpu","model_id":"microsoft/speecht5_vc","sha":"c418ba2144598f973d0fddc9fd5909a3af83de3d","task":"audio-to-audio"}`. What is interesting for us is `"model_id"` and `"task"`. What a `list_deployed_models` method would do is to loop through all main framework, loop though all items and build an output dictionary with 'task' as key and a list of model ids as value. Example: ```py >>> from huggingface_hub import InferenceClient >>> client = InferenceClient() >>> client.deployed_models()["audio-to-audio"] ["microsoft/speecht5_vc", ...] ``` Happy to discuss the API design and update it if needed :) We still have to figure out what to do if several "sha" are deployed for a single model.

github.com/huggingface/huggingface_hub

src/huggingface_hub/inference/_client.py

v0.29.1


      
                      if framework == "sentence-transformers":
                          # Model running with the `sentence-transformers` framework can work with both tasks even if not
                          # branded as such in the API response
                          models_by_task.setdefault("feature-extraction", []).append(model["model_id"])
                          models_by_task.setdefault("sentence-similarity", []).append(model["model_id"])
                      else:
                          models_by_task.setdefault(model["task"], []).append(model["model_id"])
          
              for framework in frameworks:
                  response = get_session().get(
                      f"{constants.INFERENCE_ENDPOINT}/framework/{framework}", headers=build_hf_headers(token=self.token)
                  )
                  hf_raise_for_status(response)
                  _unpack_response(framework, response.json())
          
              # Sort alphabetically for discoverability and return
              for task, models in models_by_task.items():
                  models_by_task[task] = sorted(set(models), key=lambda x: x.lower())
              return models_by_task
          
          def get_endpoint_info(self, *, model: Optional[str] = None) -> Dict[str, Any]:

Topic		Replies	Views
List models accessible via InferenceClient? Inference Endpoints on the Hub	1	90	April 9, 2025
Models with inference_client Beginners	1	64	January 28, 2025
What models are available in the Inference API? Beginners	3	1796	July 21, 2024
Example Inference API (model & code ), pls Beginners	5	39	June 28, 2025
List_models() using full-text search of hf-internal-testing 🤗Hub	0	91	June 25, 2024

Huggingface_hub.client giving error on list_deployed_models()

Related topics