Deploying Huggingface Sagemaker Models with Elastic Inference

When I try to deploy a HuggingFace Sagemaker model with elastic inference (denoted by the accelerator_type parameter) I get an error.

Deploy Snippet:

predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.t2.medium",
    accelerator_type='ml.eia2.medium'
)

Error Msg:

~/miniconda3/envs/ner/lib/python3.8/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name)
    305     """Checks if the arg is in the available options, and raises a ``ValueError`` if not."""
    306     if arg not in available_options:
--> 307         raise ValueError(
    308             "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version "
    309             "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): "

ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference.

The model deploys successfully if I do not provide an accelerator (i.e., no Elastic Inference).

Do the HuggingFace Sagemaker models support EI? If yes, how might I deploy the model successfully with EI? And if not, is EI support on the roadmap?

Much thanks in advance! :smile:

1 Like

Hey @schopra,

Sadly speaking we don’t have EI DLCs yet. We are working on it and it is on the roadmap with one of the highest priorities.
I would update this thread here when I got any news.

3 Likes

Is there by any chance a list of supported instances at this time? Thanks!

Hey @ujjirox,

supported instances for what? Training or Inference or both? You can find an overview of supported instances type for sagemaker here: Amazon SageMaker Pricing – Amazon Web Services (AWS)

Sorry! Should have been more clear. I meant for inference. I actually had tried running inference with ml.inf1.xlarge but it didn’t seem to work, hence the question.

Thanks.

Hey @ujjirox,

Inferentia is also not yet supported, since we need to create a separate DLC for the Inferentia instances, but we are on it.
Other than this every CPU / GPU machine should be supported.

1 Like