Deploying Huggingface Sagemaker Models with Elastic Inference

When I try to deploy a HuggingFace Sagemaker model with elastic inference (denoted by the accelerator_type parameter) I get an error.

Deploy Snippet:

predictor = huggingface_model.deploy(

Error Msg:

~/miniconda3/envs/ner/lib/python3.8/site-packages/sagemaker/ in _validate_arg(arg, available_options, arg_name)
    305     """Checks if the arg is in the available options, and raises a ``ValueError`` if not."""
    306     if arg not in available_options:
--> 307         raise ValueError(
    308             "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version "
    309             "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): "

ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference.

The model deploys successfully if I do not provide an accelerator (i.e., no Elastic Inference).

Do the HuggingFace Sagemaker models support EI? If yes, how might I deploy the model successfully with EI? And if not, is EI support on the roadmap?

Much thanks in advance! :smile:

1 Like

Hey @schopra,

Sadly speaking we don’t have EI DLCs yet. We are working on it and it is on the roadmap with one of the highest priorities.
I would update this thread here when I got any news.


Is there by any chance a list of supported instances at this time? Thanks!

Hey @ujjirox,

supported instances for what? Training or Inference or both? You can find an overview of supported instances type for sagemaker here: Amazon SageMaker Pricing – Amazon Web Services (AWS)

Sorry! Should have been more clear. I meant for inference. I actually had tried running inference with ml.inf1.xlarge but it didn’t seem to work, hence the question.


Hey @ujjirox,

Inferentia is also not yet supported, since we need to create a separate DLC for the Inferentia instances, but we are on it.
Other than this every CPU / GPU machine should be supported.

1 Like

Hey, any news regarding the EI DLC/ INF DLC?

Hey @asafab,

Yes, i already opened a PR for INF DLCs:
You can follow it here: [huggingface_pytorch][NEURON][build] Huggingface Neuron inference DLC by philschmid · Pull Request #1578 · aws/deep-learning-containers · GitHub

When it is merged and available we will additionally share on social media + provide an example.

very cool!
thanks for the response, you have news about the EI DLC too?

1 Like