Deploy big model to AWS Sagemaker fails

Beginners
1

Hello,
I’m trying to deploy a model on AWS Sagemake (mosaicml/mpt-7b-chat · Hugging Face)

I used the code generated by the model’s page for a quick start, it is as follow:

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'mosaicml/mpt-7b-chat',
    'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.17.0',
    pytorch_version='1.10.2',
    py_version='py38',
    env=hub,
    role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
    initial_instance_count=1, # number of instances
    instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
    'inputs': "Can you please let us know more details about your "
})

The code crashes at the “predict” call with this error: Loading /.sagemaker/mms/models/mosaicml__mpt-7b-chat requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code\u003dTrue to remove this error.

I had the same issue locally that I fixed by adding the " trust_remote_code=True" parameter to the AutoModelForCausalLM.from_pretrained constructor, but that doesn’t seem to exist with HuggingFaceModel constructor.

Any way to fix this? thank you

2 Likes
2

I had the same issue! Were you able to fix this?

3

I am facing the same issue. Any input on how to fix it?

4

Anyone with any solution for this? Seems like it is still not resolved

5

this medium article allowed me to deploy successfully Deploy MosaicML MPT-7B-Instruct on SageMaker | by Manoranjan Rajguru | May, 2023 | Medium

6

Was anyone able to fix this issue? Looks like this issue exists for all mosaicml/mpt-7b-* models. Is that right?