Save and deploy distilbert model in AWS SageMaker

🤗Transformers
1

I am trying to download the Hugging Face distilbert model, trying to save to S3. The model itself does not have a deploy method. So I am saving to S3, instantiating it and trying to deploy. May I know if this will work with Sagemaker. What I am doing wrong. Here are the steps:

model_name = ‘distilbert-base-uncased-distilled-squad’ model = DistilBertForQuestionAnswering.from_pretrained(model_name) tokenizer = DistilBertTokenizerFast.from_pretrained(model_name)

the below works and gives output context = “xxxx” question = “yyy?”

nlp = pipeline(‘question-answering’, model=model, tokenizer=tokenizer)

nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })

save the model to local folder model.save_pretrained(’./scripts/mymodel’)

zip the model file. with tarfile.open("./scripts/mymodel/model.tar.gz", “w:gz”) as tar: tar.add("./scripts/mymodel/pytorch_model.bin") tar.add("./scripts/mymodel/config.json")

upload the zipped file to S3 sagemaker.Session().upload_data(bucket=sagemaker_session_bucket, path=’./scripts/mymodel/model.tar.gz’, key_prefix=‘model’)

instantiating the saved model

bertmodel = PyTorchModel(entry_point=‘inference.py’, source_dir=‘scripts’, model_data=‘s3://’+sagemaker_session_bucket+’/model/model.tar.gz’, role=sagemaker.get_execution_role(), framework_version=‘1.5’, py_version=‘py3’)

the below does not work nlp = pipeline(‘question-answering’, model=bertmodel, tokenizer=tokenizer)

nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })

error recd - AttributeError: ‘PyTorchModel’ object has no attribute ‘config’

I am able to deploy the predictor predictor = bertmodel.deploy(initial_instance_count=1, instance_type=‘ml.m5.xlarge’)

2

@gopalkr272 I am afraid I don’t have a complete answer but review my support ticket here:
https://github.com/huggingface/transformers/issues/11043
I hope this will help you find an answer.

3

There are couple public samples showing Hugging Face model deployment on SageMaker, for example those:

1 Like