I’m trying to deploy multiple BERT models in one container behind one endpoint using the boto3 API.
If we unzip the model.tar.gz file then we have inference code, model artifacts in below structure.
I observed that dependency libraries were not being installed when SageMaker started setting up the container in cloud watch logs.
Hence I’m getting the below error.
ModuleNotFoundError: No module named ‘spacy’
Note: I followed the above structure to deploy a single model endpoint and it works fine.
Looks like there is a different structure we need to follow in case of multi-model endpoint deployment.
Could you please help me with this issue?
Can you please provide more information on how you create the
zip file and how you are deploying the endpoint? Also what is the content of your
requirements.txt? it looks like you want to use
spacy in the inference script but it is not installed
As requested, please find the details below.
- Creating zip file: As shown in above image, “model” directory contains model artifacts and “code” directory contains inference code and requirements.txt file.
- Contents of requirements.txt file
Could you try to create the
model.tar.gz following the steps in this notebook? notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHub
tar.add(model_path) is creating a wrong structure with a nested
model/ directory but the artifacts need to be on top level.