Model works but MultiDataModel doesn't

Amazon SageMaker
1

This is a continuation of my post here. I’m trying to deploy BERT for text classification with tensorflow. When I use the model.deploy() method, I can successfully get inferences from BERT. Here’s my problem: I have four different models for classification and I want to run them on the same instance, not multiple instances, to save on cost. So I tried using the MultiDataModel class, but I keep getting the following error:

image
image979×1189 72.2 KB

The CloudWatch logs don’t add any additional information, unfortunately. Here’s the structure of counterargument.tar.gz in the s3 bucket, which I cloned from my HuggingFace account and zipped.

counterargument.tar.gz
    config.json
    special_tokens_map.json
    tf_model.h5
    tokenizer.json
    tokenizer_config.json
    vocab.txt

The most puzzling thing about this error is that model.deploy() worked fine, but multi_model.deploy() doesn’t! Thanks in advance.

2

@wsunadawong when using Multi-Model-Endpoints SageMaker stores the models differently. That’s why model.deploy() seems to work and MME does not. We (Amazon & HF) are looking into it. Hopefully, we can come back with a fix as soon as possible!

1 Like
3

Could you share logs from the endpoint’s cloudwatch logs? Also, the counterargument.tar.gz is a flat directory or does that tar have a directory inside which these files exist?

4

The counterargument.tar.gz has a flat directory structure. I git cloned my huggingface repo and just zipped it. Here’s a picture of the contents:

image
image706×577 21.9 KB

5

And here’s a picture of the logs

image
image1082×1046 68.4 KB

7

Could you share more of the log. I see that “Model model loaded” message. I want to understand where invocation failed.

8

Here’s the logs which were above what I sent. There’s nothing useful below–just ping messages.

image
image1426×959 42.1 KB

9

Oh my mistake, there were more non-ping logs below. Is this what you were looking for?

image
image1413×1175 246 KB

10

The container seems to be using default model directory to lookup for model instead of MME platforms model directory. We will try to reproduce this and get back.

3 Likes
11

Any news on this by any chance?

12

Hey @Ilias,

The MultiDataModel was solved. The model.tar.gz used was wrong. You can take a look here.