Missing schema for sagemaker deployment of zero-shot image classification models

Amazon SageMaker
1

Trying to get predictions on a deployed zero-shot image classification model and it appears as if the schema for the .predict() method is missing from the documentation. I have followed the schema example from the zero-shot text classification, but have changed the text input to an image and it doesn’t seem to be working. The model has been been deployed to an endpoint successfully, but fails when trying to produce an inference. I’m assuming its to do with the format of the inputs.

I have also tried the following image representations and none have worked.

  • Raw RGB data (Binary)
  • Image itself - Object of type JpegImageFile is not JSON serializable
  • base64 encoded string is not a valid path 400 (invalid input parameters)
  • bytes - Object of type bytes is not JSON serializable
  • array of bytes - Object of type bytes is not JSON serializable
  • BytesIO object - Object of type bytes is not JSON serializable

Code provided below:

import torch, gc
import time
from sagemaker.huggingface.model import HuggingFaceModel
from sagemaker.serializers import DataSerializer, JSONSerializer, IdentitySerializer
from sagemaker.deserializers import JSONDeserializer

Hub model configuration https://huggingface.co/models

hub = {
‘HF_MODEL_ID’:‘openai/clip-vit-large-patch14’, # model_id from Models - Hugging Face
‘HF_TASK’:‘zero-shot-image-classification’ # Type of task your model is solving
}

create Hugging Face Model Class

huggingface_model = HuggingFaceModel(
env=hub, # configuration for loading model from Hub
role=role, # IAM role with permissions to create an endpoint
transformers_version=“4.17”, # Transformers version used
pytorch_version=“1.10”, # PyTorch version used
py_version=‘py38’, # Python version used
)

#image_serializer = DataSerializer(content_type=‘image/x-image’) - didn’t work

deploy model to SageMaker Inference

predictor1 = huggingface_model.deploy(
initial_instance_count=1,
instance_type=“ml.g4dn.4xlarge”,
serializer=JSONSerializer(),
deserializer=JSONDeserializer(),
)

image = Image.open(“000000039769.jpg”)
byt = image.tobytes()
mystr = base64.b64encode(byt).decode()
_dict = {“bytes”: byt}
newbytes = base64.b64decode(mystr)

data = {“inputs”: mystr,
“parameters”:{‘candidate_labels’:[“indoor”, “outdoor”]}
}
reponse = predictor1.predict(data)

2

try

{
“input”: “http://someurl.com/someimage.jpg”,
“parameters”: {“candidate_labels”: list_of_zero_shot_labels}
}

3

Hi, I tried supplying image as http url and it works. But if I want to host the model and send image byte string, how do I provide image? thanks