How to make an inference for HuggingFaceModel of type 'image-to-text'

Amazon SageMaker
1

Deploy code:

from sagemaker.huggingface import HuggingFaceModel

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'naver-clova-ix/donut-base-finetuned-cord-v2',
    'HF_TASK':'image-to-text'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.26.0',
    pytorch_version='1.13.1',
    py_version='py39',
    env=hub,
    role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)

Inference try#1:

# Provide image data as input

# example request, you always need to define "inputs"
data = {
    "inputs": 'https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/raw/main/ADE_val_00000001.jpg'
}

# request
predictor.predict(data)

fails with

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "\u0027str\u0027 object has no attribute \u0027decode\u0027"
}

Inference try#2

!wget https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/raw/main/ADE_val_00000001.jpg

# Provide image data as input
image_path = 'ADE_val_00000001.jpg'

# example request, you always need to define "inputs"
data = {
    "inputs": image_path,
}

# request
predictor.predict(data)

fails with

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "Incorrect path or url, URLs must start with `http://` or `https://`, and ADE_val_00000001.jpg is not a valid path"
}

Inference try #3

from PIL import Image
import requests
from io import BytesIO

def url_to_image(url):
    try:
        response = requests.get(url)
        response.raise_for_status()  # Check if the request was successful

        # Open the image using PIL
        image = Image.open(BytesIO(response.content))
        return image
    except Exception as e:
        print(f"Error: {e}")
        return None

# Example usage:
url = 'https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/raw/main/ADE_val_00000001.jpg'
image = url_to_image(url)

Fails with

TypeError: Object of type JpegImageFile is not JSON serializable