InternalServer Exception when deploying fine tuned model on Sagemaker

Amazon SageMaker
1

Hello everyone,

I got a fine tuned german classification model which I want to deploy on Sagemaker. To test my model I want to predict a couple of texts in the json format. Here is my code:

import csv
import json
import sagemaker
from sagemaker.s3 import S3Uploader,s3_path_join

# get the s3 bucket
sess = sagemaker.Session()
role = sagemaker.get_execution_role()
sagemaker_session_bucket = sess.default_bucket()


# uploads a given file to S3.
input_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/input")
output_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/output")
#s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)


# datset files
dataset_csv_file="test"
dataset_jsonl_file="test.jsonl"

with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+") as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        # remove @
        json.dump(row, outfile)
        outfile.write('\n')

s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

#package my fine tuned model
!cd model_token && tar zcvf model.tar.gz * 
!mv model_token/model.tar.gz ./model.tar.gz

model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line')

When I execute that batch tranform job I get the following error:

“code”: 400,
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl: “type”: “InternalServerException”,
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl: “message”: “text input must of type str (single example), List[str] (batch or single pretokenized example) or List[List[str]] (batch of pretokenized examples).”

What am I missing?

2

Hey @marlon89,

could you share the cloudwatch logs? They should provide more information to why the batch transform job is failing.
Additionally, can you share an example of your test.jsonl?

3

Sure! I am super new to the whole AWS environment but I try…

Here is a part of my test.json:

{"input": "Machen Sie so weiter...."}
{"input": "."}
{"input": "Bleiben Sie so wie bisher!"}
{"input": "Weiter machen"}
{"input": "Passt alles. "}
{"input": "F\u00fcr alle Vergleiche, die bei mir anstehen, benutze ich Ihre Plattform schon seit Jahren. Ein Beweis f\u00fcr meine Zufriedenheit. "}
{"input": "alles bestens, vielen Dank!"}

When I create a realtime endpoint with the predict function it works:

predictor = huggingface_model.deploy(
   initial_instance_count=1, 
   instance_type="ml.m5.xlarge"
)

data = {
   "inputs": "Weiter machen"
}
predictor.predict(data)
4

input needs to be inputs. Then it should work.

P.S. there might also be an additional issue with encoding, looking at "F\u00fcr alle Vergleiche, i guess your data is not properly encoded/decoded to utf-8

1 Like
5

It works! Thanks for your super fast help!!!

1 Like