Inference Hyperparameters

[philschmid]

For me it is working I deployed a

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'textattack/albert-base-v2-imdb',
	'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

here is how invoked it using boto3

import boto3
import json
client = boto3.client('sagemaker-runtime')
request = {
	'inputs': super_long_intput,
    "parameters": {'truncation': True}
}

response = client.invoke_endpoint(
EndpointName="huggingface-pytorch-inference-2021-09-24-14-26-23-337",
ContentType="application/json",
Accept="application/json",
Body=json.dumps(request),
)
response['Body'].read().decode()

and a screenshot of that without {'truncation': True} i get and error