Sagemaker Huggingface zero shot classification with candidate_labels in predict method

Amazon SageMaker
1

Hi,
I am using HF for zero shot classification for doc classification. I successfully implemented code with native HF library. After working code I deploy the model using SM, but I am not able to find anywhere how to pass the custom labels to the predict method?

Using HF library:

classifier = pipeline("zero-shot-classification")
a = classifier(
    "Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67  blah blah blah….",
    candidate_labels=["Medical Card", "ID card", "Passport" ,"education", "politics", "business"],
)

I got the prediction score for each candidate label. So all good until here :slight_smile:

Now I deployed the model using our SM:

hub_v1 = {
	'HF_MODEL_ID':'cross-encoder/nli-distilroberta-base',
	'HF_TASK':'text-classification'
}
huggingface_model_v1 = HuggingFaceModel(
	transformers_version='4.17.0',
	pytorch_version='1.10.2',
	py_version='py38',
	env=hub_v1,
	role=role, 
)
serverless_config_v1 = ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=10,
)
predictor_v1 = huggingface_model_v1.deploy(
    serverless_inference_config=serverless_config_v1
)

I called the endpoint like this

predictor_v1.predict({
	'inputs': "Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67  blah blah blah…",
    'candidate_labels' : ['WILL', 'ID', 'P60', 'OTHER']
})

i get output like this:

[{'label': 'ENTAILMENT', 'score': 0.5859580636024475}]

I need the prediction score for the input string per each of the four categories.
Any idea? Am i missing something here? Appreciate your help

2

@dineshmane you deploy your model on sagemaker with the task text-classification and in the pipeline you use zero-shot-classification. If you change your sagemaker task it should work.

Also you payload was a bit off. An example of a zero-shot-classifcation payload can be found in the documentation and I also added a json below. Reference

{
  "inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",
  "parameters": {
    "candidate_labels": ["refund", "legal", "faq"]
  }
}
1 Like
3

You the best @philschmid ! Thank you :slight_smile:
It is working now.
working code base


from sagemaker.huggingface.model import HuggingFaceModel
from sagemaker.serverless import ServerlessInferenceConfig

# Hub Model configuration. <https://huggingface.co/models>
hub_v1 = {
	'HF_MODEL_ID':'cross-encoder/nli-distilroberta-base',
	'HF_TASK':'zero-shot-classification'
}

# create Hugging Face Model Class
huggingface_model_v1 = HuggingFaceModel(
	transformers_version='4.17.0',
	pytorch_version='1.10.2',
	py_version='py38',
	env=hub_v1,
	role=role, 
)

# Specify MemorySizeInMB and MaxConcurrency in the serverless config object
serverless_config_v1 = ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=10,
)

# deploy the endpoint endpoint
predictor_v1 = huggingface_model_v1.deploy(
    serverless_inference_config=serverless_config_v1
)

predictor_v1.predict({
	'inputs': " 
Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67  blah blah ",
    "parameters":{'candidate_labels' : label_list}
})

output was like below:


{'sequence': '   Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67 blah blah  ',
 'labels': ['RENT',
  'Medical_Document',
  'p60 form',
  'uk_driving_licence',
  'WILL',
  'passport'],
 'scores': [0.27988865971565247,
  0.17847613990306854,
  0.16656287014484406,
  0.15448175370693207,
  0.125006765127182,
  0.09558377414941788]}
2 Likes