Sentence similarity models on Sagemaker

Amazon SageMaker
1

Hello,

I am wondering if the “sentence-similarity” pipelines can be used on Sagemaker from Hub with the same ease as popular piplines like “question-answering”?
I was trying to start an endpoint with sentence-similarity, but it gave me this error:

2021-09-16 13:13:49,252 [INFO ] W-sentence-transformers__pa-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: "Unknown task sentence-similarity, available tasks are ['feature-extraction', 'text-classification', 'token-classification', 'question-answering', 'table-question-answering', 'fill-mask', 'summarization', 'translation', 'text2text-generation', 'text-generation', 'zero-shot-classification', 'conversational', 'image-classification', 'translation_XX_to_YY']" : 400

This is how it was deployed inside Sagemaker Studio:


from sagemaker.huggingface import HuggingFaceModel
import sagemaker 

role = sagemaker.get_execution_role()

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'sentence-transformers/paraphrase-xlm-r-multilingual-v1',
    'HF_TASK':'sentence-similarity' # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)


# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.t2.medium"
)
2

Hello @pavel-nesterov,

Currently, the SageMaker Inference Toolkit only supports the NLP-based pipelines from the transformers library for zero-code deployments.
But you can easily create your own inference.py which contains the code you would need for that.
Here is Documentation on that: Deploy models to Amazon SageMaker
Here the code for the sentence-simliarity pipeline: huggingface_hub/sentence_similarity.py at main · huggingface/huggingface_hub · GitHub

Thank you for the request I will add it as potential feature to the roadmap.

1 Like
3

Thank you, Philipp.

I will try it as “next easiest step” in the search of the solution.
The approach I am trying to explore now is “Are there any of the 186 Sentence Similarity models can be deployed on sagemaker with no-code approach”. I checked all of them and I found some! (scerenshot is below)
The only thing is that it doesn’t have “sentence similarity” pipeline, but I think I would try “token-classification” one. I am writing it because someone later may come to the same idea and I may save some time for that explorer.

image
image1190×623 110 KB

1 Like
4

My 1st iteration failed, but I hope it will help to improve the Model Hub. Therefore I am getting deeper into your suggestion, Philipp @philschmid
I will post here how it goes (it may help someone like me, who tries to find answers in this forum).

1 Like
5

Hi Philipp! @philschmid

I read the documentation twice and concluded, that I need to get deeper understanding of how the transformers work. The best way I found (as of know) - is to go through the HF Course .

Even though I did many training jobs on Sagemaker with vanilla Pytorch, I feel like I need to

  1. get a deeper understanding of “pipelines”, “tokenisers” and “post-processing”.
  2. In my project I will also need to fine-tune a pre-trained model (because I am builing a product for not a widely-spoken language - slovenian).
  3. The link you shared leads to “Advanced features” of the HF-Sagemeker" documentation (read twice).

Considering the three points above, I am timeboxing a week to go through the HF course to see if it will me to approach overriding default methods of HF inference toolkit.

Not sure if there is a better way to approach the problem of making sentence-similarity endpoint on sagemaker. My candidate was to train bag-of-words + cosine similarity from scratch in Pytorch (and then deploy the endpoint), but I feel I will be able to benefit from learning the transformers’ basics.

Thank you for helping me to find the direction for the solution!

1 Like
6

Sounds good!
I found two solvenian model on the hub, which you can give a try to fine-tune and compare with your bag-of-words method:

1 Like
7

@philschmid I tried the custom method using inference.py, but I am faing the following error

"You need to define one of the following ['audio-classification', 'automatic-speech-recognition', 'feature-extraction', 'text-classification', 'token-classification', 'question-answering', 'table-question-answering', 'fill-mask', 'summarization', 'translation', 'text2text-generation', 'text-generation', 'zero-shot-classification', 'conversational', 'image-classification', 'image-segmentation', 'object-detection'] as env 'HF_TASK’

Will setting an environment variable solve this error?