Cannot import "Conversation" from transformers_utils.py

Amazon SageMaker
1

I am deploying a Huggingface model with Sagemaker. This is the code that I use to deploy:

import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel

sess = sagemaker.Session()
role = sagemaker.get_execution_role()
client = boto3.client('sagemaker')

def new_endpoint(name, model_path, instance):
  # create HuggingFaceModel
  model = HuggingFaceModel(
    model_data=model_path+"output/model.tar.gz",
    role=role,
    transformers_version='4.28',
    pytorch_version='2.0',
    py_version='py310',
  )
  
  # Deploy model to an endpoint
  deployed = model.deploy(
    initial_instance_count=1,
    instance_type=instance,
    endpoint_name=name,
  )

This has previously worked without issue, but in the last few days, the model no longer successfully deploys, even though I have not modified the code. Looking in the CloudWatch logs, I see that this specific line causes the error:

2024-06-11T13:30:54,406 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     from sagemaker_huggingface_inference_toolkit.transformers_utils import (

…

2024-06-11T13:30:54,406 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     from transformers.pipelines import Conversation, Pipeline

I have looked at the source code of the transformer_utils.py file, and the line that causes this error can be seen:

from huggingface_hub import HfApi, login, snapshot_download
from transformers import AutoTokenizer, pipeline
from transformers.file_utils import is_tf_available, is_torch_available
from transformers.pipelines import Conversation, Pipeline

I am unable to resolve this issue. Testing it locally, this line does not work for me either. What surprises me the most is that this error has occurred spontaneously, without the sagemaker-huggingface-inference-toolkit or transformers libraries having been updated.
Any help would be appreciated, thanks.

2

I should add that while experimenting, I have found this line to work:

from transformers.pipelines.conversational import Conversation

However, the line that is in the source code does not work:

from transformers.pipelines import Conversation, Pipeline
3

It seems a change in the packages in the transformers project was introduced some weeks ago Remove ConversationalPipeline and Conversation object by Rocketknight1 · Pull Request #31165 · huggingface/transformers · GitHub

We observed the same issue in SageMaker with our hosted model. The issue should be solved by pinning the transformers dependency to version 4.41.2 which still has the object in the expected package.

I already reported the issue, so the team can look at it: SageMaker fails because Conversation object is not found · Issue #129 · aws/sagemaker-huggingface-inference-toolkit · GitHub

3 Likes
4

Thank you!

5

Hi @wizardh , may I ask how did you pin the version.
Requirements.txt inside the model does not seem to work since hf version is depending on the container deep-learning-containers/available_images.md at master · aws/deep-learning-containers · GitHub
None of the older versions is working for me . I event tried 4.26

        transformers_version="4.26",  # transformers version used
        pytorch_version="1.13",  # pytorch version used
        py_version='py39',  # python version used

Help would be really appreciated

6

I was able to solve by having a requirements.txt inside the model folder beside inference.py . My assumption that the transformers lib will be the version from container is not right

7

@philhd Can you mention what is present in your requirements.txt?