Issue - ValueError: Unsupported model type mixtral

Amazon SageMaker
1

Trying to deploy the PreTrained version of the below model to SageMaker Endpoint.

Model Name: mistralai/Mixtral-8x7B-Instruct-v0.1

Using the g5.48xlarge instance for the same.

But getting the below error:

File “/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py”, line 124, in serve_inner

model = get_model(model_id, revision, sharded, quantize, trust_remote_code)

File “/opt/conda/lib/python3.9/site-packages/text_generation_server/models/init.py”, line 314, in get_model
raise ValueError(f"Unsupported model type {model_type}")

ValueError: Unsupported model type mixtral.

Please find the below deployed commands for the same:

from sagemaker.huggingface import get_huggingface_llm_image_uri

retrieve the llm image uri

llm_image = get_huggingface_llm_image_uri(
“huggingface”,
version=“1.1.0”
)

print(f"llm image uri: {llm_image}")

import json

from sagemaker.huggingface import HuggingFaceModel

sagemaker config

instance_type = “ml.g5.48xlarge”
number_of_gpu = 1
health_check_timeout = 1000

Define Model and Endpoint configuration parameter

config = {
‘HF_MODEL_ID’: “mistralai/Mixtral-8x7B-v0.1”, # model_id from Models - Hugging Face
‘SM_NUM_GPUS’: json.dumps(number_of_gpu), # Number of GPU used per replica
‘MAX_INPUT_LENGTH’: json.dumps(1024), # Max length of input text
‘MAX_TOTAL_TOKENS’: json.dumps(2048), # Max length of the generation (including input text)
}

create HuggingFaceModel with the image uri

llm_model = HuggingFaceModel(
role=role,
env=config,
transformers_version = ‘4.28’, # the transformers version used in the training job
pytorch_version = ‘2.0’, # the pytorch_version version used in the training job
py_version = ‘py310’,
)

llm = llm_model.deploy(
initial_instance_count=1,
instance_type=instance_type,
container_startup_health_check_timeout=health_check_timeout,
endpoint_name = ‘mixtral-inference-testing1’
)

Please let me know if I am using the right versions or if I am doing anything wrong.
Let me know if any other information is required.

Thanks.

1 Like
2

On troubleshooting, found that the initial or above mentioned error was due to the transformer version. Earlier was using 4.34. Now changed it to 4.36.2

After the change, getting the below error while the logic is trying to download the Model.

**AlgorithmError: ExecuteUserScriptError: ExitCode 1 ErrorMessage "ImportError: /opt/conda/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c106SymIntltEl **

training_function model = AutoModelForCausalLM.from_pretrained(script_args.model_id, File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 565, in from_pretrained model_class = _get_model_class(config, cls._model_mapping) File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py”, line 387, in _get_model_class supported_models = model_mapping[type(config)] File "/opt/conda/lib/python3.10/site-packag

But the same logic is working (the Model is getting downloaded and also I am able to do some training in the same) when I am running it outside the AWS Sagemaker.

1 Like