Hello I have used the following code to deploy fine tuned llama2 model with a custom inference script. the problem I have is that the custom inference.py script is not being used. I have made changes to the predict_fn and model_fn but it returns the usual response. Here’s my code: config = { 'H…

Deploying custom inference script with llama2 finetuned model

philschmid November 23, 2023, 1:22pm 2

You cannot use the llm container with a custom inference script.

Topic		Replies	Views
Sagemaker Pipelines with fintuned llama2 Amazon SageMaker	0	851	September 12, 2023
Sagemaker deployment fails for local llama2 model Amazon SageMaker	3	2254	August 17, 2023
Error loading finetuned llama2 model while running inference Amazon SageMaker	27	4793	September 20, 2023
Deploying TinyLlama Model via SageMaker Inference Endpoint with Custom Setup Amazon SageMaker	0	444	April 7, 2024
Truncated un-finished response after deploying hugging-face models Amazon SageMaker	0	375	January 19, 2024