Hello I have used the following code to deploy fine tuned llama2 model with a custom inference script. the problem I have is that the custom inference.py script is not being used. I have made changes to the predict_fn and model_fn but it returns the usual response. Here’s my code: config = { 'H…

Deploying custom inference script with llama2 finetuned model

philschmid November 23, 2023, 1:22pm 2

You cannot use the llm container with a custom inference script.

Topic		Replies	Views
Streaming output text When deploying a finetuned (SFT, DPO) model with custom inference script Amazon SageMaker	1	32	November 8, 2024
Error loading finetuned llama2 model while running inference Amazon SageMaker	27	4811	September 20, 2023
Inference Toolkit - Init and default template for custom inference Amazon SageMaker	12	2144	October 4, 2021
Loading inference.py separately from model.tar.gz Amazon SageMaker	4	1867	June 5, 2023
Sagemaker deployment fails for local llama2 model Amazon SageMaker	3	2287	August 17, 2023