Hey, Had some quick questions regarding the Inference Toolkit. Is there a way to add an init function in the custom inference.py script. I was thinking I could just add what I needed in the model_fn function but when I tried running just the basics, I got an error attached below. This leads into th…

You can add a requirements.txt into the code/ and the archive and upload it to s3 and provide it as model_data. This should work. You can use my example to test it. Additionally, does the FrameworkModel class have the attribute dependencies, but it looks way more complex to add your dependencies. …

Inference Toolkit - Init and default template for custom inference

Amazon SageMaker

ujjirox October 4, 2021, 7:21am 8

Gotchu. I figured that’s what you meant a couple minutes later.

Deploying TinyLlama Model via SageMaker Inference Endpoint with Custom Setup

Topic		Replies	Views
Help for inference.py code Amazon SageMaker	10	3491	March 8, 2022
Inference Toolkit - custom inference with multiple models Amazon SageMaker	1	359	April 4, 2024
Deploying custom inference script with llama2 finetuned model Amazon SageMaker	6	875	January 4, 2024
Infer on sagemaker with custom pipeline Amazon SageMaker	2	425	September 14, 2023
Loading inference.py separately from model.tar.gz Amazon SageMaker	4	1357	June 5, 2023

Inference Toolkit - Init and default template for custom inference

Related Topics