Hey, Had some quick questions regarding the Inference Toolkit. Is there a way to add an init function in the custom inference.py script. I was thinking I could just add what I needed in the model_fn function but when I tried running just the basics, I got an error attached below. This leads into th…

You can add a requirements.txt into the code/ and the archive and upload it to s3 and provide it as model_data. This should work. You can use my example to test it. Additionally, does the FrameworkModel class have the attribute dependencies, but it looks way more complex to add your dependencies. …

Inference Toolkit - Init and default template for custom inference

Amazon SageMaker

ujjirox October 4, 2021, 7:21am 8

Gotchu. I figured that’s what you meant a couple minutes later.

Deploying TinyLlama Model via SageMaker Inference Endpoint with Custom Setup

Topic		Replies	Views
Loading inference.py separately from model.tar.gz Amazon SageMaker	4	1861	June 5, 2023
Help for inference.py code Amazon SageMaker	10	4001	March 8, 2022
HuggingFaceModel ignores code directory Amazon SageMaker	2	15	June 17, 2025
Install custom python libraries in HuggingFace DLC Amazon SageMaker	9	2003	August 8, 2022
Inference Toolkit - custom inference with multiple models Amazon SageMaker	1	635	April 4, 2024

Inference Toolkit - Init and default template for custom inference

Related topics