Hey, Had some quick questions regarding the Inference Toolkit. Is there a way to add an init function in the custom inference.py script. I was thinking I could just add what I needed in the model_fn function but when I tried running just the basics, I got an error attached below. This leads into th…

You can add a requirements.txt into the code/ and the archive and upload it to s3 and provide it as model_data. This should work. You can use my example to test it. Additionally, does the FrameworkModel class have the attribute dependencies, but it looks way more complex to add your dependencies. …

Inference Toolkit - Init and default template for custom inference

Amazon SageMaker

ujjirox October 4, 2021, 7:21am 8

Gotchu. I figured that’s what you meant a couple minutes later.

Deploying TinyLlama Model via SageMaker Inference Endpoint with Custom Setup

Topic		Replies	Views
Help for inference.py code Amazon SageMaker	10	4006	March 8, 2022
Loading inference.py separately from model.tar.gz Amazon SageMaker	4	1879	June 5, 2023
Install custom python libraries in HuggingFace DLC Amazon SageMaker	9	2008	August 8, 2022
Facebook/bart-large-mnli inference when deployed on SageMaker Amazon SageMaker	1	1088	April 29, 2022
Inference Hyperparameters Amazon SageMaker	29	4852	October 8, 2021

Inference Toolkit - Init and default template for custom inference

Related topics