Slow inference using most recent docker image

[philschmid]

Hello @ojturner,

From your code

@ojturner

 huggingface_model = HuggingFaceModel(
    role="-",
    model_data="-",
    entry_point="-",
    source_dir="-",
    code_location="-",
    transformers_version='4.12.3',
    pytorch_version='1.9.1',
    py_version='py38',
)

it looks like you are using a custom inference.py script is that correct? could you provide? Have you tested the latency and overhead using the “zero-code” deployment, without providing inference.py.

Could you also share more information about which model/model-architecture/task you are using?