How to deploy a T5 model to AWS SageMaker for fast inference?

[philschmid]

@pierreguillou

Last point : in AWS Sagemaker Inference, I did not use the ONNX model but the Pytorch T5 model. As I saw a different output, it means that the DLC for inference does some compression on T5 (ONNX or similar?) that could explain the different output?

No SageMaker is not doing any compression on some sort of today.

Are you installing transformers==4.15 through a requirement.txt within SageMaker?