Thank tou erikkaum, now I understand.
So this feels like a serious bug to have an inference service ignoring some layers of the inference model. A big warning should show, at least.
I am sorry but to me it is a blocker for adoption of your product. It is a nice idea, but not reliable for production. I will give another try in 6 months. In the mean time I will go terraform and some autoscalable docker container. (No so easy though, I have been working on it for the past couple of day, and autoscaling with caching the model weights and with enough CPU, is not really what it was designed for.
2 Likes