Are there any docs/guides/tutorials to write custom endpoints to be hosted on Huggingface Hub’s inference endpoints? We’re particularly looking at these models which does things a little differently from normal Huggingface AutoModels (that’ll need A100): Unbabel/XCOMET-XXL · Hugging Face google/me…

Guide/Tutorial to write an inference endpoint for custom models

DisgustingOzil June 11, 2024, 3:49am 3

Hi @Saran12 Have you found any solution for writing handler.py for unsloth finetuned mistral 7b? I am facing same issue.

1 Like

Topic		Replies	Views
Help with custom handler.py for model inference endpoint Beginners	1	745	February 24, 2024
Requirements for Hosting LLM via Inference Endpoints Inference Endpoints on the Hub	2	49	June 13, 2025
Model won't load on custom inference endpoint Inference Endpoints on the Hub	2	361	June 13, 2024
Creating inference endpoint with custom handler - is this how it should work? Beginners	5	2322	November 27, 2022
Issue with Deploying LoRA-adapted Model on Hugging Face Endpoint Beginners	10	129	April 26, 2025