Creating inference endpoint with custom handler - is this how it should work?

Beginners
1

I don’t seem to quite understand how to deploy a model with a custom handler as an inference endpoint. Specifically, I’m trying to use this model (which alread has a custom handler): philschmid/stable-diffusion-v1-4-endpoints · Hugging Face

I went through these steps trying to follow the tutorial:

  1. Initialize Git LFS: git lfs install
  2. Clone the desired repo: git clone https://huggingface.co/philschmid/stable-diffusion-v1-4-endpoints
  3. Install huggingface-hub python package (in a venv): python -m venv venv + source venv/bin/activate + pip install huggingface-hub
  4. Login to Huggingface: huggingface-cli login → Choose Yes when asked to add token as git credential
  5. Run this command (don’t really understand what it does): git config --global credential.helper store

As far as I understand, I don’t need to change anything in the code, since it already has a custom handler. I now need to push the whole thing to a repo that I created before. It’s currently empty: shttps://huggingface.co/eykarim/stable-diffusion-v1

  1. I move up one folder into /stable-diffusion-v1-4-endpoints
  2. I delete the existing .git so I can push the whole code to my own repo: sudo rm -r .git
  3. I initialize a new git: git init + git remote add origin master + git remote set-url origin https://huggingface.co/eykarim/stable-diffusion-v1
  4. Add all files to commit to my new remote repo: git add * + git commit -m "my first commit" + git push origin master

At this point I’m getting a git error error failed to push some refs to https://huggingface.co/eykarim/stable-diffusion-v1

As far as I understand this should be the right way. Once I have code with a handler.py in my repo, I should be able to find it in the inference endpoints search mask, right? (see screenshot)

Bildschirmfoto_2022-11-21_10-42-47
Bildschirmfoto_2022-11-21_10-42-47981×620 54.2 KB

Is this the right way to do it? And if so: Maybe someone could help me fix the Git error? That would be great!

2

Anybody? If someone has a link/resource that could help me solve this by myself, I’d be more than happy! :slight_smile:

3

cc @philschmid

btw, might be better to post this in the “inference endpoints” section, rather than “beginners”

1 Like
4

Hello @eykarim,

You can check out this tutorial: Custom Inference with Hugging Face Inference Endpoints
or the documentation on how to create custom handlers: Create custom Inference Handler

Those include e2e example with uploading the model.

FYI we also have a dedicated inference endpoints section in the forum.

1 Like
5

Hi Phil! Thanks for the tutorial. I worked through it a second time yesterday and now have a model deployed: eykarim/stable-diffusion-v1 · Hugging Face

As I wrote in my original post, I was assuming that once I have a model deployed with a custom handler (marked as “endpoints-template”) I should be able to deploy it with an Inference Endpoint. However, the model does not show up in the search bar, nor do many of the other models. I don’t understand what requirements a model needs to fullfil in order to be deployed as an Inference Endpoint. Why do some models with an endpoint handler show up in the search bar and others don’t?

Bildschirmfoto_2022-11-26_20-17-13
Bildschirmfoto_2022-11-26_20-17-13914×422 36.5 KB

6

I responded in your other thread: Why is my model not showing up in the search bar for Inference Endpoints? - #2 by philschmid

1 Like