Adding Preprocessing to Hosted Inference API

My question answering model requires an extra step of preprocessing the input and the output. How can I add those preprocessing scripts to work with Hosted Inference API (Widget on the website, too)?

hey @yigitbekir, as far as i know the inference api does not support custom pre- / post-processing logic, but you could easily include these steps within a dedicated web / streamlit application :slight_smile:

if you need a custom widget, you can propose one on the huggingface_hub library here: GitHub - huggingface/huggingface_hub: Client library to download and publish models and other files on the huggingface.co hub

1 Like

Hello! There is a section in the Huggingface Hub Docs that describes Generic Inference API.

In my understanding this Generic Inference API allows you to override the default pipeline used in Hosted Inference API.

In order to make this work you must:

  • put your code to pipeline.py
  • specify requirements in requirements.txt
  • specify library_name: generic tag-value in README.md

I tried it myself but I get following errors:

  • “Distant resource does not have an ETag, we won’t be able to reliably ensure reproducibility.”.
  • Or something like “the model took very long to load”. Probably it’s because of long wait for Docker container to install my required dependencies and load the model.

Probably there is a way to make this Generic Inference API work.

Now I’m planning to switch to Huggingface Spaces - probably I can set up my own processing logic for inference there.

Here are some example repositories that make use of Generic Inference API: one and two.

1 Like

upd: I’ve deployed my demo application to Huggingface Spaces and all works fine. You can specify your custom logic of data processing there as well as list of dependencies (both python and debian system packages) that are installed while building Docker container on Space launch.
@yigitbekir

1 Like