How do I deploy my trained LORA adapter to a ChatUI in huggingface?

Here’s my trained LORA adapter on Mistral8x7b: dominic5/Project5_V3_Mistral8x7b_V2.2.5 · Hugging Face. How do I deploy it to a huggingchatUI interface such that it looks like this? Zephyr Gemma Chat - a Hugging Face Space by HuggingFaceH4.

I’m asking since the tutorials and guides I’ve seen so far are only for models that aren’t adapters.

Thank you.


ChatUI works with a TGI backend, hence you can follow this thread curious about the plans for supporting PEFT and LoRa. · Issue #482 · huggingface/text-generation-inference · GitHub.

See also GitHub - huggingface/chat-ui: Open source codebase powering the HuggingChat app

Update, vLLM now supports LoRa serving: Using LoRA adapters — vLLM. vLLM is a framework similar to TGI.

Update, TGI now supports multi-LoRa inference: Enable multiple LoRa adapters by drbh · Pull Request #2010 · huggingface/text-generation-inference · GitHub

1 Like