PEFT on HuggingChat in Spaces

bsolie · July 14, 2024, 11:03am

I am trying to start HuggingChat in a space with a PEFT fine-tuned model. I am getting the “cannot find config.json” error. This seems related to https://discuss.huggingface.co/t/how-do-i-deploy-my-trained-lora-adapter-to-a-chatui-in-huggingface/75544

I do see that TGI now supports LORA here:

github.com/huggingface/text-generation-inference

Enable multiple LoRa adapters

huggingface:main ← huggingface:lora-internal

opened 08:14PM - 04 Jun 24 UTC

drbh

+2785 -76

This PR is a work in progress to add support for mutliple loras to be loaded at …startup and then use 0 or 1 adapters in a request by specifying the adapter id. ## Example usage download adapter without auto merging ```bash text-generation-server download-weights predibase/dbpedia text-generation-server download-weights predibase/customer_support ``` start server with multiple LoRa adapters ```bash LORA_ADAPTERS=predibase/customer_support,predibase/dbpedia \ text-generation-launcher --model-id mistralai/Mistral-7B-v0.1 ``` sending request without adapter id ```bash curl 127.0.0.1:3000/generate \ -X POST \ -H 'Content-Type: application/json' \ -d '{ "inputs": "What are 3 unique words that describe you?", "parameters": { "max_new_tokens": 40 } }' ``` ```json { "generated_text": "\n\nI’m a very passionate person. I’m very driven. I’m very determined.\n\nWhat is your favorite thing about being a teacher?\n\nI love the fact" } ``` with first LoRa adapter specified ```bash curl 127.0.0.1:3000/generate \ -X POST \ -H 'Content-Type: application/json' \ -d '{ "inputs": "What are 3 unique words that describe you?", "parameters": { "max_new_tokens": 40, "adapter_id": "predibase/customer_support" } }' ``` ```json { "generated_text": "\n\nI’m not sure if I can come up with 3 unique words that describe me, but I’ll try.\n\n1. Creative\n2. Funny\n3." } ``` with second LoRa adapter specified ```bash curl 127.0.0.1:3000/generate \ -X POST \ -H 'Content-Type: application/json' \ -d '{ "inputs": "You are given the title and the body of an article below. Please determine the type of the article.### Title: Great White Whale\n\n### Body: Great White Whale is the debut album by the Canadian rock band Secret and Whisper. The album was in the works for about a year and was released on February 12 2008.", "parameters": { "max_new_tokens": 40, "adapter_id": "predibase/dbpedia" } }' ``` ```json { "generated_text": "8" } ```

Is there something I am missing? Do there need to be changes to HuggingChat? Or should TGI automatically pick up the LORA status of my model?

Brand new to this, and excited to be making progress on my project, but stuck here for now. Any pointers appreciated.

Topic		Replies	Views
Can't find 'adapter_config.json' Spaces	1	566	March 30, 2024
Can not find adapter_config.json using PeftConfig.from_pretrained Beginners	8	12597	July 11, 2024
Cluster tried to load adapter_config.json even though model is not PEFT-based model Inference Endpoints on the Hub	1	487	April 5, 2024
Issue with Deploying LoRA-adapted Model on Hugging Face Endpoint Beginners	9	68	April 10, 2025
HuggingChat Repo: Error: Not found: /chat Spaces	0	379	June 15, 2023

PEFT on HuggingChat in Spaces

Related topics