Lm studio and Chat ui doesn't work with module

Beginners
1

Hey there I can’t use Lm STudio and Chat ui I want to use my AI Assistant local but my laptop doesnt support models offline
I want to use it in python but I don’t like terminal shape I want it to work on web page on local host
And I want system prompt and I don’t know where am I going to add it in my code .
I-m programmer but I confused :confused:

1 Like
2

Hmmm?

3 
🥲 Failed to load the model

Error loading model.

(Exit code: -1073740791). Unknown error. Try a different model and/or config.
1 Like
4

try disabling flash attention, i’m not sure why it’s being listed as an unknown architecture

You could ask bartowski. He seems to know a lot about that software, and he’s a friendly guy.
Generally, it’s best to ask in the community for that software or model. I can help with simple errors, but…

5

{
“memory”: {
“ram_capacity”: “3.84 GB”,
“ram_unused”: “449.29 MB”
},
“gpu”: {
“type”: “Nvidia CUDA”,
“vram_recommended_capacity”: “2.00 GB”,
“vram_unused”: “1.65 GB”
},
“os”: {
“platform”: “win32”,
“version”: “10.0.22631”,
“supports_avx2”: true
},
“app”: {
“version”: “0.2.23”,
“downloadsDir”: “C:\Users\fanavaran\.cache\lm-studio\models”
},
“model”: {}
}

1 Like
6

I think that software runs both the LLM server and client locally, but I don’t think it’s possible to run both on that machine…:cold_face:
There are some models with a size of around 1GB, but… not many.

1 Like
7

Is there any way to program it? like js or python?

1 Like
8

There are also APIs for JS, but if you can use Python, it’s the easiest. You can use this HF, or there are several well-known and inexpensive cloud services, such as Groq.
If you use HF, the Inference API is easy to use with famous models for free. There is also a way to operate the Gradio Space in Spaces from a Python client to obtain output. Scraping is also possible.

Also, this is not suitable for incorporating into something, but it is useful.

Nice links from HF Discord

9

So can you give me a source code for run c4ai-command-r-plus-08-2024 module and custom system prompt because I have too many problems with it.

1 Like
10

This model is a gated model, so you will need to obtain permission individually. In addition, since the Inference API is turned off, it is difficult to use it via the cloud. You can use it locally in the same way as the sample, but you will need a GPU with amazing performance. As a trick, it seems that you can use it in the Inference API if you have an OpenAI API key.

11

Yes, but I don’t have a powerful GPU…

Just I make suggestion to huggingface.co to give us in edit assistant to hide system prompt :heart:

1 Like
12

If you use Spaces in private mode, it will be treated as a 404 error for other people, and even if it’s public, the things stored in Secrets won’t be visible to other people.
The HF server administrator will be able to see them, though.