Inference Endpoints - No working code examples

neopolita · January 29, 2025, 10:52am

You might want to use the /completion endpoint instead:

examples/server/README.md

master

# LLaMA.cpp HTTP Server

Fast, lightweight, pure C/C++ HTTP server based on [httplib](https://github.com/yhirose/cpp-httplib), [nlohmann::json](https://github.com/nlohmann/json) and **llama.cpp**.

Set of LLM REST APIs and a simple web front end to interact with llama.cpp.

**Features:**
 * LLM inference of F16 and quantized models on GPU and CPU
 * [OpenAI API](https://github.com/openai/openai-openapi) compatible chat completions and embeddings routes
 * Reranking endoint (WIP: https://github.com/ggerganov/llama.cpp/pull/9510)
 * Parallel decoding with multi-user support
 * Continuous batching
 * Multimodal (wip)
 * Monitoring endpoints
 * Schema-constrained JSON response format

The project is under active development, and we are [looking for feedback and contributors](https://github.com/ggerganov/llama.cpp/issues/4216).

## Usage

This file has been truncated. show original

Topic		Replies	Views
HF Inference Endpoints don't finish Initializing Inference Endpoints on the Hub	0	240	March 28, 2024
Unable to get inference results after deploying model to Inferende Endpoints Inference Endpoints on the Hub	0	14	May 8, 2025
Dumb Question: Seeing that my inference API links not working Beginners	1	72	July 10, 2025
Inference Endpoints / Model choices / Help Inference Endpoints on the Hub	1	30	July 10, 2025
Dedicated inference endpoint failing to initialize Inference Endpoints on the Hub	0	42	November 21, 2024

Inference Endpoints - No working code examples

Related topics