Hi everyone,
I have recently started using batch inference for OpenAI and Google Gemini to reduce token costs (see Batch inference with Gemini | Generative AI on Vertex AI | Google Cloud Documentation ). Is there an equivalent to this on HuggingFace serverless providers via huggingface_hub? Thank you!
Whether batch inference is supported depends on the endpoint. I don’t think many HF endpoints support it…
Short, direct answer first, then details:
There is no direct equivalent of OpenAI’s Batch API or Vertex AI’s Gemini batch prediction (with special discounted pricing and job objects) on Hugging Face serverless Inference Providers via huggingface_hub.
Hugging Face serverless (HF Inference + other Providers) is priced as a normal online API: you pay for compute time per request, not a separate “batch mode” with cheaper tokens.(Hugging Face)
If you want “batch-like” behavior and cost reductions, you have to use:
Below, I’ll spell out:
What OpenAI / Gemini batch actually give you (so we can compare).
How Hugging Face serverless Inference Providers work.
Concrete ways to reduce cost with Hugging Face:
With OpenAI and Gemini:
OpenAI Batch API:
Vertex AI Gemini Batch prediction:
So your mental model now is:
“I have a large offline workload. I can pay less per token by submitting it as a batch job instead of hammering the online API.”
That’s exactly what we need to compare against on Hugging Face.
Hugging Face’s Inference Providers layer is a unified router over many serverless providers (HF Inference, Together, SambaNova, etc.), exposed via the InferenceClient and OpenAI-compatible endpoints.(Hugging Face)
Key points:
HF Inference is the serverless Inference API that used to be called “Inference API (serverless)”. It’s now just one provider (“hf-inference”) behind the router.(Hugging Face)
Pricing model for HF Inference:
Other Inference Providers (Together, SambaNova, etc.) are also billed pay-as-you-go. HF’s own forum answers state that “usage fees for each Inference Provider’s endpoint would apply directly.”(Hugging Face Forums)
Important for your question:
InferenceClient reference provides methods like chat_completion, text_generation, embeddings, etc., but again no create_batch_job / get_batch_result API.(Hugging Face)So:
For serverless Inference Providers, Hugging Face only exposes online (synchronous or streaming) APIs, not an asynchronous batch job API with separate discounted pricing.
There is no 1:1 equivalent of “OpenAI Batch API” or “Vertex Batch prediction” at the Provider-router level.
Even though there is no direct batch-discount API, you can still reduce effective cost per token with a combination of model choice, infrastructure choice, and usage patterns.
Think of three levels:
I’ll walk through each, plus general token-saving techniques.
Within the serverless Providers themselves, you mainly have two levers:
HF Inference pricing is compute-based: hardware-time × price.(Hugging Face)
Implications:
Smaller / more efficient models (e.g. 1–8B LLMs, distilled models, specialized classifiers) will:
You can:
Use the model selector on huggingface.co and check which Providers are available and what their prices are.(Hugging Face)
In the router, you can choose:
:fastest, :cheapest, depending on the provider’s naming scheme."hf-inference") vs a more expensive one.This is analogous to switching from GPT-4 Turbo to GPT-4o-mini: you’re changing the model as your main cost lever.
This does not change the price per token, but it can:
Two main approaches:
Batch multiple inputs into a single call, if the task supports it (e.g. some classification or embedding endpoints accept lists).
Example pattern:
from huggingface_hub import InferenceClient
# See docs: https://huggingface.co/docs/huggingface_hub/en/package_reference/inference_client # noqa: E501
client = InferenceClient(provider="hf-inference") # or another Provider
texts = [
"I love this product!",
"This is terrible...",
"It's ok, not great.",
]
# Example: some text classification models allow list inputs
preds = client.text_classification(
texts,
model="cardiffnlp/twitter-roberta-base-sentiment-latest",
)
for t, p in zip(texts, preds):
print(t, "→", p[0]["label"], p[0]["score"])
Not every method supports list inputs; it depends on the task/model.
Use AsyncInferenceClient to run many requests in parallel with a controlled concurrency limit:
import asyncio
from huggingface_hub import AsyncInferenceClient
# See docs: https://huggingface.co/docs/huggingface_hub/en/package_reference/inference_client # noqa: E501
client = AsyncInferenceClient(provider="hf-inference")
prompts = [
{"role": "user", "content": "Explain batch inference in one sentence."},
{"role": "user", "content": "List three ways to save tokens."},
# ...
]
async def run_one(msg):
resp = await client.chat_completion(
model="meta-llama/Llama-3.2-3B-Instruct:fast", # example cheap-ish model
messages=[msg],
max_tokens=256,
)
return resp.choices[0].message.content
async def main():
sem = asyncio.Semaphore(10) # tune to respect rate limits
async def guarded(m):
async with sem:
return await run_one(m)
results = await asyncio.gather(*(guarded(p) for p in prompts))
for p, r in zip(prompts, results):
print(p["content"], "→", r[:80], "...")
asyncio.run(main())
Again: this improves throughput and utilization, not pricing tiers. But if your load is modest and you just want to process offline datasets faster, this might already be “good enough” without leaving serverless.
If you want something that genuinely looks like “batch jobs”, you should look at Hugging Face Jobs, which is a separate feature from Providers.
From the official Jobs docs and CLI guide:(Hugging Face)
Conceptually, HF Jobs behaves like:
“Docker run on Hugging Face GPUs” with job scheduling and logs.
How this differs from serverless Providers:
a10g-small for N hours).This is much closer to Vertex batch prediction than Providers are.
Very rough workflow:
Put your batch inference script in a repo:
# repo structure
batch_infer/
- run.py
- requirements.txt
In run.py, write your offline inference loop using transformers, vLLM, or TEI, reading from a dataset and writing out predictions.
Launch a job:
hf jobs run \
--name my-batch-infer \
--flavor a10g-small \
python:3.12 \
bash -lc "pip install -r batch_infer/requirements.txt && python batch_infer/run.py"
# docs: https://huggingface.co/docs/huggingface_hub/en/guides/jobs # noqa: E501
Advantages for “token cost”:
By running on your own job, you can:
The HF Jobs docs and related blog posts show examples of this, including large-scale LLM batch inference with vLLM.(Hugging Face)
HF’s general inference guide says you can point InferenceClient to a local or remote server running vLLM, TGI, llama.cpp, etc., as long as it uses an OpenAI-compatible API.(Hugging Face)
Example pattern from the docs:
from huggingface_hub import InferenceClient
# Example server: vLLM / TGI / litellm proxy exposing OpenAI-style /v1/chat/completions
# See: https://huggingface.co/docs/huggingface_hub/en/guides/inference # noqa: E501
client = InferenceClient(model="http://localhost:8080")
resp = client.chat_completion(
model="meta-llama/Llama-3.2-3B-Instruct",
messages=[{"role": "user", "content": "Short answer: why batch inference?"}],
max_tokens=128,
)
print(resp.choices[0].message.content)
With something like vLLM:
LLM object, load an HF model, and pass lists of prompts, with strong throughput as batch size increases.(docs.vllm.ai)You can deploy this:
Cost-wise:
This is essentially “build your own Vertex Batch but using open models from Hugging Face and a general-purpose job system.”
These are independent of whether you use serverless Providers, HF Jobs, or your own infra. They directly reduce tokens → reduce cost.
Shorter prompts and contexts:
Control max_new_tokens / max_tokens:
Specialized small models instead of big general LLMs:
Embeddings:
Caching and deduplication:
Even if HF does not give you a “batch token discount,” these simple steps can easily cut your total token usage by 2–5×, which usually dominates any 50% discount you get from Batch in other ecosystems.
Given how you’re already using OpenAI and Gemini batch, here’s a practical mapping to Hugging Face:
If you want a pure serverless, “no infra” approach:
AsyncInferenceClient to run high-throughput offline jobs (e.g., reading prompts from a file and writing out results), but treat this as “my own batch script using a real-time API”, not a special Batch API.If you want a true batch-optimized workflow (closest to Vertex Batch):
Move the offline workloads to HF Jobs:
Pay for GPU time instead of Provider tokens, and tune batch size and scheduling for maximum throughput.
If you already have cloud infra (GCP / AWS / etc.):
You can emulate Vertex Batch by:
Then call these endpoints from your existing pipelines in almost exactly the same way as you do today with Vertex batch jobs.
In other words:
No direct Batch API / discount on HF serverless Providers.
Yes, you can absolutely get “batch-style” cost savings by:
Thanks for pointing me in the right direction @John6666 ! It seems that at least NOVITA does support batch API according to this recent blogpost: Batch API: Reduce Bandwidth Waste and Improve API Efficiency
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.