How Do You Design a Voice AI Architecture That Achieves ~500ms First-Token Latency While Remaining Cost-Effective?

Research
1

Hi everyone,

I’m an AI developer working on a real-time voice AI system and would appreciate advice from people who have experience building low-latency conversational agents.

My goal is to achieve:

  • First response token in around 500 ms or less

  • Good speech quality and natural conversations

  • Cost-effective architecture that can scale

  • Support for real-time streaming audio

I’m trying to understand the best architecture and component choices across the entire pipeline:

  • Audio transport (WebRTC, WebSocket, etc.)

  • Voice Activity Detection / End-of-Utterance detection

  • Speech-to-Text (Deepgram, Gladia, AssemblyAI, etc.)

  • LLMs (Gemini Live, OpenAI Realtime, Qwen Omni, custom pipelines, etc.)

  • Text-to-Speech (ElevenLabs, Cartesia, Telnyx, OpenAI, etc.)

  • Orchestration frameworks (LiveKit, Pipecat, custom architecture)

For those who have built production-grade voice agents:

  1. What architecture are you using to achieve the lowest possible latency?

  2. Which components contribute the most to latency?

  3. Is a speech-to-speech model better than a traditional STT → LLM → TTS pipeline?

  4. What first-token latency are you seeing in production?

  5. Which providers offer the best balance of latency, quality, and cost?

  6. Are there any architectural mistakes that commonly increase latency?

I’d love to hear real-world numbers, benchmarks, and lessons learned from production deployments.

Thanks!

2

~500ms first-token is doable with a cascaded pipeline but it’s tight, most production STT → LLM → TTS stacks land 600ms-1.2s, and hitting 500ms means every stage has to be near best-case (fast VAD/endpointing ~50-80ms, streaming STT 150-250ms like Deepgram or AssemblyAI’s streaming models, a genuinely fast-TTFT LLM at 200-400ms, streaming TTS with sub-100ms first byte like Cartesia or ElevenLabs Flash). LiveKit’s Agents framework or Pipecat are the two go-to orchestrators for exactly this, both handle the streaming/barge-in plumbing so you’re not building that yourself.

On speech-to-speech vs pipeline: speech-to-speech (OpenAI Realtime, Gemini Live) trims model latency to ~200-300ms and gets you free interruption handling and prosody awareness, but you lose per-stage observability/debuggability and you’re locked into one vendor’s voice and behavior. Cascaded pipelines are still the default for production/enterprise because you can swap components, get full transcripts for auditing, and tune each stage independently, at the cost of a bit more latency and more integration work. Biggest latency killer people report is the LLM’s TTFT when using a large/general model instead of a small fast one tuned for short conversational turns, that’s usually where most of your 500ms budget disappears if you’re not careful. Hugging Face’s own speech-to-speech repo is worth a look too if you want a fully open, self-hostable reference pipeline (VAD → STT → LLM → TTS, swappable components, runs their Reachy Mini robots in production).

3

I’ve built multiple Voice AI systems, and the answer to your question really depends on what technical constraints you have, though I can try to point you to some common techniques

Latency

Latency in Voice AI systems is less about the speed of the underlying agentic process than about how quickly the person you’re speaking with receives an acknowledgment. If you’re building a one-shot agent, you can stream tokens to your preferred TTS and send partial audio chunks. Now, of course, your agent is unlikely to be one-shot; fortunately, other techniques can help here too.

If you have to wait for an underlying agentic process to complete, you can use “filler phrases” to significantly reduce the perceived latency of your Voice AI. While it sounds very simplistic on the surface, it genuinely has a massive impact on the user experience, and you can take advantage of many different approaches to make those filler phrases relevant.

For example: transcript → classifier → filler phrase selection

Suppose you’re building a drive-thru, and the customer asks whether an item is in stock. Clearly, the voice agent doesn’t know the answer itself, so it has to use a tool call to check the current menu state. You would have a menu_question classification that can reply with multiple different phrases, such as:

Let me check that for you
I'll check, one moment

That will buy you some time while the rest of the agent process is happening, though if that process is long-lived or responds with a structured response that you need to parse, there are now tools for reading partial JSON/YAML/etc so once the relevant field you need to read is present, you can start reading that response rather than having to wait for the entire structured response. PyndaticAI for example has some functions for this.

Speech Quality

This is highly subjective and depends on the target audience, not just what sounds “good” to your or my ears, as well as the speech requirements. For example, some people test what they’re speaking to by having the agent repeat what it just said and listening to see whether it sounds identical. If the response sounds identical, they’ll know they’re speaking to a bot and either leave the interaction or request a human.

Cost-Effective Scaling

This is again a difficult question to answer, as multiple factors drive the cost of your voice agent.

  1. LLM Token costs
  2. ASR audio time cost
  3. TTS chunk time cost

You can reduce this with clever caching, so you don’t always have to recompute responses in real time, though that is not always possible. So, reducing your cost will require multiple steps, depending on where you are in your application’s life cycle.

  1. Fine-tuning smaller LLMs
  2. Prompt engineering to optimize token usage
  3. Branching agentic logic
    1. Using the drive-thru example, an intent classifier to send requests to specialist agents

In my experience, one major component people often overlook in this process is: “Who is this voice agent interacting with that drives the most value?” Starbucks is a great example; their agent doesn’t interact directly with the customer, but with the barista. It helps them remember recipes, customer preferences, and inputs the order for the barista into the point-of-sale. This cycle creates a low-cost system that still meets key KPIs by speeding up order taking and improving order accuracy.

Orchestration Frameworks

The majority of tools like PipeCat provide relatively robust orchestration, though you’ll be responsible for exposing that orchestration to whoever needs to have control over it. These platforms do come with the cost of living in their ecosystem and paradigms. They also expose your Voice AI to the risk that these platforms fail and become unsupported, requiring a ground-up rebuild.

I make my choice based on the engineering, capital, and time resources available to me. If it’s a personal project, I will tend to lean on PipeCat. If it’s for an enterprise-scale system, I almost always go with a custom-built platform, because I want to own the control plane and not rely on a third party to make design decisions for my team or me.

Websocket versus WebRTC

Does every chunk of audio matter? If so, then websockets are likely your best option since they enforce ordered responses from the TTS. If losing audio is acceptable, then WebRTC is a great choice, as it ensures that the Voice AI’s response state is not significantly behind the current conversational state.

The Challenging Parts

Honestly, all of the above breaks down to design decisions; you will regret some, and you won’t regret others. Fortunately, that’s the beauty of code; we can always rescaffold something if it was a mistake.

The real challenge is understanding how your Voice AI impacts your KPIs. This requires a comprehensive understanding of the domain you’re working in, as well as realizing that high-level KPI’s are no longer as powerful as they used to be. Again, using the drive-thru example, the accuracy KPI isn’t really a single metric, as it is made up of multiple smaller ones. Specifically, the cases of modification, removal, and addition accuracy.

I hope this was helpful.

4

I would avoid optimizing only median first-token latency. In voice, p95 end-of-turn detection, interruption recovery, and time-to-correct-a-false-start often determine whether the interaction feels natural. A system that answers in 450 ms but speaks over the user every tenth turn will feel slower than a consistent 700 ms pipeline.

Measure the stages with one trace ID: audio arrival, VAD decision, stable transcript, model first token, TTS first byte, and audible playback. Then test under packet loss and barge-in, not only clean lab audio. That usually reveals whether the budget belongs in the model or in endpointing/transport.

5

Latency is well covered above, so here’s the “cost-effective” half. TTS is where the per-minute bill swings the most, so it’s worth choosing per use case rather than picking one voice for everything.

List prices per 1M characters (~1,000 minutes of speech at ~1k chars/min):

- Google Standard / WaveNet: $4

- OpenAI tts-1, Deepgram Aura-1: $15

- Deepgram Aura-2, OpenAI tts-1-hd, Google Chirp 3 HD: $30

- ElevenLabs Flash/Turbo (~75 ms latency): $50

- ElevenLabs Multilingual v2/v3: $100

For a voice agent doing ~500k chars a month (~500 min), that’s $2 vs $7.50 vs $25 vs $50 for the same minutes, a 25x spread. One pattern that works: use a fast low-latency voice for the filler phrases and short acknowledgements AemonAlgiz mentioned, where first byte matters most, and don’t pay flagship rates for longer passages you can buffer. Monthly free tiers also go a long way while prototyping: Google gives 1–4M chars/month depending on voice class, and Azure gives 500k.