Voice agent latency degrades after turn 7-8 despite fixed system prompt + limited history — looking for mitigation ideas beyond what we've already tried

Suppose the prompt layout is approximately:

stable system prompt
+ current RAG
+ last six completed exchanges
+ new user message

While the conversation is still filling the six-exchange window, each new prompt can look mostly append-only. When the seventh completed exchange arrives, the oldest exchange is removed from near the front of the dynamic history and a new one is appended at the end.

Even if the resulting prompt has nearly the same number of tokens, it may no longer share the same long exact prefix with the previous rendered prompt.

That distinction matters because prompt/KV reuse is tied to the rendered token sequence, not merely to semantic similarity or total token count. Ollama’s recent releases explicitly mention improvements to prompt caching and KV reuse, so I would avoid the blanket statement that it necessarily reprocesses everything on every call (Ollama releases). At the same time, reuse is model-, version-, template-, and prompt-shape-dependent, so it should be measured rather than assumed.

In my synthetic control:

• the first six-exchange eviction occurred at request turn 8;
• median prompt_eval_duration was about 0.206 s before the eviction and about 0.465 s afterward;
• the boundary request was about 0.513 s;
• an append-only control at the corresponding turn was about 0.152 s;
• the computed common-prefix fraction fell to roughly 54%;
• debug logs showed little useful checkpoint restoration at that boundary.

The wall-clock first-content latency was noisier and did not show an equally clean single-turn jump, so I would treat this as a credible mechanism candidate, not the production root cause.

A cheap production test would compare turns 6–9 under:

A. append-only history control
B. last six completed exchanges
C. last six individual speaker messages
D. same token count, but one early prompt mutation
E. same token count with a stable prefix

The important artifacts are the exact rendered prompts or hashes, common-prefix length, prompt_eval_count, prompt_eval_duration, and cache/checkpoint debug lines. This also resolves an ambiguity in “six turns”: six speaker messages and six complete exchanges reach their first eviction at different request numbers.

I think a running summary is reasonable, but I would evaluate three independent questions:

1. Information loss: did it drop or mutate a correction, commitment, exact number, consent scope, or completed tool result?
2. Prefix churn: did rewriting the summary invalidate a large reusable prompt prefix?
3. Compute contention: did generating the summary occupy the same Ollama runner/GPU when the next voice turn arrived?

A structure like this seems safer:

Canonical operational state
├── exact identity/contact fields
├── appointment state
├── consent and cancellation state
├── completed tool results
└── versioned events

Prompt projection
├── stable policy and tool definitions
├── compact current-task state
├── periodically refreshed lossy summary
├── recent raw turns
└── relevant RAG evidence for this turn

The free-text summary can hold the caller’s goal, relevant background, unresolved questions, and conversational commitments. I would not make it the only source of truth for exact phone numbers, dates, authorization, consent, cancellations, or whether an external action succeeded.

This is also consistent with Retell’s current recommendation to keep explicit per-call state on the server rather than asking the model to infer all operational state from the transcript (custom LLM best practices).

Refresh cadence

The more promising result in my cache probe was not simply “put the summary later.” It was keep it stable for an epoch.

When the summary was rewritten every turn, prompt evaluation remained around 0.50 s in both of the tested positions. When it was refreshed only every few turns, refresh turns were relatively expensive (around 0.56 s), but the append-like turns between refreshes were around 0.15 s.

So I would try refreshing on events such as:

• a task-phase transition;
• a confirmed correction or tool result;
• a token threshold;
• an important fact about to leave the recent raw window;
• a coarse epoch rather than every turn.

I would not claim that placement is irrelevant — my placement variation was limited — only that update frequency and stability mattered much more in that fixture.

Scheduling the summary

A background coroutine is not automatically background capacity.

With a single runner slot, a summary request submitted first made the short voice request wait several seconds. Starting maintenance only after the first speakable voice clause largely protected response onset.

A reasonable admission policy is:

Summary work is pending
|
|-- Can it wait?
|      -> start after first speakable clause or turn completion
|
|-- Can it use separate capacity?
|      -> separate resident model, Ollama instance, GPU/server, or CPU path
|
|-- Must it overlap on the same model?
       -> verify actual parallel slots and benchmark the tradeoff

I would also coalesce pending summaries (“latest version wins”), stamp the input/output with a state version, discard stale results, and avoid submitting maintenance while a voice request is already waiting.

Ollama’s documented default is one parallel request per model. When capacity is unavailable, requests queue; increasing OLLAMA_NUM_PARALLEL also increases context-memory allocation roughly with parallel count × configured context (Ollama FAQ).

There are two important cautions:

1. Setting the environment variable does not prove the model architecture actually received that many slots.
2. Parallel inference shares GPU compute; it can improve fairness without improving total throughput.

In an earlier Qwen3.5 control, the server logged that the architecture did not support parallel requests and created only one slot. So that result was invalid as a parallel=2 test. In the later Llama 3.1 8B control, I verified runner.parallel=2, two distinct slot IDs, and actual overlap.

For the valid Llama 3.1 comparison:

So two slots reduced the queue-induced penalty from roughly 6.7–7.6 s to about 0.63 s — around a 91% reduction in that particular penalty.

But it did not create free compute:

• the voice request under contention was still about 70% slower than the parallel=2 solo baseline;
• summary generation also became slower;
• model memory rose by roughly 256 MiB at a 2K context in that T4 fixture;
• peak power increased;
• total pair completion time was approximately flat or slightly worse.

I would therefore describe parallel=2 as a latency/fairness guardrail when overlap is unavoidable, not as the first-line solution and not as a throughput doubling switch.

Preferred order:

1. avoid unnecessary every-turn maintenance;
2. coalesce it;
3. start it after the first voice clause or after the turn;
4. isolate it onto separate capacity where practical;
5. only then benchmark true same-model parallelism.

If using it, verify the server’s actual runner/slot logs rather than trusting configuration alone. There are model-specific cases where requested parallelism is downgraded; for example, this has been reported for Qwen3.5 (Ollama issue #14621).

Ollama’s native API streams by default; stream:false disables it (Ollama streaming documentation). Retell accepts partial response events grouped by response_id, with only the final event marked content_complete:true (LLM WebSocket protocol).

The useful pipeline is therefore:

Ollama NDJSON stream
→ FastAPI stream reader
→ short speakable-clause buffer
→ serialized Retell WebSocket writer
→ Retell TTS
→ audio playout

I would check for buffering at every boundary:

• stream:false accidentally set;
• SDK returning a completed response rather than an async stream;
• chunks accumulated into a list/string before forwarding;
• reverse-proxy buffering;
• waiting for a long full sentence rather than a short clause;
• WebSocket writer backlog or multiple competing writers;
• time between ollama_first_content, first_speakable_clause, and first_retell_send.

Retell explicitly says it starts speaking when it has the first complete sentence, and recommends streaming rather than putting the whole generation inside caller silence (best practices).

In my short fixture:

first content             ~0.825 s
first speakable clause    ~1.12 s
full completion           ~2.63 s

So forwarding the first short clause could avoid about 1.51 s compared with full-response buffering. The gap from first content to a speakable clause was only about 0.29 s, which also suggests that forwarding single-token fragments may not be worth the TTS overhead or awkward prosody.

A simple flush rule could be:

flush when punctuation + minimum length is reached
OR when a bounded maximum wait expires

Streaming will not fix model load, queueing, prompt prefill, TTFT itself, GPU contention, or TTS/ASR latency. It only lets useful generated content leave the backend earlier.

I would distinguish:

direct routing:
  choose one model before generating this turn

cascade:
  call the small model, then possibly call the large model too

For a voice hot path, direct routing is usually the safer default because a cascade adds another synchronous model call on escalation. Retell’s own guidance notes that chained LLM calls may be unfavorable for conversational latency (custom LLM best practices).

Rather than turn >= 7, possible routing inputs are:

• task type and ambiguity;
• latency deadline;
• action/tool risk;
• current queue/GPU pressure;
• model residency;
• acceptable degradation for this response.

A smaller model may be suitable for a short acknowledgement, simple clarification, fixed workflow navigation, or reading already-confirmed state. I would keep the stronger model for multiple corrections, ambiguous intent, sensitive replies, high-risk tool selection, and conflicting memory/state.

The residency branch matters:

Both models fit and remain resident
    -> benchmark direct routing

Only one model fits
    -> measure unload/load and frequent swap cost

Helper shares the same GPU
    -> measure contention, not just standalone helper speed

Separate capacity exists
    -> helper/background processing becomes more credible

In my state-extraction probes, a Qwen3.5 4B residual model improved recall, but when it was called broadly the synchronous extraction path was still around 3.1 seconds. “Small model” did not automatically mean “safe for every voice turn.”

So I think your idea is a legitimate overload policy, but I would route by task/deadline and verify residency rather than tying it directly to conversation depth.

KV/cache

I would separate:

1. Model residency — weights remain loaded (keep_alive).
2. Runtime-owned prompt/KV reuse — Ollama internally reuses eligible prompt state/checkpoints.
3. Application-owned persistent conversation state — the caller receives a session handle and explicitly resumes that exact inference state next turn.

keep_alive:-1 addresses (1), not (3).

Current Ollama does have internal prompt caching, and recent releases include KV-reuse improvements (releases). But the currently documented interfaces do not expose the portable application-owned conversation handle you appear to be looking for. The OpenAI Responses compatibility is explicitly non-stateful: previous_response_id and conversation are not supported (OpenAI compatibility).

That does not imply zero internal reuse. The practical test matrix is:

exact repeat
append-only messages
first sliding-window eviction
one early-token mutation
summary rewritten every turn
summary held stable for an epoch
RAG inserted early
RAG appended late

For each condition, save the exact rendered prompt/template, prompt token count, common prefix, Ollama version, prompt-eval metrics, and debug cache/checkpoint lines.

Speculative decoding

Ollama currently documents draft_num_predict: the maximum speculative draft tokens per step when a draft model is available, with support for a separate draft model or embedded MTP tensors (Modelfile reference).

That changes the answer from “not supported” to “conditionally available.” It does not establish that a useful compatible draft setup exists for llama3.1:8b-instruct-q4_K_M, nor that it would improve the stage currently causing your dropped calls.

I would use this branch:

load / queue / prompt prefill / relay dominates
    -> speculative decoding is not the first lever

decode-to-first-clause dominates
and a compatible draft/MTP path exists
    -> benchmark it

no practical compatible draft path
    -> only consider a runtime migration after estimating the gain

A few patterns from mature realtime/VTuber-style agents seem directly transferable here.

Scope everything to a response generation

response_id / generation_id
├── RAG
├── summary
├── Ollama generation
├── outbound text chunks
├── TTS/audio
├── tool calls
└── state-update candidates

Retell assigns an auto-incrementing response_id; when it asks for a new response, prior responses are discarded (WebSocket reference). I would mirror that lifecycle internally: cancel old work where possible, and reject late chunks/actions even when cancellation is not instantaneous.

Keep the history aligned with what the caller actually heard

Generated text, sent-to-TTS text, and actually played text are not always the same after an interruption.

Open-LLM-VTuber retains only content before interruption in chat history/memory. LiveKit similarly truncates conversation history to the portion the user heard.

That avoids two opposite errors:

• keeping an unseen generated tail and acting as though the caller heard it;
• dropping everything and making the agent repeat content that was already spoken.

It may also prevent “ghost history” from adding unnecessary assistant tokens over many interrupted turns.

Treat tool writes separately from generated speech

Retell now recommends server-owned call state, idempotent side effects, current-response checks, and raw-frame logging (best practices). This matters because a response may be discarded after a tool write has already committed.

For read-only work such as retrieval, cancellation and late-result rejection are usually straightforward. For appointment creation, cancellation, SMS, payment, or call termination, I would add idempotency and an explicit commit boundary. Pipecat’s APIs similarly distinguish function-call cancellation/interruption behavior (function-call frames).

One practical lifecycle is:

final/stable transcript
→ parser or LLM candidate
→ evidence / correction / negation checks
→ current-response and authorization check
→ idempotent deterministic commit
→ spoken confirmation

In my text-level probes, letting the LLM directly patch canonical state produced wrong or unauthorized updates. Letting it propose evidence-backed candidates while deterministic code validated/committed them prevented wrong commits in the tested fixtures. That is only a small mechanism probe, but it reinforces the separation between conversational memory and operational state.