Coordinating a real-time voice model with a separate task-execution agent - architecture writeup, looking for feedback

Research
1

Coordinating a real-time voice model with a separate task-execution agent — architecture writeup, looking for feedback

Posting this in Research since it’s really a question about model coordination more than a specific library issue.

The problem we were solving: an agent needs to hold a full-duplex voice conversation while also doing complex visual reasoning over a screen and executing device actions in the background without either one blocking the other.

Full-duplex speech models like Moshi (parallel-stream, processing user and system audio simultaneously) and the dual-tower approach in Generative Spoken Dialogue Language Modeling are genuinely good at natural turn-taking and simple tool calls. What they’re not necessarily built for is complex, long-horizon visual reasoning over a device task, which is the other half of what we needed.

So instead of trying to get one model to do both jobs well, we split it: a real-time voice model stays present in the conversation, a separate and stronger model handles the actual device-control task in the background, and the two coordinate asynchronously through a task queue rather than sharing one agent loop.

A few design decisions that came out of this that might be relevant to others working on similar foreground/background coordination problems:

  • Task completion and task success are tracked as genuinely separate signals. Completed just means execution reached an end state: it says nothing about whether the goal was achieved. Keeping these distinct means the conversational layer never has to guess whether a “finished” backend task actually succeeded.
  • Results reaching the foreground go through a queue with a 500ms aggregation window: if a second result lands while the window’s open, it extends and the results get merged before delivery, rather than firing off multiple competing responses in a row.
  • Two runtime-specific message types get converted to ordinary UserMessage objects at the model boundary: one carries environment state changes (avoiding a full system-prompt rewrite, and the cache invalidation that comes with it, every time something shifts), the other carries runtime events like loop-detection corrections or backend task results, through one consistent channel.
  • Device tasks are executed strictly serially by design: most tasks need exclusive ownership of a screen and input path, and we decided the ambiguity of concurrent execution wasn’t worth trying to solve yet.

Full technical writeup: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/

This is dev-board stage, not validated across every device/OS/audio configuration. Interested in how others here have approached coordinating a real-time conversational model with a separate, more capable model for the actual task, particularly curious about interruption handling when the backend is mid-task and new voice input arrives, since that’s the part we’re least confident is fully solved.

2

The async task-queue split makes sense, but I’d add a third signal next to completed and succeeded: still relevant. One failure mode this design invites is a task succeeding after the user has already revised or abandoned the intent, so the foreground delivers a now-wrong result.

I’d key each task and result to a conversation-state generation. The 500ms merge window still helps with presentation timing, but does the queue have any reverse path for the foreground to supersede or cancel work that’s already in flight?

3

Good point. We already support explicit cancellation for queued and in-flight work. We intentionally do not infer that every new turn or conversation end supersedes existing background work: users may have independent requests in flight, and completed work is expected to report back after standby.

An intent generation would be appropriate if we add an explicit “replace this work” semantic.For the current design, cancellation remains explicit to avoid discarding unrelated work.

4

The split is sensible, and “still relevant” should probably be a first-class state beside completed and succeeded. A backend action can succeed perfectly after the user has changed intent, navigated elsewhere, or explicitly cancelled.

I would attach an intent/version token to each task and require the foreground agent to reconcile the result against the current token before speaking or applying it. Cancellation also needs an acknowledgement path: “stop requested” is not the same as “side effects can no longer occur.” The 500 ms aggregation window helps presentation, but ordering and supersession rules are what keep a polished response from reporting an obsolete success.

5

Spot on. User intent can easily shift during the window between task completion and speech output. Because the announcement itself must wait for the foreground agent to finish its current output, this temporal window is amplified even further.

At the same time, in practical scenarios, the side effects on the device have usually already taken place by the time a task completes. In those cases, the response functions more like a notification to keep both the user and the foreground agent in sync with system changes, rather than an “obsolete success” that should simply be dropped. Therefore, whether to aggressively filter task responses based on intent requires further consideration.