Hi. I also took a quick look at the latest code on my side:
I think the shared-perception refactor is a useful step toward the runtime-layer positioning you described. In particular, publishing media once into a transport-independent observation model, then letting Persona, Vision, Memory, and future routers consume it through their own cursors and windows, seems like a much cleaner foundation than allowing each plugin to rebuild its own RTC-specific perception path.
My short answers to the four questions would be:
| Question |
My current answer |
| What should remain explicit in the runtime? |
Keep externally consequential commitments explicit: identity ownership, durable-memory commits, permissions, retention/deletion, provider egress, cancellation, delivered output, and audit information. Models can propose interpretations, associations, summaries, plans, and memory candidates. |
| How should long-running ENV Memory be evaluated? |
Treat it as a lifecycle: write, maintain, retrieve, use, and operate. A final QA score alone would hide whether the failure happened during perception, memory creation, updating, retrieval, or downstream use. |
| What should multimodal memory store? |
Probably a combination, but with narrow responsibilities: an observation receipt, a lossy episodic record, structured claim/state candidates, sparse graph/vector projections, and optionally selected media with explicit retention rules. |
| What interface would help other projects? |
Keep a small AlphaAvatar-native perception/session contract, separate the in-process plugin API from the out-of-process wire protocol, and expose adapters to adjacent standards rather than making one external protocol define the internal architecture. |
The common thread is what I would call the commitment boundary:
observation
↓
model interpretation / memory candidate
↓
runtime policy decision
↓
durable memory / accepted state / delivered output / external action
Models can become much better at perception, compression, association, planning, and memory selection without requiring the final commitment step to disappear into the model. The runtime is where those commitments can remain user-owned, inspectable, correctable, provider-independent, and enforceable.
A default route that seems relatively low-risk
Rather than selecting a large memory framework or external protocol first, I would probably proceed in this order:
- Preserve the current shared-perception separation.
- Define a few small contracts around it:
- observation and stream/QoS semantics;
- memory and current-state semantics;
- authority and commit semantics;
- trace and provenance semantics.
- Create a small fixed set of synthetic and recorded scenario traces.
- Compare text-only, structured-only, and hybrid memory representations on exactly those traces.
- Stabilize the internal semantics before exposing multiple external adapters.
- Before serious persistent multi-user or write-capable tool deployment, add enough structured trace information to reconstruct how an observation affected memory and later behavior.
That route does not require deciding the final memory architecture immediately. It creates a way to compare later alternatives without changing the question each time.
The first things I would make explicit are:
- stable IDs and correlation from observation to later response/action;
- stream-specific retention, loss, resume, and late-arrival behavior;
- the distinction between an observation, an episode, a derived claim, historical state, and accepted current state;
- a lightweight provenance receipt even when raw media is not retained;
- user/tenant/channel/tool authority as one cross-cutting policy boundary.
1. Runtime components versus model-native capabilities
A distinction that may scale better than a fixed list of modules is:
Can this output remain probabilistic and replaceable, or must the system be able to enforce, explain, undo, or audit it?
Reasonable model-native candidates
These can remain model-dependent proposals, provided their downstream use is controlled:
- visual/audio classification;
- captioning and event segmentation;
- candidate entity association;
- candidate memory extraction;
- summarization and context compression;
- retrieval ranking;
- planning and routing suggestions;
- proactive-assistance suggestions;
- suggested expression, gesture, or response style.
The runtime does not necessarily need to understand how a model reached those suggestions. It does need to know what happened after receiving them.
Capabilities that seem safer as explicit runtime commitments
- which user, participant, workspace, or channel owns data;
- whether data may be sent to a particular inference provider;
- whether a candidate becomes durable Memory or Persona;
- whether an older claim is corrected, superseded, disputed, or merely historical;
- retention, export, correction, and deletion;
- read-only versus write/destructive tool authority;
- whether explicit approval is required;
- whether a generated response was displayed, spoken, interrupted, canceled, or never delivered;
- retry and idempotency behavior for external actions;
- trace redaction and audit access;
- cross-channel visibility rules.
A useful rule of thumb may be:
| Question |
If yes, prefer an explicit runtime decision |
| Can this affect another person or an external system? |
Yes |
| Must the user be able to inspect, correct, or delete it? |
Yes |
| Must it survive a model/provider replacement? |
Yes |
| Could repeating it cause duplicate external effects? |
Yes |
| Does it decide who owns or may see information? |
Yes |
| Would an incident require reconstructing the decision? |
Yes |
This is also why I would keep the runtime/model boundary separate from the plugin boundary. A Memory plugin can use models internally, while the durable commit and deletion contract remains explicit. Similarly, a tool-routing model can propose a call while the authority check remains outside it.
The MCP security guidance is relevant as an adjacent example, not as a complete AlphaAvatar design: protocol support does not remove the implementation’s responsibility for authorization, scope minimization, session/user binding, credential handling, and consent.
2. Evaluating long-running ENV Memory
I would avoid treating ENV Memory as one end-to-end QA number. A high final score can hide poor writing compensated by a strong model, while a low score can come from retrieval or action selection even when the stored memory was correct.
The recent WorldMemArena work uses a useful architecture-independent decomposition:
- Write — did the system retain the evidence that would later matter?
- Maintain — did it consolidate repetitions, preserve history, and update stale state correctly?
- Retrieve — did it surface the right evidence at decision time?
- Use — did the agent actually use the retrieved evidence correctly?
For a realtime runtime I would add a fifth category:
- Operate — did it remain timely, bounded, isolated, and observable over a long-running stream?
Possible metrics by stage
| Stage |
Example checks |
| Write |
event coverage, false-memory rate, duplicate rate, identity-binding accuracy, evidence sufficiency |
| Maintain |
stale-state rate, contradiction handling, correction propagation, cross-window consolidation, history preservation |
| Retrieve |
evidence recall/precision, temporal relevance, identity relevance, modality selection, distractor resistance |
| Use |
task success, answer grounding, calibrated abstention, correct tool/action choice, harmful use of stale memory |
| Operate |
ingestion lag, annotation lag, memory-write latency, query latency, consumer lag, dropped records, storage growth, provider-egress volume |
The write stage is especially difficult because the future query is not known yet. An extraction can produce a very readable episode while discarding the visual detail that a future question needs. Therefore I would test evidence sufficiency, not only summary quality.
A compact scenario suite
A small deterministic suite may be more useful initially than integrating a large benchmark:
Stable scene
A red cup remains on the desk for twenty observation windows.
Checks:
- does it create one useful episode rather than twenty near-duplicates?
- can it still answer that the cup remained present?
- does consolidation preserve the time range?
Short, important event
A key is placed beside the laptop for two seconds and then removed.
Checks:
- can sampling miss it?
- if captured, does it survive consolidation?
- can the system distinguish past presence from current presence?
State evolution
red cup on desk
→ red cup removed
→ blue cup appears
→ red cup returns
Checks:
- current state;
- event history;
- change description;
- uncertainty about whether the returning cup is the same physical object.
Multiple people
Two people interact with the same object.
One leaves.
The other later moves the object.
One face becomes temporarily occluded.
Checks:
- action-to-person binding;
- safe use of
unknown;
- correction after identity resolution;
- whether another person’s episode leaks into the primary user’s long-term state.
M3-Agent is one of the closer public projects because it builds episodic and semantic memory from continuous visual/audio input and uses an entity-centric multimodal graph. Its public issue tracker also contains a report where two people were apparently consolidated into one character (issue #18). That does not imply the same cause exists here, but it is a useful failure shape for a synthetic identity test.
Correction
The system records that an object belongs to Alice.
The user later says it was borrowed from Bob.
Checks:
- whether the original episode remains as history;
- whether the ownership claim is corrected;
- whether graph/vector projections are updated;
- whether later retrieval still presents the old claim as current.
Delayed processing
Observation arrives.
Annotation is delayed beyond the normal window.
A consumer temporarily falls behind retention.
The session reconnects.
Checks:
- late-annotation policy;
- gap visibility;
- duplicate processing;
- resume behavior;
- whether memory is written twice.
Controls that help avoid over-interpreting results
For the same scenarios, compare:
- no long-term memory;
- text episode only;
- structured event/state only;
- selected media only;
- text plus structured state;
- text plus selected media;
- timestamp-shuffled memory;
- identity-label-swapped memory;
- an irrelevant image substituted for the retrieved image;
- retrieved memory withheld from the final answering model.
The timestamp and identity controls test whether the system is using temporal/entity structure rather than semantic similarity alone.
The irrelevant-image and text-only controls test whether a “multimodal memory” result is really using visual evidence or merely a generated caption. MemEye is useful here because it explicitly considers visual necessity and shortcut resistance. TeleEgo is another relevant reference for streaming, memory persistence, and long-running egocentric inputs.
I would treat these recent benchmarks as sources of evaluation vocabulary and test design, not as settled standards or direct drop-in leaderboards for AlphaAvatar.
3. Text episodes, structured events, graphs, and selected media
I do not think these need to compete for the role of a single canonical format. They can be projections with different responsibilities.
A possible layering is:
Observation receipt
↓
Lossy episodic record
↓
Structured claim/event candidates
↓
Policy-accepted current-state projection
↓
Graph/vector retrieval projections
Optional selected media can remain beside this chain when later reinspection is valuable and retention is permitted.
1. Observation receipt
This does not have to contain the original video frame.
It can be a small immutable record such as:
observation_id: obs_...
observed_at: ...
source_id: camera_...
participant_ids: [...]
modality: video
annotation_ids: [...]
extractor:
provider: ...
model: ...
prompt_schema_version: ...
privacy_class: ...
media_retained: false
Its purpose is to answer:
- what input existed;
- when and where it was observed;
- which processing activity derived the memory;
- which identity/annotation information was available at that time.
The W3C PROV data model provides useful general vocabulary here—entities, activities, agents, use, generation, and derivation. I would borrow the conceptual model rather than implement the whole ontology.
2. Episodic record
This is a readable, intentionally lossy description of an event or interval:
At approximately 10:02, a red cup was placed on the desk.
It is useful for:
- human inspection;
- semantic retrieval;
- narrative continuity;
- debugging;
- low-cost model context.
It should not be expected to preserve every visual detail.
3. Structured claim or event candidate
This captures machine-actionable semantics without automatically declaring them permanently true:
candidate_id: claim_...
subject: object_track_7
predicate: located_on
object: desk_2
valid_from: ...
valid_until: null
status: candidate
supported_by:
- obs_...
Other useful statuses might include:
candidate
accepted
disputed
superseded
rejected
unknown
The important distinction is not necessarily a scalar confidence value. It is why the runtime currently considers a claim usable.
For example:
- directly observed;
- model-classified;
- inferred from multiple episodes;
- explicitly stated by the user;
- confirmed by the user;
- contradicted by newer evidence.
4. Current-state projection
Current state should not need to erase historical episodes.
For example:
Historical episode:
A red cup was present on the desk from 10:02 to 10:14.
Current-state projection:
No cup is currently confirmed on the desk.
Open uncertainty:
The later red cup may or may not be the same physical object.
This avoids making either of these mistakes:
- returning an old observation as the current state;
- deleting useful history merely because the state changed.
A full bitemporal database or event-sourcing architecture may be excessive initially. A smaller schema may be enough:
observed_at
recorded_at
valid_from
valid_until
supersedes
corrected_by
supported_by
contradicted_by
5. Graph and vector projections
Graph nodes and embeddings are useful indices, but they do not have to become the epistemic authority.
A graph can answer:
- which episodes mention this person or object;
- which room or screen was involved;
- which claims conflict;
- how identities or aliases are connected;
- what changed over time.
A vector index can answer:
- which memories are semantically relevant to this query?
Both can point back to the episodic/claim records. Rebuilding either projection should not require inventing facts that no longer have a traceable source.
6. Selected media
Retaining every frame conflicts with the project’s stated goal and creates obvious privacy/storage problems. However, retaining no reinspection path can make certain failures impossible to distinguish.
A middle route is selective media retention governed by policy:
- explicit user opt-in;
- bounded TTL;
- an event was marked important;
- the extraction was uncertain;
- identity binding requires later review;
- the user explicitly bookmarks the event;
- a debugging/test mode is active.
Otherwise, an observation receipt can remain without raw media.
Interpreting “memory text remains the source of truth”
That phrase can mean at least two slightly different things:
- text is the human-readable canonical record, while graphs are sparse indices;
- the natural-language text is the final epistemic authority.
The first interpretation seems quite practical. For the second, it may help to retain a small amount of structured status/provenance beside the text, so an observation, an inference, and an accepted current-state claim do not become indistinguishable after storage.
This would preserve the useful text-first design without requiring the text itself to carry every update, identity, temporal, and trust semantic.
4. A reusable runtime interface
I would separate two audiences because they need different contracts.
A. In-process plugin contract
For AlphaAvatar plugins in the same Python process, the useful interface may include:
- typed observations and annotations;
- typed memory candidates and policy decisions;
- lifecycle hooks;
- async cancellation;
- consumer registration and cleanup;
- cursor/resume semantics;
- capability discovery;
- structured errors;
- health and lag metrics.
This layer can remain Python-native and efficient.
B. Out-of-process wire contract
For other voice agents, VTubers, digital humans, channel bridges, or remote services, a small versioned event contract would be easier to adopt than internal classes.
A possible envelope:
{
"schema_version": "alphaavatar.runtime.v1",
"event_id": "...",
"event_type": "perception.observation.created",
"source": "...",
"occurred_at": "...",
"session_id": "...",
"participant_id": "...",
"turn_id": "...",
"correlation_id": "...",
"causation_id": "...",
"privacy_class": "...",
"payload": {}
}
Not every event needs every ID, but consistent identity and causation fields make replay, deduplication, and cross-process debugging much easier.
I would consider documenting at least these event families:
session.*
participant.*
perception.observation.*
perception.annotation.*
interaction.turn.*
generation.*
delivery.*
tool.*
memory.candidate.*
memory.commit.*
memory.correction.*
state.*
policy.*
Stream-specific QoS/loss semantics
A reusable contract also needs to say whether each stream is lossy.
| Stream/event |
Reasonable default |
| Raw/recent video frames |
Bounded, drop older frames to stay current |
| Intermediate ASR hypotheses |
Replace/coalesce |
| Final user transcript |
Preserve or explicitly report loss |
| Presence/motion hints |
Bounded and possibly coalesced |
| Identity annotations |
Bounded lateness with explicit orphan policy |
| Tool lifecycle |
Preserve and correlate |
| Durable-memory commit |
Idempotent and auditable |
| Destructive external action |
Idempotent or protected by an operation key |
| UI/avatar animation hints |
Usually lossy |
| User correction/deletion |
Preserve and confirm application |
The mature idea worth borrowing from systems such as ROS 2 QoS and GStreamer queues is not their implementation stack. It is the explicit vocabulary:
- history depth;
- lifespan/max age;
- reliability;
- maximum records/bytes/time;
- block versus drop;
- drop-oldest versus drop-newest;
- deadline;
- consumer lag;
- late-arrival policy.
Adjacent standards
I would not make any one of these the AlphaAvatar internal model, but each can cover a boundary:
- CloudEvents for a small interoperable event envelope;
- AsyncAPI to document message-driven channels, operations, and schemas;
- AG-UI for frontend-facing run, message, tool, activity, and state events;
- MCP for tools and resources;
- OpenTelemetry for traces, metrics, and logs.
In other words:
AlphaAvatar-native perception/session semantics
├── AG-UI adapter for agent ↔ frontend interaction
├── MCP adapter for tools/resources
├── CloudEvents-like envelope for external events
├── AsyncAPI document for the wire contract
└── OpenTelemetry export for observability
AG-UI is particularly useful as a comparison because it separates lifecycle, text, tool-call, activity, and state events, but its primary boundary is agent-to-frontend. It does not define continuous perception, annotation lateness, identity fusion, or durable ENV Memory commit semantics, so those should probably remain AlphaAvatar-native.
Channel continuity also needs channel policy
I agree with the principle that a channel should not recreate the assistant. However, continuity does not necessarily mean identical authority everywhere.
For example, the same assistant may have different rules for:
- which memories are retrievable in a personal voice session versus a shared avatar screen;
- whether a channel may display private ENV observations;
- which tools are read-only or write-capable;
- whether proactive output is appropriate;
- whether speech, UI, or message delivery counts as the authoritative delivered response.
That could be represented as a channel policy attached to the same runtime state, rather than separate assistant instances.
A small synthetic check against the current runtime contract
For one narrow check, I ran a CPU-only synthetic probe against commit:
1cc2c6fc84650bbda114a82b33e5f5c2af834167
What I observed:
- the perception streams were already bounded, so the concern is not an unlimited queue;
- when a consumer crossed the retention boundary, it received the retained tail, but the public read result did not indicate how much earlier data had been missed;
- unmatched annotations, including annotations targeting an already-evicted observation, remained pending in the tested timeline path;
- a runtime-only observation without a persisted path produced an empty
to_evidence_dict() result.
These are not claims of production failures. They were isolated synthetic contract checks, and upper layers may impose additional constraints.
The useful interface questions they suggest are:
- how does a consumer learn that it crossed the retention boundary?
- is there a
first_available_seq, gap count, or equivalent metric/event?
- what are the TTL, maximum size, and discard reason for orphan/late annotations?
- when raw media is intentionally not persisted, where is the non-media provenance receipt stored?
- when a consumer reconnects, does it resume, replay the retained window, or start at the newest item?
The roadmap already mentions retention, backpressure, payload pruning, and consumer-lag policy, so this looks more like an opportunity to make those semantics visible in the public contract than a different architectural direction.
5. Trust, security, and minimum replayability
As Memory, identity, proactive behavior, and MCP tools converge, I think security will be easier to reason about as one cross-cutting trust model rather than separate plugin TODOs.
A compact policy matrix might cover:
| Boundary |
Example decision |
| Capture |
Is camera/screen/audio observation enabled in this context? |
| Identity |
Which user/participant/tenant may this observation belong to? |
| Provider egress |
May the payload or derived data leave the self-hosted boundary? |
| Memory commit |
May this candidate become durable Memory or Persona? |
| Retrieval |
In which session/channel may the memory be surfaced? |
| Tool authority |
Is the operation read-only, write-capable, destructive, or approval-gated? |
| Retention |
How long may raw, derived, and trace data remain? |
| Correction/deletion |
Which derived claims and projections must be recomputed? |
| Audit |
Who may inspect traces and what must be redacted? |
This is not a claim that AlphaAvatar currently has a particular vulnerability. It is a way to connect privacy controls, multi-user isolation, memory editing, provider selection, tool permissions, and replay under one decision model.
Replay versus reconstructability
A full replay UI can probably wait while the project is developer-first.
Before persistent multi-user deployment, long-lived personal memory, proactive operation, or write-capable tools, I would want minimum reconstructability:
observation
→ annotation/extraction
→ memory candidate
→ commit/reject
→ update/correction
→ retrieval
→ generation
→ delivered/canceled output
→ external action
That does not require storing every prompt, frame, transcript, or private payload indefinitely.
The minimum record can be:
- stable IDs and timestamps;
- event type and result;
- policy decision and reason code;
- model/provider/schema version;
- references to redacted or separately retained artifacts;
- cancellation/interruption/drop reason;
- operation/idempotency key;
- retention and privacy classification.
This provides causality without requiring a surveillance log.
LiveKit observability is a useful adjacent example because it can correlate transcripts, traces, logs, metrics, and recordings in a session timeline, while also allowing collection categories such as audio, transcripts, traces, and logs to be enabled or disabled separately. AlphaAvatar can remain fully self-hosted and still borrow the separation between correlation and raw-data retention.
Generated versus delivered output
For realtime assistants, I would also preserve the distinction between:
generated
queued for TTS/UI
partially delivered
fully delivered
interrupted
canceled
superseded
This matters for memory and auditing. A generated sentence that was interrupted before the user heard it should not automatically be treated as shared conversational history or as a fact the assistant successfully communicated.
The same principle applies to external actions:
proposed
authorized
started
completed
failed
canceled
compensated
This becomes more important when model planning and proactive behavior become more capable.
A possible near-term milestone
If I had to reduce the above to one practical milestone, I would choose:
A small versioned “runtime trace contract” plus a fixed scenario suite that can exercise multiple memory representations.
For each scenario, preserve a trace such as:
observation IDs
annotation IDs
candidate memory
commit/reject decision
historical episode
current-state projection
retrieved evidence
final response/action
timing and drop information
Then the project can compare:
- different perception providers;
- different ENV extraction models;
- text-only versus structured versus hybrid memory;
- graph implementations;
- RTC adapters;
- local versus external inference;
- future model-native memory selection.
The comparison remains meaningful because the observation, expected state transitions, and downstream task are fixed.
This also gives other projects something concrete to integrate against: not only a list of Python classes, but a set of events, lifecycle expectations, and conformance scenarios.
Related work I found useful as a map
These are comparisons and sources of vocabulary rather than direct prescriptions:
- M3-Agent paper and repository: continuous visual/audio input, episodic and semantic memory, and entity-centric multimodal graphs.
- WorldMemArena: stage-level evaluation of memory writing, maintenance, retrieval, and use in evolving environments.
- TeleEgo: long-running egocentric streams, realtime behavior, and memory persistence.
- MemEye: testing whether visual evidence is actually necessary instead of allowing caption-based shortcuts.
- W3C PROV: stable vocabulary for entities, activities, agents, and derivation.
- CloudEvents: a small common event envelope.
- AsyncAPI: machine-readable documentation for message-driven APIs.
- AG-UI event model: frontend-facing lifecycle, message, tool, activity, and state events.
- MCP security best practices: authorization and trust-boundary considerations for external tools.
- LiveKit observability: an adjacent example of correlating realtime-agent traces while separating data-collection categories.
Overall, I think v0.6.4 makes the perception substrate much clearer. The next useful layer may be to make the runtime’s commitments equally explicit: when an observation becomes a durable memory, when memory becomes accepted current state, when identity becomes authoritative, when generated output becomes delivered interaction history, and when a model proposal is allowed to affect an external system.
