NEON-CITY/CosySim and the NEXUS project

[KnackAU]

OK, this is an old one. This started out as me trying to get realtime voice working 8 months ago. I ended up having to learn spec decoding, chunking, finetuned router models etc. I built up a 1100 test and telemety suite around it, Custom LMStudio clients and server etc. It eventually evolved into what you see today. There is ALOT more here than meets the eye. This is a complete self managed, self improving, self finetuning, self coding local management system with a vector DB with 6 cascades that leverages notebooklm in ways it isn’t exactly meant to be used, With an entire Scene Creation system/Simulator with drag and drop or even single prompt scene generation and asset studio. Every scene uses a local AI for something. All scenes are independant - yet act inside a living, breathing city with a real economy, day/night cycles, alliances, news cycles etc.

Each agent has their own phone and can call, text, hack any other agent. You have The same phone. and much much more. It really is just a showcase of what is possible. All old credentials are expired/changed etc, so don’t bother. lol.

Anyway, as usual If you any part useful or interesting, feel free to take and use or improve on yourself! This is presented here for one purpose only - Providing as much information and idea’s as possible to everyone so people can take inspiration or find something useful and advance the field in some way!

CosySim

A local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world remembers.

35 launch targets · ~1,040 skills · 38-stage interceptor pipeline · 6-tier knowledge router · a training flywheel —
built almost entirely through agentic coding, and published so humans and AI agents can learn from it.

---

Why this repo exists. CosySim is meant to be read. It is a working, end-to-end example of what local agents + agentic coding can build: a living cyberpunk city whose residents reason on a local model, recall the past from a persistent knowledge base, react to a live economy and faction war, and quietly turn every interaction into training data that improves the next one. Take any piece you like — the interceptor pipeline, the LMStudio steering, the NLM↔Nexus flywheel, the ARGUS toolkit — and use it in your own project.

Start here

Pick the door that matches why you came:

You want to…	Go to	Deep-dive doc
Run it in 5 minutes	Quickstart	docs/OPERATIONS.md
Understand how it fits together	Overview & Architecture	docs/ARCHITECTURE.md
See the game / living world	NEON CITY	scene code in content/scenes/
Learn how agents are steered	Engine Internals	docs/MCP_FRAMEWORK.md
Understand the AI brain (local → frontier)	NLM + NEXUS	docs/NEXUS.md
Train / finetune / self-improve	CONTROL	engine/training/, training/
Wire external services	Integrations, Apps & CLI	docs/*_API_REFERENCE.md
Do web-app reconnaissance	ARGUS	docs/ARGUS_METHODOLOGY.md
Create scenes / assets	Creation Kit & Asset Studio	docs/DESIGN_SYSTEM_V2.md
Browse everything	—	docs/INDEX.md

Quickstart

Prerequisites: Python 3.13, LMStudio running on :1234 with a chat model loaded. Optional: ComfyUI (:8188) for image/video, a TTS server (:8600) for voice.

# 1. Install
pip install -r requirements.txt && npm install

# 2. Configure secrets (nothing real is committed — see "Security & configuration")
cp .env.example .env          # then fill in any keys you have; LMStudio works with no auth

# 3. Launch
python tui.py                 # interactive Terminal UI (recommended) — ←/→/↑/↓ to navigate, Enter to launch
python launcher.py --core     # or: auto-start core services + main scenes
python launcher.py neoncity   # or: a single scene → http://localhost:5563
python launcher.py --list     # see all 35 targets + live port status

Then open the hub at http://localhost:8500 — the NEON CITY landing page — and jack in.

# Handy
python cli.py ask "prompt"           # query the local model stack (38 models)
python scripts/oracle.py             # full system diagnostic (health · errors · perf)
python scripts/smart_test.py --smoke # fast test sweep (~15 files)

Security & configuration

This repo is safe to fork: no live credentials are committed. Real secrets live only in gitignored local files.

• .env (gitignored) — copy from .env.example and fill in what you have. Loaded automatically by engine/config.py. LMStudio needs no auth by default, so an empty .env is enough to run the world.
• config/default.yaml — all tunables; secret-shaped values resolve from ${ENV_VAR} at read time via a SecretManager hook.
• config/nlm_rpcids.yaml (gitignored, runtime-regenerable) — Google rpc IDs + keys; ship-safe template is config/nlm_rpcids.example.yaml with every key redacted.
• HARs, heap snapshots, dumps, cookies, account pools, and backups are all gitignored — capture intelligence stays local.

---

What is CosySim?

The NEONCITY “Dark Renaissance” landing — the front door to a city of local agents.

CosySim is a local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world they live in actually remembers. It runs 35 launch targets (18 game scenes + 11 services + 6 creation tools) as Flask/Socket.IO servers on your own machine, powered entirely by local inference (LMStudio), a persistent knowledge layer (Nexus KMS), and NotebookLM distillation. No cloud API is required for core gameplay.

It is built to be read. This is a flagship example of what agentic coding + local agents can produce: ~1,040 skills across 99 packs, a 36-stage interceptor pipeline, a 6-tier knowledge router, and a training flywheel — all wired together and documented so that humans and AI agents alike can learn from it and borrow from it.

The elevator pitch: Spin up a neon cyberpunk city on your laptop. Talk to its residents. They reason with a local model, recall what you did last week from a persistent knowledge base, react to a live economy and faction war, and quietly turn every conversation into training data that makes the next conversation better.

Why it’s unique

Claim	How it’s actually true (in the code)
Every NPC is a real local agent	Each reply runs through AgentGovernor.reply() in engine/mcp/comms_framework.py → VirtualAgent → LMSClient.infer_stream() against LMStudio on localhost:1234. No scripted dialog trees.
The city remembers	State is a live tree (MCPFramework singleton: scenes → characters → world → factions). Persistent memory, rules, and 3.7K+ Q&A pairs live in Nexus KMS (SQLite + FTS5, :8700).
Frontier-level results from local models	The 6-tier NexusQueryRouter answers from cache/vector/FTS first and only falls back to local inference last — then auto-stores the answer. NotebookLM (free Gemini, grounded) sits between as a distillation tier. Every interaction makes the next one cheaper and sharper.
It governs itself	36 interceptors (engine/agents/interceptors/) inject mood, knowledge, scene rules, faction standing, and heat awareness — then shape, log, and sync the response. Skills enforce cooldowns, costs, and prerequisites.
Open and inspectable	Vanilla-JS frontend (no build step), Google-style docstrings, version-stamped change logs, and a deep docs/ tree with INDEX.md as the door.

Architecture overview

CosySim is a reusable engine + swappable content + tunable config. Scenes subclass BaseScene; the engine provides agents, governance, knowledge, world simulation, and inference; YAML config tunes behavior without touching code.

┌──────────────────────────────────────────────────────────────────────────┐
│  BROWSER — Neon HUD v2  (vanilla JS · Jinja2 · Socket.IO client)          │
│  cosysim-telemetry.js · cosysim-particles.js · design_tokens.css (v2)     │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  Socket.IO / REST
┌─────────────────────────────────▼────────────────────────────────────────┐
│  35 LAUNCH TARGETS  (ports 5555–8800)                                     │
│  ┌── GAME (18) ──────┐ ┌── SERVICE (11) ─────┐ ┌── CREATION (6) ──────┐   │
│  │ penthouse, neoncity│ │ nexus_kms, hub,     │ │ canvas, asset_studio,│   │
│  │ oracle, casino,    │ │ tts, command_center,│ │ creation_kit,        │   │
│  │ heist, tavern, …   │ │ intel_hub, …        │ │ creator, …           │   │
│  └───────────────────┘ └─────────────────────┘ └──────────────────────┘   │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  SKILLS  +  MCP PIPELINE                                                   │
│  engine/skills/builtin/  ~1,040 @skill across 99 packs                     │
│  engine/mcp/  MCPFramework state tree · 43 tool modules · DialogSystem     │
│  engine/agents/interceptors/  36 pre/post hooks (AgentGovernor)           │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  ENGINE LAYER  (engine/)                                                   │
│   lmstudio/  ServerController · LMLink federation · StreamProcessor        │
│   nexus/     NexusClient · 6-tier QueryRouter · KnowledgePipeline · NLM    │
│   world/     WorldSim (economy ticks) · PlayerState · Missions · Crew      │
│   agents/    VirtualAgent · AgentGovernor · AgentRouter · OutputEvaluator  │
│   training/  DataCollector · BenchmarkRunner · FinetuneOrchestrator        │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  external processes
┌─────────────────────────────────▼────────────────────────────────────────┐
│  LMStudio :1234  ·  Nexus KMS :8700  ·  ComfyUI :8188  ·  TTS :8600       │
│  (local LLM)        (knowledge)         (image/video)     (Qwen3 voice)    │
└──────────────────────────────────────────────────────────────────────────┘

The engine subsystems map cleanly onto folders, so the diagram doubles as a directory map:

Layer	Where	Does what
Browser HUD	content/shared/, scene templates/ + static/	Neon HUD v2, particles, telemetry — pure vanilla JS, no build step
Targets	content/scenes/{name}/, engine/control_plane_registry.py	35 Flask/FastAPI/Streamlit/Node servers, each a BaseScene subclass
Skills + MCP	engine/skills/, engine/mcp/	@skill capabilities, the MCPFramework state tree, 43 @mcp_tool modules, governance
Engine	engine/lmstudio · nexus · world · agents · training	inference, knowledge, simulation, agent lifecycle, the data flywheel
External	LMStudio · Nexus KMS · ComfyUI · TTS	Local inference, knowledge backbone, media generation, voice

How a player message becomes an NPC reply

This is the heart of the system — the path a single message takes is the same for every NPC in every scene. It is implemented in AgentGovernor.reply() (engine/mcp/comms_framework.py) and the interceptor pipeline.

Player message  ──Socket.IO / REST──►  Scene  ──►  DialogSystem.add_turn()
        │
        ▼
 ┌─ AgentGovernor.reply() ────────────────────────────────────────────────┐
 │                                                                         │
 │  1. Build ResponseContext   (scene, agent_id, user_message, manifest)   │
 │                                                                         │
 │  2. AUTO skills             run BEFORE the LLM; cooldown + prereq gated  │
 │       e.g. search_memory → result injected into context                 │
 │                                                                         │
 │  3. run_pre()  ─ ~21 PRE interceptors, ascending priority ───────────►  │
 │       5  MoodDrift  ·  6 NexusPrompt (hydrate knowledge)                 │
 │       8  CharRegistry  ·  10 Router (pick model)  ·  15 Scene context    │
 │       40 FactionContext  ·  50 PersonalityGuard  ·  60 PolicyEnforcer    │
 │       (any PRE interceptor may set ctx["abort"] / ctx["skip_llm"])       │
 │                                                                         │
 │  4. LLM CALL   VirtualAgent.reply(governance_context)                    │
 │                  └► LMSClient.infer_stream()  ──SSE──►  LMStudio :1234    │
 │                                                                         │
 │  5. Parse   StreamProcessor / ContentRouter extract inline tags:        │
 │       [MOOD:happy]  [IMAGE:prompt]  [ACTION:x]  [STAT:health+10] [VOICE] │
 │                                                                         │
 │  6. run_post()  ─ ~7 POST interceptors ─────────────────────────────►   │
 │       75 HeatAwareness · 80 ResponseShaper · 85 TTS · 90 Logger          │
 │       92 MoodSync/SpectatorBroadcast · 93 Relationship                   │
 └─────────────────────────────────────┬───────────────────────────────────┘
                                       ▼
   STATE WRITES + CASCADE
     • CharacterRegistry mood / relationship deltas
     • PlayerState credits / rep / heat   ([STAT:…] tags)
     • ComfyUI image queue                ([IMAGE:…] tags)
     • OutputEvaluator.score → DataCollector  (training flywheel)
     • WorldSim / EventCascade → broadcast to other subscribed scenes
                                       ▼
   Scene emits via Socket.IO:  chat message · HUD update · portrait mood · TTS audio

A few details that make this more than a wrapper:

• ResponseContext is a mutable bag passed to every interceptor — system_prompt, messages, reply, auto_results, plus post-call metadata like mood_tags, response_id, and is_stateful. Interceptors read and write it in priority order; any PRE interceptor can abort the LLM call entirely.
• Skills have three trigger types (engine/mcp/comms_framework.py): auto (run before the call, result injected), optional (offered to the model as a tool), and required (the model must call it). Auto-skill invocation is throttled by a real COOLDOWN_TRACKER and prerequisite chain (v1.59.0).
• Stream tags are the action channel. The LLM doesn’t just talk — it emits [STAT:health+10], [MOOD:x], [IMAGE:prompt] inline, parsed by StreamProcessor (_RE_STAT = \[STAT:(\w+)([+-]\d+)\]) and routed to game state, mood, and the ComfyUI queue.
• The cascade is what makes the city feel alive. A WorldSim tick (~60s) emits SimEvents (economy / faction shift / NPC action / weather); EventCascade broadcasts to every subscribed scene, so an NPC’s action in one room ripples through the HUD and faction standings everywhere.

The self-improving loop (why local models punch above their weight)

Every answer flows through the NexusQueryRouter, which escalates only as far as it must and caches the result at the end:

Request → ① Q&A Cache (instant, 0 tokens)
        → ② Vector Search (Gemini Embedding + ChromaDB)
        → ③ FTS5 full-text knowledge
        → ④ Nexus Smart Ask (server-side pipeline)
        → ⑤ NotebookLM Ask (grounded, free Gemini)
        → ⑥ LMStudio fallback (local inference) → auto-stored back to ①

The first time a question is asked it costs compute; every subsequent identical question is served free from Nexus. Meanwhile OutputEvaluator scores replies and DataCollector turns them into JSONL — feeding BenchmarkRunner and FinetuneOrchestrator. The system is designed to compound: more play → more knowledge + more training data → better local responses.

---

NEON CITY — the living world & game mechanics

The city breathes. Six factions fight for control. The night never ends. — content/scenes/neoncity/neoncity_scene.py

NEON CITY is CosySim’s flagship: not a single screen but a persistent, autonomous world that the GAME-pillar scenes all plug into. The economy keeps moving, factions keep fighting over turf, NPCs keep walking their daily routines, and the weather keeps rolling through neon-soaked rain — whether or not a player is logged in. When you do walk in, the world has a memory: your last heist raised the heat, your faction standing changes the prices you’re quoted, and a botched job last night is still rippling through three other scenes.

The whole thing runs on local inference (LMStudio). No cloud, no API keys for the simulation itself — a city full of agents reasoning on your own GPU.

One world, many windows

The GAME pillar is a set of independent Flask/Socket.IO scenes (each on its own port, inheriting BaseScene), but they are windows into the same world state, not separate games. NEON CITY (:5563) is the hub; the rest are districts and venues you move through.

Scene	Display name	What it is
neoncity	NEON CITY	Living-world hub: economy, factions, missions, crew, cyberspace, board mode
phone	SIGNAL	Cyberdeck — messaging, 0xGH0ST contacts, mini-games, the always-on HUD
penthouse	THE PENTHOUSE	3D character room (three.js r184), curtain-wall skyline, autonomous NPCs
heist	THE HEIST	Crew-driven heist runs that feed heat and faction consequences back into the city
arena	THE COLOSSEUM	Combat matches; bookmaking and fighter queues spawned by the world sim
casino	CLUB NOIR	Gambling, VIP access gated by faction standing
lounge	THE VELVET PIT	Speakeasy social scene, ambient events, brokered deals
tavern	THE RUSTY ANCHOR	Fantasy RPG tavern, barter economy, dice
realm	LitRPG world	Dual-agent (Director + companion), d20 combat, inventory
grid	THE GRID	District board / strategy layer over territory control
games	THE ARCADE	Trivia, chess puzzles, leaderboards, AI opponents
…		plus gallery, cyberspace, and the broader GAME catalogue

The authoritative catalogue lives in engine/control_plane_registry.py (resolved to ports by engine/port_registry.py); the launcher, TUI, and the in-browser THE TERMINAL catalogue all derive their lists from it. See docs/SCENES.md for the full pillar tables.

The living world: two coordinated daemons

NEON CITY’s “aliveness” comes from background daemon threads in engine/world/ that tick on a game clock (1 real second ≈ 1 game minute; 60 real seconds = 1 game hour).

WorldSim (engine/world/world_sim.py) is the event engine. It fires scripted-but-stochastic events on per-task intervals — NPC actions every ~60s, ambient mood shifts, faction shifts, 0xGH0ST hacker messages, arena match queues, major world events, and passive economy ticks. Every SimEvent is (a) appended to a 200-entry ring buffer, (b) persisted to Nexus KMS as world_sim history, and (c) broadcast on the EventBus. A get_digest(scene) call returns “what you missed” the moment you enter a scene.

LivingWorld (engine/world/living_world.py) is the orchestrator. Each tick it coordinates every subsystem in order:

1. Game clock        → read time-of-day from WorldState
2. NPC routines      → RoutineManager moves NPCs to scheduled locations
3. Faction AI        → each faction makes one strategic decision (every 5th tick)
4. Market tick       → supply/demand drift, territory-weighted prices
5. Weather cycle     → Markov transition (CLEAR→NEON_RAIN→STORM→BLACKOUT…)
6. Stochastic events → fire + propagate consequences to market & NPCs

The subsystems, by file

Module	Responsibility
world_sim.py	Event templates, ring buffer, Nexus persistence, EventBus broadcast
living_world.py	Master tick loop, weather Markov chain, event consequence propagation
faction_ai.py	Autonomous per-faction decisions (expand / defend / sabotage / raid / negotiate)
market.py	Supply/demand pricing, buy/sell settlement, world-event shocks
territory.py	6 factions × 16 districts control map, crew HQ bonuses, faction war triggers
npc_routines.py / npc_state.py	NPC daily schedules and runtime location/activity state
player_state.py	Persistent credits / rep / heat / health / skills / faction standings
skill_progression.py	XP curves, d20-style skill checks, player level 1–50
mission.py / mission_chains.py	Missions, branching chains, outcome-driven consequences
crew.py	Recruitable NPC crews, role-fit skill-check operations
inventory.py / equipment_effects.py	Items, equipped-gear stat bonuses, consumable effects
faction_gates.py	Standing-driven shop access, pricing, mission visibility
event_cascade.py	Fans world events out to the scenes that subscribe to them

“The city remembers” — the v1.59 / v1.60 feedback loops

Earlier versions simulated a world but the simulation was largely cosmetic — agents could say [STAT:arousal+10], a faction could “expand territory”, and nothing actually changed. The v1.59 “consequential world” and v1.60 “Living Systems” passes closed those loops. This is the part worth studying: it’s where a pile of independent systems became a world with cause and effect.

Loop	Before	After (the closed loop)	Where
Stat tags applied	[STAT:trust-5] parsed then discarded	StatSyncInterceptor (priority 91) writes tags to real character state before mood rules evaluate them	engine/agents/interceptors/stat_sync.py
Economy settlement	Buying/selling was flavour text	Market._settle_buy/_settle_sell debit the wallet, move inventory, and raise heat for illegal goods	engine/world/market.py
World→market shocks	World events never reached the market	Market.subscribe_to_world_events() maps gang_war→weapons up, festival→luxury up, shortage→category surge	living_world._init_subsystems → market.py
Faction gating	Standing only gated casino VIP	faction_gates gives every scene shop access, ally discounts (−10%), rival surcharges (+15%), and standing-locked missions	engine/world/faction_gates.py
Player-aware factions	Factions ignored the player	Faction AI weights decisions by your standing — allies expand near you, rivals raid your turf; shifts broadcast on NEONCITY_FACTION_SHIFT	engine/world/faction_ai.py
Mission consequences	Missions resolved in isolation	MissionManager._apply_consequences applies rep/heat/faction-control deltas on success and failure; chains branch on outcome & standing	mission.py + mission_chains.py
Equipment matters	Equipped gear gave +0	get_equipment_bonuses aggregates equipped items into skill/stat deltas that skill-checks consume	engine/world/equipment_effects.py
Cross-scene ripples	Events stayed local	EventCascade fans WorldSim events to subscribed scenes (e.g. a CRIME event reaches phone, tavern, casino, heist)	engine/world/event_cascade.py

The net effect: a heist that goes wrong raises city-wide heat, which HeatAwarenessInterceptor (pipeline priority 75) makes NPCs aware of; the resulting faction shift reprices the black market; and the next mission in the chain unlocks a different branch because your standing changed. Every magic number here is config-driven (mission.consequences.*, economy.event_shocks.*, territory.faction_ai.*) so the whole consequence economy is tunable without touching code.

[KnackAU]

Game mechanics

Persistent player state lives in engine/world/player_state.py (a thread-safe singleton persisted to data/player_state.json, broadcasting hud_update over Socket.IO so the Neon HUD stays live across every scene):

• Vitals — credits (₵), reputation, heat / wanted level (0–100), health, hunger, energy
• Skills & XP — 8 skills (hacking, combat, stealth, social, tech, driving, medicine, trading) on a use-based XP curve (skill_progression.py), with d20-style checks: success = roll(1–20) + skill_level*4 + modifier ≥ difficulty, scaling from Trivial(5) to Legendary(25), and a global player level 1–50
• Factions — six powers (OmniCorp, NeoTech, BlackMarket, Ghost_Net, SynthSec, DeepState) each with its own personality and a standing scale of −100 (sworn enemy) → 0 → +100 (trusted ally)
• Territory — those factions contest 16 districts; control flows from missions, crew ops, and world events, and a >10% swing in one tick triggers a faction war that can cascade to adjacent districts
• Economy — six good categories (weapons, tech, consumables, contraband, intel, luxury) priced as base · (1 + (demand − supply)/100) with territory multipliers layered on top
• Inventory & equipment — items carry rarity/condition; equipping cyberware/weapons grants real skill and stat bonuses; consumables resolve effects by category
• Crew ops — recruit NPCs you’ve built relationships with into role-based crews; operations resolve via probabilistic skill checks (SUCCESS / PARTIAL / FAILURE) that shift loyalty and pay out scaled rewards
• Missions & chains — four branching storylines (heist escalation, faction war, deep-state defection, street-to-syndicate) where outcome and standing route you down divergent paths

See docs/GAME_SYSTEMS.md and docs/ECONOMY_GUIDE.md for the full mechanics.

Local-agent simulations: NPCs that perceive, decide, act

The defining trick of NEON CITY is that its inhabitants are local LLM agents running a real agent loop, not scripted dialogue trees. engine/agents/agent_loop.py runs a tick-based cycle for every character in a scene:

1. Perceive — observe location, nearby characters, and recent events (including world-sim digests)
2. Decide — VirtualAgentManager produces a structured JSON action against a fixed schema (speak, move, interact, idle, flirt, …) — batched across agents for parallel inference
3. Execute — the action is applied to the scene, broadcast over Socket.IO, and logged to the EventChain

DECISION_SCHEMA = {
    "type": "object",
    "properties": {
        "action": {"type": "string",
                   "enum": ["speak", "move", "interact", "idle",
                            "flirt", "touch", "kiss", "cuddle", "intimate"]},
        "target":  {"type": "string"},
        "message": {"type": "string"},
    },
    "required": ["action"],
}

Every reply an agent emits flows through the MCP interceptor pipeline (36 interceptors, priority-ordered), which is what wires dialogue into the world: NexusPrompt hydrates context from the knowledge base, FactionContextInterceptor (pri 40) injects the speaker’s standing toward you, HeatAwarenessInterceptor (pri 75) makes NPCs react to your wanted level, StatSyncInterceptor (pri 91) applies stat changes, and SpectatorBroadcastInterceptor (pri 92) pushes danmaku to onlookers. NPCs even drift through NaturalMoodDrift neurochemistry tagging between turns. Agent decisions are also fed into the DataCollector for the self-improvement training loop and auto-registered into Nexus’s agent registry. The architecture of that pipeline is documented in docs/MCP_FRAMEWORK.md and docs/ARCHITECTURE.md.

The result is a city where the bartender remembers the slight, the rival faction lieutenant prices you out, and a stranger across the lounge is — genuinely — deciding what to do next, locally, on your machine.

---

Engine internals: how agents are steered

The Oracle scene — a neural-consciousness terminal in NeonCity that doubles as the project’s All-Seeing Eye observability dashboard (real-time error feed, service-health grid, trace links).

Most “AI character” demos are a system prompt and a while loop. CosySim is the opposite: every agent reply passes through a governed pipeline of ~38 interceptors, the model’s own output is parsed for inline control tags that mutate game state, and inference itself is steered by a custom LMStudio client/server that does model affinity, federation, speculative decoding, and ephemeral tool servers — all running on local hardware. This section is the deep dive. Everything below is grounded in real modules you can open and read.

Why read this? It’s a working reference implementation of agent governance, structured-output steering, and observability that you can borrow wholesale. The patterns are deliberately small and composable — an interceptor is ~40 lines; a control tag is a regex plus a state write.

The shape of one reply

When a scene asks a character to respond, it doesn’t call the LLM directly. It calls an AgentGovernor (engine/mcp/comms_framework.py) which orchestrates the whole flow:

user_message
   │
   ▼
AgentGovernor.reply()
   ├─ 1. Load SceneManifest (which skills this scene exposes)
   ├─ 2. Run AUTO skills  ── cooldown + prerequisite gated ──▶ ctx["auto_results"]
   ├─ 3. pipeline.run_pre(ctx)    ◀── ~38 interceptors, priority-ordered
   │        (mutate system_prompt + messages: mood, memory, scene, rules…)
   ├─ 4. LLM call (custom LMStudio client)  ──▶ ctx["reply"], response_id, tool_calls
   ├─ 5. ContentRouter.parse_full(reply)     ──▶ ctx["parsed"]  (single pass)
   └─ 6. pipeline.run_post(ctx)   ◀── same interceptors, post phase
            (apply [STAT], sync mood, broadcast danmaku, log, shape)
   ▼
final reply (tags stripped, state mutated, telemetry emitted)

The carrier is a single mutable ResponseContext (a dict subclass). Every interceptor reads and writes well-known keys (system_prompt, messages, reply, parsed, mood_tags, abort, skip_llm…). Any interceptor can short-circuit the chain by setting ctx["abort"] = True, or skip the LLM entirely (ctx["skip_llm"] = True) to provide a canned reply. The pipeline never lets one bad interceptor crash a reply — each hook is wrapped, and failures are logged through the Oracle, not swallowed.

1. The interceptor pipeline (~38 hooks, by priority)

Interceptors subclass InterceptorBase and override pre_call(ctx) and/or post_call(ctx). They’re registered in engine/agents/interceptors/__init__.py and sorted by an integer priority (lower runs first). Each can declare applicable_scenes to limit itself to specific scenes. The registry logs its count at import time, so the live number is always visible in the logs.

The pipeline is the embodiment of the project’s design philosophy: behaviour is layered, not monolithic. Context flows in (pre, low→high priority) and gets applied on the way out (post). Pre-call tiers hydrate the prompt; post-call tiers turn the model’s words into consequences.

The full pipeline by priority (pre-call hydration → LLM → post-call application)

Pri	Interceptor	Phase	What it does
1	ContentIntensityInterceptor	pre	Injects the scene’s content profile/intensity ceiling
4	NeurochemistryInterceptor	pre	Injects derived emotional state from 6 neurotransmitters
5	NaturalMoodDriftInterceptor	pre	Applies natural stat drift, sweeps expired buffs, adds an “inner feeling” line
6	NexusPromptInterceptor	pre	Hydrates context from the Nexus knowledge base
7	ConversationRecapInterceptor	pre	Short-term conversational memory
7	CharacterMemoryInterceptor	pre	RAG character memory injection
8	CharacterRegistryInterceptor	pre	Character identity / persona injection
10	RouterMessageInjector	pre	Drains the agent-to-agent inbox (AgentRouter) into context
12	DialogDirectiveInterceptor	pre	must_include / style_lock directives
15	NarrativeModInterceptor	pre	Stage / narrative-mod context injection
15	Penthouse/Phone/Lounge/GallerySceneInterceptor	pre	Per-scene context (scene-scoped)
15	WorldStateInterceptor	pre	World time, weather, active events
16	UniversalSceneInterceptor	pre	Fallback scene context
17	AmbientEventInterceptor	pre	Random ambient micro-events
20	AutoResultInjector	pre	Injects results of AUTO-triggered skills
22	ReputationInterceptor	pre	Reputation context block
30	SkillAwarenessInterceptor	pre	Tells the model which skills/tools it may call
35	GameInterceptor	both	Game session + rules (merged in v3.1)
40	FactionContextInterceptor	pre	Faction-standing injection
45	DialogueGateInterceptor	pre	Reputation-gated dialogue options
46	RelationshipContextInterceptor	pre	Relationship metrics
50	PersonalityGuardInterceptor	both	Personality-consistency guard
55	ConversationVarietyInterceptor	both	Anti-repetition + expressiveness
60	PolicyEnforcerInterceptor	post	Enforces InteractionPolicy (length, tone, forbidden topics)
70	MemoryEnhancerInterceptor	pre	Deep RAG recall
75	HeatAwarenessInterceptor	pre	Wanted-level / “heat” awareness
80	ResponseShaperInterceptor	post	Formatting / shaping
85	TTSStyleInterceptor	post	Extracts [VOICE:...] → TTS style
88	StimulusDetectInterceptor	post	NLP stimulus detection → feeds neurochemistry
90	ActivityLoggerInterceptor	post	EventChain + training-data logging
91	StatSyncInterceptor	post	Applies [STAT:x±y] tags to character state
92	MoodSyncInterceptor	post	Syncs mood, auto-fires threshold rules
92	SpectatorBroadcastInterceptor	post	Broadcasts reply as danmaku
93	RelationshipEventInterceptor	post	Detects relationship buffs
95	GrammarScannerInterceptor	post	Flags output-quality issues

(Plus NeurochemistryInterceptor, CharacterMemoryInterceptor, WorldStateInterceptor, etc. that live in other engine subsystems and register into the same list — hence “~38”.)

Writing a new one is intentionally trivial — and you can register it from anywhere with a decorator:

from engine.agents.interceptors import register_interceptor
from engine.mcp.comms_framework import InterceptorBase, ResponseContext

@register_interceptor
class WeatherMoodInterceptor(InterceptorBase):
    name = "weather_mood"
    priority = 18          # runs after world state (15), before skills (30)
    applicable_scenes = {"neoncity", "penthouse"}

    def pre_call(self, ctx: ResponseContext) -> None:
        ctx["system_prompt"] += "\n[It is raining outside; the mood is contemplative.]"

2. Stream tags — the model steers the world

CosySim treats the LLM’s output as a control channel, not just text. Characters emit inline tags that the engine parses and applies:

Tag	Example	Applied by	Effect
[MOOD:x]	[MOOD:playful intensity=0.8]	MoodSyncInterceptor (92)	Sets mood, fires threshold rules
[STAT:x±n]	[STAT:arousal+10] [STAT:trust=70]	StatSyncInterceptor (91)	Mutates character game state
[ACTION:x]	[ACTION:pour a drink]	post-call / spectator	Drives animation / narration
[IMAGE:x]	[IMAGE:a selfie in the penthouse]	scene image pipeline	Triggers ComfyUI generation
[VOICE:x]	[VOICE:whisper]	TTSStyleInterceptor (85)	Selects TTS delivery style

There’s a single canonical parser — ContentRouter.parse_full() in engine/agents/content_router.py — that runs once per reply (step 5 above) and produces a ParsedResponse. Every downstream interceptor reads ctx["parsed"] instead of re-scanning with its own regex. For streaming, the mirror is StreamProcessor (engine/agents/stream_processor.py), which accumulates tags incrementally off the v1 SSE event stream and fires callbacks (on_mood, on_image_request, on_stat_delta) in real time — so a [MOOD:...] lights up the UI before the sentence finishes.

The keystone is StatSyncInterceptor (priority 91). Before v1.59 these tags were parsed and discarded — a character could say [STAT:trust+10] and nothing happened. Now the loop is closed: stat tags route through the CharacterStateCoordinator (only known stats, with LLM-alias normalization like desire→horniness), and because StatSync runs just before MoodSyncInterceptor (92), the freshly-updated stats are visible to the threshold-rule auto-evaluation MoodSync performs. A character’s words have mechanical consequences, and those consequences cascade into rule-driven behaviour — all in one reply.

reply: "I lean closer, heart racing. [MOOD:flirtatious] [STAT:arousal+15] [ACTION:lean in]"
   │
   ▼ ContentRouter.parse_full()  → ParsedResponse(mood=flirtatious, stat_updates=[arousal+15], actions=[lean in])
   ▼ StatSync(91): coordinator.update("aria", arousal=+15)        → state mutated
   ▼ MoodSync(92): set mood; arousal now > threshold → rule fires → directive injected next turn
   ▼ SpectatorBroadcast(92): danmaku "Aria: I lean closer…" in mood color
   ▼ TTSStyle(85)/clean text: tags stripped → "I lean closer, heart racing."

3. The custom LMStudio client/server

All inference is local, through a hand-written native-v1 client — no OpenAI-compat shim. engine/lmstudio/ is a full control plane over LMStudio:

• LMSClient (lms_client.py) — implements every endpoint of the LMStudio v1 REST API (/api/v1/chat, model load/unload/download). It exposes the steering knobs that matter: stateful chats via previous_response_id/response_id (conversation branching by reusing any historical id), structured output (JSON-schema enforcement at the logit level), full sampling control (top_k, min_p, repeat_penalty, reasoning mode, per-request context_length), image input for VLMs, and typed SSE streaming across all 19 event types.
• ServerController (server_controller.py) — CosySim is both client and server to LMStudio. The controller does server-side lifecycle: load/unload models, configure inference, per-agent model instances (create_agent_instance("aria", ...)), TTL-based auto-unload of idle instances, and per-model health (VRAM, request counts, idle time) that feeds the Oracle dashboard.
• LMLinkManager (lmlink_manager.py) — federation. Connects multiple LMStudio instances (local + remote over Tailscale) and routes each request to the best peer by model affinity, capability, load, and failover. Peers track latency (EMA), error rate, and consecutive failures; transient health blips retry with exponential backoff + jitter rather than flipping a peer unhealthy.
• TaskQueue (task_queue.py) — a priority queue with model-affinity routing: CODE tasks go to *coder* models, VISION to *vl*/*llava*, ROUTER to tiny *0.6b* models, etc. Workers auto-start on first submit().
• Ephemeral MCP tool servers — tools are offered to the model per-request via the v1 integrations field. MCP.ephemeral("http://localhost:8600/mcp/sse") references a server by URL (no pre-registration), with allowed_tools and auth headers; MCP.plugin("mcp/cosysim") references a registered one. This is how a character gains tool access for one call without standing infrastructure.
• Speculative decoding — client.enable_speculative(main_model, draft_model) loads a main+draft pair; LMStudio then activates spec decoding automatically and CosySim passes draft_model through the chat payload. Real throughput gains, fully local.

Per-agent affinity, federation, and the task queue together mean a single rig (or a small fleet) can run a tiny router model, a chat model, a coder model, and a vision model concurrently — each agent steered onto the right one.

4. The Oracle — one name, two entities

The Oracle is deliberately dual, and that duality is the project’s signature flourish.

The telemetry backbone (engine/observability/oracle.py) is the project-wide observability facade. One import — from engine.observability.oracle import get_logger — and on first use it wires the entire stack: a StructuredLogger root handler (→ SQLite + JSONL, queryable and traceable), the CosyLogger ring buffer (→ the in-game Phone feed), and an _OracleHandler that fires only on ERROR+ (~0.2ms cost). Errors flow into the ErrorAggregator, which fingerprints them — stripping IDs, numbers, and paths to a stable hash — so 500 log lines collapse into “LMStudio auth failed: 47× in 5min, affecting phone + lounge + tavern, started 14:32.” It’s hardened: a bounded-LRU flood guard caps memory under a storm of unique fingerprints, a throttled rate-alert hook emits one CRITICAL line instead of silence, and a post-install self-check confirms the handlers actually attached (a silent no-op install is exactly the failure mode it guards against). diagnose() and scripts/oracle.py print health, top errors, LLM p95, Nexus KB stats, per-model VRAM, and Gemini service status in one ASCII-safe report.

The in-game scene (content/scenes/oracle/oracle_scene.py) is a neural-consciousness terminal in NeonCity’s core — meditation, LLM-driven fortune readings, city-pulse displays — and it surfaces the very same telemetry through an “All-Seeing Eye” dashboard: a real-time error feed, a service-health grid, and trace links, all over Socket.IO. The thing watching the city is the same thing watching the code. That’s not a gimmick — it means the project’s observability has a face, and debugging is a first-class, in-world experience.

5. Neurochemistry + mood drift

Underneath the mood tags is a genuine affect model. engine/characters/neurochemistry.py gives every character 6 neurotransmitters — dopamine, serotonin, oxytocin, cortisol, adrenaline, endorphins — each with a baseline, a half-life decay curve, and a stimulus catalog (kiss, rejection, crew_victory, level_up…) that applies clamped deltas. Emotions are computed, not hardcoded: high dopamine + low cortisol → Confident; high cortisol + high adrenaline → Panicked. The NeurochemistryInterceptor (priority 4) injects this derived state into the system prompt at the very front of the pipeline, and StimulusDetectInterceptor (88) closes the loop by detecting stimuli in the conversation post-call and feeding them back.

NaturalMoodDriftInterceptor (priority 5) makes the world feel alive between turns: arousal cools, tiredness accumulates, anger fades, happiness regresses toward a personality mean — deliberately slow, so emotions shift gradually rather than snapping. It piggybacks buff-expiry and tag-decay sweeps onto every call and slips the agent a one-line “inner feeling” cue. So a character isn’t a static persona answering questions — it’s a drifting emotional state that your words (and [STAT:]/[MOOD:] tags, and the threshold rules they trigger) continuously nudge.

---

NLM + Nexus — frontier-grade AI from local models

The Oracle’s All-Seeing Eye surfaces query-router provenance — which tier answered each query, with confidence and tokens-saved logged in Oracle format.

Local models are cheap, private, and fast — but a 0.6B–8B model running in LMStudio is not GPT-class on its own. CosySim closes that gap not by making the model bigger, but by making the model ask less and remember more. Two subsystems do the heavy lifting:

• Nexus KMS — a persistent SQLite + FTS5 + vector knowledge backbone (:8700) that every agent, scene, and dev session reads from and writes back to.
• NotebookLM (NLM) — Google’s Gemini, driven headlessly through a reverse-engineered private RPC stack, used as a free distillation and grounding layer.

The thesis is simple and provable in the code: the first time a question is asked it costs compute; every subsequent time it is served from Nexus for free. Expensive frontier-grade reasoning happens once, gets distilled into the knowledge base, and is thereafter answered locally — instantly. The local model becomes the last resort, not the first.

This is the part of CosySim most worth borrowing. The whole pipeline is open and grounded in real modules — read along.

The 7-tier query router

engine/nexus/query_router.py (NexusQueryRouter) is the heart of the system. Every information-retrieval request — agent context hydration, a player question, a dev lookup — passes through a confidence-gated cascade, cheapest tier first. Each tier either clears the min_confidence bar and returns, or falls through to the next.

#	Tier	Mechanism	Cost	Confidence
0	Local session cache	In-process MD5-keyed dict, TTL local_cache_ttl (300s)	~0	inherited
1	Q&A cache	client.find_qa exact/fuzzy match, scored by word-overlap relevance (≥0.4 to count)	~0, instant	up to 0.90
2	Vector search	Gemini Embedding 2 → ChromaDB cosine over knowledge/qa/code/news	fast	up to 0.92
2.5	File Search	Google managed RAG with grounded citations over uploaded docs	API call	0.85
3	FTS knowledge	SQLite FTS5 across Nexus entries, title-overlap + length scored	fast	up to 0.85
4	Nexus smart-ask	Server-side hybrid pipeline (FTS + NLM) via client.ask(depth=…)	medium	variable
5	Direct NLM	nlm_unified_ask — free, Gemini-grounded answer with citations	slow	~0.8
6	LLM fallback	Local LMStudio inference (engine.lmstudio.chat)	local GPU	0.6

The thresholds are real, tuned constants and every one is config-overridable (nexus.query_router.*):

CACHE_CONFIDENCE   = 0.90   # Q&A cache hit
VECTOR_CONFIDENCE  = 0.82   # strong vector match
FILE_SEARCH_CONFIDENCE = 0.85  # grounded in uploaded docs
SEARCH_HIGH = 0.75 / SEARCH_MEDIUM = 0.50 / SEARCH_LOW = 0.30
MIN_ANSWER_LENGTH = 20

Two details that make it robust rather than naive:

• Relevance gating, not first-result-wins. Tier 1 doesn’t trust the top Q&A row blindly — _question_relevance computes a stop-word-filtered Jaccard overlap and scales confidence by it (0.4 overlap → 0.72 conf, 1.0 → 0.90). A weak match falls through instead of returning a confidently-wrong answer.
• Provenance logging. Every resolution logs tier=…, confidence=…, tokens_saved=… in Oracle format, and per-agent hit counts are tracked (agent_queries / agent_hits) — so you can see exactly which tier answered, for whom, and how much GPU it saved.

The self-improving flywheel

This is what makes local models punch above their weight. Look at tiers 3–6 in query(): every answer that required real work is written back as a Nexus Q&A pair, which promotes it to tier 1 for all future queries.

# Tier 6: LLM Fallback — store the answer back in Nexus for future reuse
if use_llm:
    result = self._llm_fallback(question, ...)
    if result.answer and len(result.answer) >= self.MIN_ANSWER_LENGTH:
        self._store_qa(client, question, result.answer, ...)   # → promotes to tier 1
        self._stats.answers_stored += 1

And _store_qa doesn’t just cache — it also feeds the training flywheel (_feed_training_flywheel → collect_from_qa), so every fallback simultaneously becomes a future cache hit and a fine-tuning example. The loop is closed:

expensive answer (NLM / LLM)
        │  store_qa
        ▼
Nexus Q&A pair  ──────────►  future query hits tier 1 (free, instant)
        │  collect_from_qa
        ▼
TrainingFlywheel example  ─►  fine-tune local model
        │
        ▼
better local fallback  ────►  cheaper tier 6, more cache hits next cycle

RouterStats.hit_rate() measures the payoff directly: hits ÷ total queries. As the cache fills, the rate climbs and llm_fallbacks falls. The nlm_router.py variant adds an explicit savings_report() breaking out answered_without_gpu = cache_hits + fts_hits + nlm_hits and estimated_tokens_saved — the system reports its own compounding ROI.

NLM chain-prompting: where frontier reasoning enters

NLM is the system’s gateway to Gemini — for free, at NotebookLM rate limits. engine/nexus/nlm_chain.py (NLMChainEngine) turns a single question into multi-step distillation and routes the results straight back into Nexus.

Chains are declarative (defined in config/nlm_notebooks.yaml), each step’s output piped into the next via a {previous_output} template variable:

engine = NLMChainEngine()

# progressive research: overview → details → examples → gaps
engine.execute_chain("architecture-review", notebook_id,
                     variables={"task_description": "..."})

# reverse-generate a whole Q&A set from one notebook
engine.distill_notebook("coding", questions=[...])

# weekly fleet sweep across all notebooks
engine.run_batch("weekly-review")

Crucially, execute_chain persists as it goes: the final synthesis is stored as a Nexus entry, and every substantive step is stored as a Q&A pair (_store_qa_in_nexus). So a single chain run — one burst of Gemini-grade reasoning — seeds dozens of tier-1 cache entries that the local stack serves forever after. generate_action_manifest even uses the task_decompose chain to turn a fuzzy task description into a JSON, agent-executable plan.

Behind it, nlm_direct_client.py (NLMDirectClient) speaks the raw batchexecute / GenerateFreeFormStreamed RPC protocol with browser-attached auth (SAPISIDHASH), a 302-operation rpcid registry, and full multimodality — text, URL, YouTube, image, audio, video, PDF in; reports, podcasts, mind-maps, flashcards out. Every output can become the next call’s input — recursive self-improvement is the architecture, not an afterthought.

The cache pipeline — Gemini as both generator and evaluator

engine/nexus/cache_pipeline.py runs a 10-stage (A–J) cycle that mass-produces evaluated cache entries:

A — seed high-quality session turns → Nexus (no NLM)
B — upload source pyramid + history chunks → NLM Notebook A
C — raw generation: flashcards + quiz + data tables
D — structured generation: CSV mode + code-gen mode
E — parse + dedup candidates
F — NLM self-evaluation: ESSENTIAL / USEFUL / SKIP   ← Gemini grades its own output
G — store approved pairs in Nexus Q&A cache
H — Excel review sheet for human-in-the-loop
I — upload approved pairs back as a source (compounding next cycle)
J — gap analysis → scheduler tasks

Stage F uses Gemini to filter Gemini — only ESSENTIAL/USEFUL pairs survive. Stage I feeds approved knowledge back in as a source, so each cycle compounds on the last.

The knowledge pipeline: one funnel, consistent quality

Every knowledge source — sessions, URL crawls, agent submissions, NLM distillation, manual notes — routes through a single funnel, engine/nexus/knowledge_pipeline.py (KnowledgePipeline.ingest):

ingest → validate → dedup → store → embed → Q&A → notify → train

Each stage is deliberate: content-hash dedup (SHA-256 of title + first 500 chars) blocks near-duplicates; a quality heuristic gates Q&A generation (quality ≥ 0.5); successful entries auto-embed into ChromaDB and auto-generate rule-based Q&A pairs; and everything feeds the DataCollector as a knowledge_synthesizer training example. The result: anything that enters Nexus is immediately discoverable by all retrieval tiers — FTS, vector, and Q&A cache — with no manual bookkeeping.

Why this punches above local weight

• Frontier reasoning is amortized to zero. Gemini-grade answers (via NLM) are computed once and distilled into a free, instant local cache. The marginal cost of the 1000th identical query is a dict lookup.
• Confidence gating prevents quality collapse. Cheap tiers only answer when they’re actually confident; otherwise the question escalates toward grounded Gemini. You get cache speed without cache staleness lies.
• Grounded citations on demand. Tiers 2.5 and 5 return answers with source citations (File Search + NLM), so even “frontier” answers are verifiable, not hallucinated.
• The system trains the system. Every fallback is both a cache write and a fine-tuning datum — the local model that handles tier 6 next month was taught by the Gemini that handled tier 5 this month.
• It’s all observable. router.stats, savings_report(), and Oracle provenance logs make the flywheel measurable — you can watch the hit rate climb and the GPU calls fall.

from engine.nexus.query_router import get_query_router

router = get_query_router()
res = router.query("How does the interceptor pipeline work?")
print(res.source, res.confidence, res.tokens_saved)   # e.g. "cache" 0.90 450
print(router.stats.to_dict())   # cache/vector/file_search/nlm/llm breakdown + hit rate

The whole stack is open, local-first, and self-documenting — a working example of how to give a small local model a memory that compounds and a tutor that’s free.

[KnackAU]

CONTROL — How CosySim Trains and Governs Itself

The Oracle dashboard surfaces scheduler health, auto-loop cycles, and per-task timeout/error counts in real time.

Most AI demos are read-only: a model answers, you move on. CosySim’s CONTROL plane is the opposite. Every conversation, tool call, routing decision, and code edit becomes a training signal. A scheduler daemon wakes up on a cron-like cadence, checks whether enough new signal has accumulated, fine-tunes small local models on it, benchmarks the result against the incumbent, and promotes the winner — all on your own GPU, with no human in the loop and no data leaving the machine.

This is the part of the project most worth borrowing. It’s a working, end-to-end example of a local self-improvement loop: a data flywheel, a fine-tune orchestrator, an evaluation gate, an autonomous cycle controller, and an agent governor — wired together through a single scheduler.

The flywheel in one sentence: more interactions → richer datasets → better local models → better runtime behaviour → more interactions. See docs/TRAINING.md for the full pipeline walkthrough.

---

The five moving parts

Layer	Module	Role
Flywheel	training/data_collector.py, engine/nexus/training_flywheel.py	Capture every runtime event as a typed training example
Zoo	training/model_zoo.py	Single source of truth: 16 ModelSpec entries, each with its own dataset key, train threshold, and base model
Trainer	training/finetune_orchestrator.py, training/auto_train.py	QLoRA / Unsloth fine-tune jobs with queue, progress, checkpoint, auto-merge
Gate	training/evaluation_gate.py, training/model_registry.py	Benchmark before/after; promote only if quality holds or improves
Controller	engine/nexus/auto_loop.py, engine/nexus/scheduler_daemon.py	Closed-loop orchestration on a schedule; the AgentGovernor caps live agents

---

1. The DataCollector flywheel — learning from your own interactions

DataCollector (training/data_collector.py) is a thread-safe, non-blocking JSONL appender that runtime components call as they work. It writes per-type live files to training/datasets/collected/{model_type}_live.jsonl. Every typed signal has a dedicated capture method:

collector.collect_tool_call(user_input, tool_name, params, success=True)  # → tool_dispatch
collector.collect_grammar_error(bad_text, fixed_text, error_type="json")  # → grammar_scanner
collector.collect_output_rating(output, rating=4, source="feed")          # → output_evaluator
collector.collect_conversation(system_prompt, history, response, rating)  # → conversational
collector.collect_code(prompt, code, language="python")                   # → coder
collector.collect_agent_decision(...) / collect_agent_outcome(...)        # self-improvement loop

Failures here never crash the caller — each method is wrapped and logged through the Oracle, so the act of collecting training data can’t break the act of serving the user.

In parallel, TrainingFlywheel (engine/nexus/training_flywheel.py) harvests higher-level signal from the knowledge system — collect_from_qa, collect_from_nlm, collect_from_routing, collect_preference — into a SQLite-backed store with content-hash dedup, then exports in Alpaca, ShareGPT, or DPO formats (export_jsonl, export_sharegpt, export_dpo). The training-sync scheduler task drains Nexus Q&A into this store daily and auto-exports once 50+ unexported, quality-filtered examples accumulate.

2. The Model Zoo — one registry, many tiny specialists

MODEL_ZOO (training/model_zoo.py) is the declarative heart of the system: 16 ModelSpec entries, each declaring everything needed to train and evaluate one small specialist model.

"router_v3": ModelSpec(
    id="router_v3",
    base_model_alias="qwen-270m",        # Qwen2.5-0.5B-Instruct
    task_type="classification",
    dataset_key="router_v3",
    train_threshold=500,                  # auto-train fires at 500 collected examples
    collect_from=["agent_routing_events", "intent_labels"],
    auto_promote=True,
    priority=2,
)

The fleet spans evaluators (qa_evaluator, output_evaluator), classifiers (router_v2/v3, conversation_analyzer), structured-output models (tool_dispatch), detectors (grammar_scanner), and generators (syntax_fixer, knowledge_synthesizer, coder, conversational) — plus voice backends. The philosophy: don’t fine-tune one big model; train a swarm of cheap 270M–3B specialists that each do one job well and run locally in LMStudio. Base models are resolved through aliases (qwen-270m → Qwen/Qwen2.5-0.5B-Instruct, llama-3b → meta-llama/Llama-3.2-3B-Instruct).

3. The FinetuneOrchestrator — QLoRA jobs as first-class objects

FinetuneOrchestrator (training/finetune_orchestrator.py) manages the full job lifecycle as persisted FinetuneJob records (training/jobs.jsonl): PENDING → RUNNING → DONE/FAILED/CANCELLED, with live progress, step/loss parsing, best-loss tracking, and auto-merge of the LoRA adapter on success.

Rather than depend on a heavyweight training harness in-process, it generates a standalone, cross-platform Unsloth training script per job and runs it as a subprocess (configurable via COSYSIM_TRAIN_PYTHON or training.python_executable, honouring the project’s venv rule). Hyperparameters scale with model size via FinetuneConfig — a 270M model gets lora_r=8, batch_size=8; a 3B model gets lora_r=32, batch_size=2, seq_len=2048. On completion it notifies the ModelRegistry.

Router v3 retrain — the canonical full cycle

RouterFinetuneCycle (engine/nexus/router_finetune_cycle.py) is the cleanest worked example of an end-to-end retrain:

1. Load training/datasets/router_v3.jsonl (16 router categories)
2. Split 90/10 train/val with a fixed seed, converting each example to Alpaca format
3. Submit + run a fine-tune job through the orchestrator
4. Register the resulting adapter

Trigger it directly with python training/run_router_v3.py, or let the weekly router-finetune-cycle scheduler task (dataset → train → benchmark → promote) run it autonomously.

4. The evaluation gate — no degraded model ever gets promoted

A self-improving system that can’t tell better from worse will happily train itself into the ground. evaluation_gate.py is the safety valve. It benchmarks the candidate against the incumbent and applies an explicit GatePolicy:

Policy	Rule
NO_REGRESSION	candidate must score ≥ threshold × baseline
MUST_IMPROVE	a named metric must increase
PARETO_DOMINANT	candidate may not be dominated on any metric
CUSTOM	caller-supplied evaluation function

Per-type benchmark prompt suites (router, tag-extraction, response-validate, general) score accuracy, latency, and consistency over multiple runs. Only models that clear the gate reach ModelRegistry, which supports single-score auto_promote and multi-criteria Pareto promotion — and that registry is what LMStudio loads as the active model.

5. The AutoLoop — closing the loop without a human

AutoLoop (engine/nexus/auto_loop.py) is the controller that turns the parts above into an autonomous cycle. It registers five scheduler callbacks and records every run in a SQLite cycle ledger (data/auto_loop.db):

Cycle	Cadence	What it does
Experiment execution	every_2h	Runs the oldest PENDING experiment; one per cycle to keep load predictable
Eval sweep	every_30m	OnlineEvaluator.auto_check() — promote/rollback models past their thresholds
Training check	every_4h	check_and_train_all_zoo() — fine-tune any zoo model past its train_threshold
Impact assessment	every_6h	Finalize before/after impact snapshots, compute deltas
Full daily cycle	daily	All four in sequence → a Markdown Daily Improvement Report stored in Nexus

Each promotion, rollback, and training run is logged to the ImpactTracker, so the system keeps an auditable trail of what it changed about itself and what happened next. get_loop_status() exposes a health label (healthy / degraded / stalled) for the Oracle dashboard.

6. The scheduler — 90+ tasks, now with per-task timeouts

scheduler_daemon.py is a lightweight, cron-like daemon (not the agent task scheduler) that drives all of the above plus dozens of maintenance, knowledge, and content tasks — Nexus health, dedup, QA generation, news distillation, world-sim ticks, governance audits, model benchmarks, and the training tasks already described.

The v1.60.0 hardening pass is itself a good example of the project’s “fix the real problem” ethos. The original symptom: a hung external news fetch could block the entire scheduler loop for tens of seconds. The fix was structural, not a patch:

• Per-task hard timeouts — every callback runs in a worker thread joined with a timeout; a hung task is abandoned (its daemon thread is detached, never blocking the loop) and recorded with a timeout_count. Default is configurable via scheduler.default_timeout_seconds; network-bound tasks like news-fetch get tighter caps.
• Honest “not implemented” stubs — register_stub() / make_not_implemented() log one clear warning and return a sentinel that status records as not_implemented, instead of silently faking success and hiding missing functionality.
• Non-blocking Nexus logging — task results are posted to Nexus on a fire-and-forget daemon thread that gives up immediately if Nexus is unreachable, so a down knowledge service can’t stall the loop it’s supposed to observe.

python -m engine.nexus.scheduler_daemon status      # full task grid: next-due, run/error/timeout counts
python -m engine.nexus.scheduler_daemon run <id>    # run one task now
python -m training.auto_train --status              # candidate counts vs thresholds
python -m training.auto_train --dry-run             # see what would train, train nothing

---

Governing the live agents — budgets, cooldowns, prerequisites

Self-improvement also means keeping the runtime agents in line. Every character reply flows through the AgentGovernor (engine/mcp/comms_framework.py), which wraps a CharacterAgent and enforces the full governance pipeline: build a ResponseContext, run auto-skills, run the 36-interceptor pre-call chain, call the LLM, parse tags, run the post-call chain.

Two governance mechanisms matter most for control:

• InteractionPolicy caps each agent per scene — max_reply_tokens, tool_call_limit (rounds of tool calls per reply), tone/topic constraints, and in-character enforcement. Unset fields impose no constraint, so policies are additive.
• Cooldowns + prerequisites (v1.59.0): the auto-skill path previously bypassed the registry’s throttling, so an auto skill could fire every single turn regardless of its declared cooldown. The governor now consults COOLDOWN_TRACKER.can_use() and checks that each skill’s prerequisites were actually used before invoking it — and marks usage only after a successful call.

The result is a system where the agents are budgeted and rate-limited turn by turn, the scheduler is timeout-bounded task by task, and the models themselves are gated promotion by promotion — three layers of control over a system designed to keep changing itself.

Deeper dives: docs/TRAINING.md (flywheel + fine-tuning), docs/MCP_FRAMEWORK.md (governor + interceptor pipeline), docs/OPERATIONS.md (running the daemons), docs/NEXUS.md (knowledge flywheel inputs).

---

Integrations, Apps & CLI

NEONOS — the CosySim system surface where engine integrations, apps, and CLI converge

CosySim runs on local inference, but it does not run in a vacuum. The same engine that powers 35 scenes also exposes a deep integration layer (engine/integrations/), a fleet of standalone apps (apps/*.py), and a single unified CLI (cli.py). Everything reuses the same engine singletons, the same account pool, and the same secure config — so a HAR you capture in the browser, a Colab GPU you rent for free, and a NotebookLM notebook you distill all become first-class inputs to your local agents.

This is the part of the project most worth borrowing from: it is a worked example of how to wire cloud frontier models and local models into one coherent system without leaking a single secret into the repo.

Deep dives live in docs/INTEGRATIONS_SDK.md and docs/APPS.md. Per-service protocol specs are in the *_API_REFERENCE.md files.

---

The Integration Suite (engine/integrations/)

Each integration is a typed Python client that authenticates with session cookies from a shared account pool (or an env-supplied API key) and speaks the service’s real wire protocol — batchexecute, gRPC-web, or REST — reverse-engineered from HAR captures and V8 heap snapshots with ARGUS. No vendor SDK lock-in, no browser automation in the hot path.

Domain	Module(s)	What it enables
GitHub Copilot	github_copilot_client.py	Chat + model listing against the Copilot Individual API (38 frontier models — Claude, GPT, Gemini) via a GitHub browser session → short-lived Bearer token. Powers cli.py ask and the proxy.
NotebookLM	nlm_direct_client.py, notebooklm_sdk.py, nlm_rpc_registry.py	Multi-turn grounded notebook chat, source ingest (text/URL/YouTube/image/audio/video/PDF), audio overviews, flashcards, mind maps, export-to-Sheets. The SDK wraps 37 rpcids + 24 gRPC methods with full docstrings — built for agents.
Gemini (consumer + Labs)	gemini_direct_client.py, gemini_extended_client.py, aistudio_client.py, appcatalyst_client.py, opal_client.py	Direct Gemini chat (batchexecute), AI Studio MakerSuite (136 methods, structured JSON output), AppCatalyst REST access to Gemini 3 Flash Preview, and Opal creative workspace.
Managed RAG & caching	file_search_client.py, context_cache_client.py	Google AI File Search — persistent doc/code stores with grounded citations, distilled back to local Nexus (“Google is the teacher, NEXUS is the student”). Context Cache reuses 50K±token prefixes (CLAUDE.md + context) across calls.
Workspace	google_drive_client.py, gsheets_client.py, google_docs_client.py, appscript_client.py, gas_client.py, workspace_gemini_client.py	Drive upload/download/permissions, Sheets v4 CRUD, Docs create/export + Gemini content gen, Apps Script project/code/execution control, and the Gemini features embedded inside Workspace apps.
Colab (free GPU)	colab_client.py, colab_gpu_manager.py, colab_venv_manager.py, colab_notebook_builder.py, colab_tunnel_server.py	Drive a Colab runtime as a remote compute backend: AI Agent tasks, kernel exec over WebSocket, venv/notebook provisioning, and an ngrok tunnel server exposing the GPU as an inference endpoint.
Compute routing	compute_router.py	Unifies Colab tunnels, the Colab AI agent, and LMStudio behind one inference interface — tracks per-account quotas and tiers, falls back gracefully.
Account & auth plumbing	google_account_pool.py, github_account_importer.py, har_parser.py, har_extractor.py, rpcid_updater.py, rpc_proxy.py	Round-robin multi-account cookie pool, HAR → pool import, and a live rpcid updater so rotated Google RPC IDs self-heal from the YAML registry.
Other	google_aim_client.py, homeassistant.py, anythingllm.py, artifact_bus.py	Google AI Mode (udm=50) search threads, Home Assistant control, AnythingLLM bridge, and a cross-service artifact bus.

Secure by construction

Secrets never touch the repo. Clients read keys from os.environ (e.g. appcatalyst_client.py resolves APPCATALYST_API_KEY / GOOGLE_API_KEY, aistudio_client.py loads a rotating key list from GOOGLE_AISTUDIO_KEYS) and cookies from a gitignored pool. The repo ships only structure:

.env.example          # committed — shows the shape, no real values
.env / .env.local     # gitignored
config/secrets.yaml   # gitignored; *.example.* committed
data/accounts/pool.json, data/credentials/, **/client_secret*.json  # gitignored

# .gitignore — v1.61.0: "never commit real values"
.env*
config/secrets.yaml
data/credentials/
**/*credentials*.json
**/client_secret*.json

See docs/CONFIGURATION.md for the full secret layout.

---

Standalone Apps (apps/*.py)

Every major subsystem has a thin, self-contained CLI entry point. They share apps/_bootstrap.py, which auto-re-execs into .venv/Scripts/python.exe (no manual activation), puts the project root on sys.path, and sets the CWD — then forwards to the engine. The apps are facades: the real logic lives in engine/, so an app and its in-process callers always behave identically.

App	Purpose
apps/nexus.py	Nexus KMS — search, ask, add knowledge, sessions, NLM (docs/NEXUS.md)
apps/argus.py	Web-app recon — HAR/heap mining, bundle decompile, CDP scripting (docs/ARGUS.md)
apps/lmstudio.py	Local LLM status, model list, quick inference, benchmark
apps/oracle.py	System diagnostics — health, error aggregation, traces, perf
apps/ask.py	Unified query router → Copilot (38 models) / NotebookLM / LMStudio
apps/filestore.py	Gemini File Search managed RAG — store CRUD, upload, query
apps/training.py	Dataset + fine-tuning pipeline, benchmarks, live-traffic curation
apps/cdp.py, apps/har.py, apps/heap.py	Chrome DevTools, HAR, and V8 heap toolkits
apps/account.py, apps/launch.py, apps/cleanup.py, apps/test.py	Account pool, scene launcher, disk cleanup, smart test runner

Multi-protocol AI gateway

Two proxy servers turn the whole stack into an OpenAI/Anthropic/Gemini-compatible endpoint — point any existing tool at it and get frontier models:

• apps/multi_proxy.py → scripts/model_proxy_direct.py on :5801 — zero-conversion: each protocol serializes straight to/from the Copilot backend with no intermediate format (≈7× faster). OpenAI, Anthropic, and Gemini request shapes are all served natively, including tool-call parsing.
• apps/proxy.py → on :5800 — the original normalized gateway.

python apps/multi_proxy.py --default opus --list-models   # serve all 3 protocols on :5801

---

The Unified CLI (cli.py)

cli.py is the front door — 16 commands in four groups, each routing to a script, module, or app via the venv. Run it from anywhere; it handles the environment for you.

  AI & Models:   ask  nlm  nexus  filestore  proxy
  Analysis:      argus  har  heap  cdp
  Operations:    oracle  test  scene  launch  cleanup
  Accounts:      account

python cli.py ask "Explain the interceptor pipeline"     # → Copilot / NLM / local
python cli.py nexus search "economy ticks"               # local knowledge base
python cli.py filestore bootstrap-all                     # Gemini managed RAG over the codebase
python cli.py account import github.har                   # HAR cookies → account pool
python cli.py argus har capture.har --report             # deep API recon
python cli.py oracle --errors                             # what's broken, ranked

How a command reaches the engine

cli.py account import → cmd_account() parses subcommands, then calls into engine.integrations.har_parser (Google services) or re-execs engine.integrations.github_account_importer (GitHub). The CLI owns argument shape; the engine owns behavior. Same for nlm, nexus, argus, and the rest — the CLI never reimplements logic, it dispatches into the shared engine modules also used by the scenes and the in-process MCP pipeline.

The throughline across all three layers: one engine, many faces. A cookie captured by cli.py account, a notebook seeded by apps/nexus.py, and a GPU tunnel opened by compute_router are equally available to a Flask scene, a skill, or your own script — which is exactly what makes this a useful reference implementation for agentic, local-first systems.

---

[KnackAU]

ARGUS — the reconnaissance protocol

ARGUS — Automated Reconnaissance & General-purpose Universal Surveyor. A first-class, target-agnostic toolkit for mapping any web application’s API surface, auth, feature flags, real-time protocols, and AI-agent internals — and feeding what it learns straight back into CosySim’s live systems.

CosySim is local-first, but it doesn’t live in a vacuum. It talks to a lot of undocumented web APIs — Google’s batchexecute endpoints behind NotebookLM, Gemini, and AI Studio; startup WebSocket protocols; AI-agent platforms. ARGUS is the muscle that reverse-engineers those surfaces. It lives in scripts/argus/ and is technique-driven, not target-driven: every tool works on any target. Think of it less as a scanner and more as a reusable recon technique library you can lift wholesale into your own projects.

The operating philosophy, straight from scripts/argus/README.md:

Knowledge is the prize. We don’t exploit — we learn. Capture everything, decode offline, never modify live state until the surface is fully mapped.

What’s in the box

ARGUS is layered: a generic core toolkit, a CLI, an MCP server so local agents can drive it, and a set of specialized analyzers/decoders/discovery modules.

Layer	Module	Role
Core toolkit	scripts/argus/toolkit.py	16 application-agnostic functions — bundle decompile, heap mine, CDP eval, flag injection, token refresh, agent extraction
CLI	scripts/argus/analyze.py	har / heap / compare / heap-diff / dir / deep subcommands
HAR engine	analyzers/har_analyzer.py	Endpoint + auth + protocol + token extraction from HTTP archives
Heap engine	analyzers/heap_analyzer.py	V8 heap snapshot parser + string classifier
Decoders	decoders/	batchexecute, grpc_web, heap_diffing
Discovery	discovery/	rpcid_detector, feature_flag_probe, proto_reconstructor, endpoint_registry
CDP	cdp_bridge.py, network_monitor.py	Async Chrome DevTools Protocol client + live traffic capture
MCP server	argus_mcp_server.py	Exposes browser/recon tools so LMStudio agents drive Chrome autonomously
Feedback	nexus_sink.py, rpcid_mapper.py	Persist every discovery into Nexus KMS + the live RPC registry

The four core techniques

1. HAR analysis — the richest single source

One captured browsing session yields every request/response pair, headers, cookies, timing, and bodies. HARAnalyzer auto-detects the protocol (REST, GraphQL, gRPC-web, batchexecute, WebSocket upgrades) and groups endpoints by service, decoding JWTs and pattern-matching API keys along the way.

python -m scripts.argus.analyze har capture.har --report   # → Markdown intel report
python -m scripts.argus.analyze compare loggedout.har admin.har   # diff roles to find gated endpoints

2. Heap snapshot mining — what the network never shows

V8 heap snapshots contain every string the JS runtime has interned — compiled-in config, unused API routes, internal gRPC service names, RPC IDs, and secrets that never transit the wire. Two engines run over them: a regex scanner with 100+ patterns (mine_heap) and a full V8 graph walker (mine_heap_deep) that reconstructs objects and script sources.

python -m scripts.argus.analyze heap snapshot.heapsnapshot
python -m scripts.argus.analyze heap-diff before.heap after.heap   # isolate strings a single action introduced

The classifier buckets strings into URLs, API endpoints, method names, service paths, RPC IDs, and credential-shaped tokens — covering JWTs, K8s *.svc.cluster.local addresses, STUN/TURN servers, Statsig caches, protobuf definitions, and leaked model reasoning.

3. Bundle decompilation — the complete feature map

A minified SPA bundle holds the entire app logic. decompile_bundle() extracts feature-gate enums, API route strings, environment variables (VITE_* / NEXT_PUBLIC_* / REACT_APP_*), CI/CD paths, and monitoring DSNs. In one documented run, the bundle revealed 17× more URL paths than live traffic — most endpoints gate features the current user can’t reach.

4. CDP scripting — drive the live browser

cdp_bridge.py is a full async Chrome DevTools Protocol client (Chrome on --remote-debugging-port=9223). It enables programmatic JS execution, localStorage feature-flag injection, network capture, and WebSocket frame interception — the only way to map real-time protocols, since HAR captures only the HTTP upgrade, not the frames.

from scripts.argus.toolkit import cdp_eval, inject_statsig_gates
cdp_eval("document.title", cdp_port=9223)
inject_statsig_gates("https://app.example.com", {"some_gate": True})

ARGUS is explicit about the distinction that matters most: client-only vs server-enforced. Flipping a Statsig gate in localStorage reveals UI, but if the endpoint checks the flag server-side, every call still 403s. Every finding is tagged accordingly — see the security-assessment checklist in the methodology guide.

The closing loop — intelligence feeds the system

This is what makes ARGUS part of CosySim rather than a bolt-on scanner. Discoveries don’t sit in a report — they flow back into the running framework:

HAR / heap / bundle
        │  decode offline
        ▼
  rpcid_detector ── compares live traffic against the known baseline
        │  new rpcid?
        ├──────────────► ArgusNexusSink  → Nexus KMS (category="argus")
        │                  store_new_rpcid() + add_qa() → agents query via nexus_search
        │
        └──────────────► RpcidUpdater (engine/integrations/rpcid_updater.py)
                           writes config/nlm_rpcids.yaml + data/nlm_rpc_registry.json
                           → live NLM/Gemini ops pick up new rpcids at call time

When Google rotates an NLM/Gemini frontend build and rpcids change, a fresh capture run through ARGUS re-discovers them, RpcidUpdater patches both the YAML source-of-truth and the JSON runtime cache, and get_rpcid() resolves the new value on the next call — no code change, no redeploy. Meanwhile ArgusNexusSink files every new rpcid, endpoint, and feature flag into Nexus KMS as both a knowledge entry and a Q&A pair, so any agent can ask “what is rpcid X?” and get the answer ARGUS learned. Recon becomes institutional memory.

Agents driving recon

Because ARGUS ships an in-process MCP server (argus_mcp_server.py, FastMCP/SSE on :8010) and the CDP capabilities are also registered as MCP skills, local LMStudio agents can run reconnaissance themselves — screenshot a page and ask a vision model what it sees, navigate, click, fill, intercept. The same toolkit a human runs from the CLI is callable by an autonomous agent inside the MCP interceptor pipeline.

Proven results

The methodology is distilled from 370+ exploration sessions against two real targets — a voice-AI platform and a text-AI platform with a virtual OS. Headline numbers (full reports in data/argus/reports/):

Metric	Target A (voice)	Target B (text + virtual OS)
API methods discovered	53	20+
Feature flags mapped	27 gates, 14 configs	—
JWTs decoded	3	2
Internal IPs found	3	2 (K8s)
Sub-agents extracted	0	5
Apps / tools mapped	0	12
Chain-of-thought fragments	0	15+
Protobuf schemas reconstructed	0	1
Security findings	14	—

A sharp, reusable finding: text apps leak reasoning, voice apps don't

The voice platform yielded zero chain-of-thought fragments — the model runs server-side and only an audio stream reaches the client. The text platform leaked 15+ reasoning fragments, because full model output (including <think> blocks the UI filters but never garbage-collects) was streamed as text and lingered in heap memory. extract_chain_of_thought() and extract_agent_messages() turn that residue into a reconstructed multi-agent dispatch trace. The lesson generalizes: text-streaming AI apps are far more exposed to heap extraction than voice ones.

Try it

python -m scripts.argus.analyze har path/to/file.har --report   # any HAR → report
python -m scripts.argus.analyze heap path/to/file.heapsnapshot  # any heap snapshot
python -m scripts.argus.analyze deep path/to/captures/          # full automated pipeline

Whenever you hand CosySim a HAR file, a heap snapshot, or a web app, ARGUS is meant to run automatically — that’s the standing convention in the project. The thirteen techniques (HAR, heap, bundle, flags, CDP, WebSocket, tokens, profile CRUD, env mapping, security assessment, agent orchestration, chain-of-thought, schema extraction) are written up as step-by-step playbooks you can borrow for any target.

---

Creation Pillar — Asset Studio, Creation Kit & Media Generation

CosySim’s third pillar (alongside games and services) is creation: a set of tools that turn natural-language intent into game-ready assets and even entire scenes — all running on local hardware. Three things make it distinctive:

1. One generation entry point (AssetStudioCore.generate(asset_type, params)) routes images, portraits, voice, video, items, SVG and audio through a single, flag-gated orchestrator.
2. A vision-model feedback loop — generated images are scored by a local Qwen3-VL model, so the studio can benchmark sampler/CFG/step sweeps and keep the highest-scoring settings automatically.
3. Everything is a skill. Every generator is exposed as an @skill an agent can call, and a /api/inject_to_scene route lets an asset flow straight from generation into a live scene’s static folder with a hot-reload socket event. Agents create content and wire it in.

All inference is local: image/video/portrait via ComfyUI (:8188), voice via the TTS manager, and LLM-assisted items/SVG + the VL quality inspector via LMStudio (:1234). Nothing leaves the machine.

Asset Studio — the unified generation engine

The Asset Studio scene (content/scenes/asset_studio/) is a Flask/Socket.IO front end over engine/asset_studio/. The architectural heart is AssetStudioCore (engine/asset_studio/studio_core.py), a singleton that owns the whole lifecycle:

from engine.asset_studio import get_studio_core
core = get_studio_core()
result = core.generate("portrait", {"character_id": "aria", "mood": "happy"})
# → routes to PortraitGenerator → registers in AssetLibrary → caches to Nexus
#   → emits `asset_generated` over Socket.IO → returns {url, prompt, asset_id, ...}

generate() does five things in order: route to the right generator (lazy-loaded from _GENERATOR_MAP), register the result in the SQLite asset library, optionally cache metadata to Nexus KMS, emit an asset_generated socket event for live scenes, and return a normalized dict. Every asset type is gated by config feature flags so a deployment can disable, say, video or adult content without touching code:

Asset type	Generator	Backend	Required flag(s)
image	ImageGenerator	ComfyUI	asset_studio.comfyui_enabled
portrait	PortraitGenerator	ComfyUI + PortraitCache	asset_studio.comfyui_enabled
video	VideoGenerator	ComfyUI (Wan 2.2)	comfyui_enabled + video_enabled
voice	VoiceGenerator	TTS manager	asset_studio.tts_enabled
item	ItemGenerator	LMStudio + ComfyUI icon	asset_studio.lms_enabled
svg	SvgGenerator	LMStudio	asset_studio.lms_enabled
audio	AudioGenerator	synthesized	— (always on)

core.health() rolls up live status of ComfyUI (/system_stats), the TTS backends, LMStudio readiness, and per-type library counts — exactly the kind of monitoring hook the project’s conventions require.

The asset library (catalogue + provenance)

AssetLibrary (engine/asset_studio/asset_library.py) is a thread-safe SQLite catalogue (data/asset_library.db). Every generated asset is registered with full provenance — asset_type, scene, character_id, mood, preset_id, the exact positive/negative prompt, duration_ms, a cached flag, and JSON metadata — and indexed by type/scene/character/recency. It supports filtered+paginated list_assets(), full-text search over title/prompt, favorites, bulk delete, and stats(). Because the prompt and preset are stored, any asset is reproducible.

Prompts & presets — coherent style by default

Generators don’t take raw prompts. PromptBuilder (prompt_builder.py) composes them from a subject, a scene-context template (penthouse, lounge, tavern, casino, neoncity, arena, …), a mood modifier (14 moods from neutral to seductive), and style/negative tags from a StylePreset. Portraits additionally pull a character’s physical description from Nexus KMS (get_nexus_client().ask(...)) so a portrait actually looks like the character. PresetManager ships 8 built-in presets — dark_renaissance (the v1.58 default), cyberpunk, fantasy, noir, anime, photorealistic, pixel_art, minimal — and users can store custom presets in Nexus.

ComfyUI-backed media — workflows, not hardcoded graphs

WorkflowManager (workflow_manager.py) is the full ComfyUI client: node/model discovery via /object_info (cached 5 min), capability checks (has_node("FaceDetailer")), priority-based select_model(), and the complete queue → poll /history → download outputs lifecycle — all degrading gracefully when ComfyUI is offline.

The graphs themselves are built dynamically by workflow_builder.py, which exposes 15 professional workflows in WORKFLOW_REGISTRY (each with label, category, resolution, speed, and requires_nodes for capability gating):

• Portraits: portrait_fast, portrait_hires (auto-selected when FaceDetailer + UltralyticsDetectorProvider are present), portrait_refiner (dual-pass: base → 1.5× upscale → img2img refiner).
• Scene art: scene_background (widescreen cinematic), character_card (full-body 832×1216), message_image (8-step Lightning).
• Video (Wan 2.2 dual-model GGUF, UnetLoaderGGUF + two-stage KSamplerAdvanced): video_wan_t2v, video_wan_i2v, video_wan_landscape, video_wan_portrait_fast, video_wan_character_hq — e.g. 272×352 portrait, 105 frames @16fps (~6.5s).

LoRA stacking is handled by composable chain helpers (_build_lora_chain for SDXL, _build_video_lora_chain for Wan), and portraits push their result URL into PortraitCache so live scenes display the new art immediately.

The VL feedback loop — generation that scores itself

This is the part worth borrowing. WorkflowManager.check_image_quality() base64-encodes a generated image, sends it to a local Qwen3-VL model via LMStudio, and parses a structured verdict:

{ "score": 0-10, "issues": [], "strengths": [], "suggestion": "..." }

The TuningEngine (tuning_engine.py) builds on this to do automated parameter search. You give it a base param set and a sweep ({"cfg": [1.0, 1.5, 2.0], "steps": [8, 20]}); it generates the Cartesian product of variants in a background thread, scores each with Qwen3-VL, persists every run to a metrics DB (data/asset_studio/tuning_metrics.db), and picks the best variant by VL score (falling back to fastest on ties). It ships with 6 “proven profiles” seeded from real working ComfyUI exports (e.g. proven_portrait_fast: lcm/exponential, cfg 1.5, 20 steps, Lightning 8-step LoRA), and get_best_settings(workflow_id) returns the top-N tuned param sets from history. The result is a studio that learns which settings produce good images on your models — no human eyeballing a grid.

Agent-friendly by construction

Content creation here is built for autonomous agents as a first-class user:

• Skills, not just endpoints. asset_studio_skills.py registers generate_image, generate_portrait, generate_voice, create_game_item, generate_svg, list_assets, and studio_health as @skill-decorated functions (categories MEDIA/GAME/SYSTEM, with cooldowns and costs) — so any CharacterAgent governed by the MCP pipeline can create assets mid-conversation.
• Stream-tag integration. The engine’s [IMAGE:prompt] stream tag means an LLM can emit an image request inline in its reply and have it rendered.
• Inject straight into a scene. POST /api/inject_to_scene copies a generated asset into content/scenes/{scene}/static/img/ and emits scene_asset_updated for live reload — closing the loop from idea → asset → in the running game.
• Reproducible & monitored. Stored prompts/presets make every asset reproducible; failures are caught and surfaced through the Oracle observability format.

Creation Kit — building scenes without hand-coding

If the Asset Studio makes the contents, the Creation Kit (content/scenes/creation_kit/) makes the containers. It’s a visual, drag-and-drop scene editor backed by engine/creation/:

• Component registry (component_registry.py) — 45 components across 7 categories (layout, display, input, data, media, game, nav). Each maps to existing shared CosySim CSS/HTML patterns and carries a prop schema, a Jinja2 export template, and asset_hint metadata so portrait/image components know to pull from the Asset Studio. Reuse over reinvention: components render the same markup the hand-built scenes use.
• Layout persistence — designs save as JSON (data/layouts/), with live preview and pre-shipped rebuilds of real scenes (tavern, grid, arena, lounge, casino) plus templates (chat room, dashboard, shop, dungeon, terminal, …).
• HTML/CSS/JS export → registered scene. export_* helpers turn a layout into a working scene, and create_scene() (scene_template.py) scaffolds the directory and auto-registers it in control_plane_registry.py and config/launcher.yaml — so an exported scene is immediately launchable via python launcher.py <name> and even gets a generated test file.
• Character Wizard — a 6-stage pipeline (character_wizard.py: Archetype → Appearance → Voice → Stats → Story → Memory Seed) exposed over /api/wizard/*, producing a fully registered CharacterAgent with personality, backstory, and seeded RAG memories — ready to drop into any scene.
• Unified asset browser. /api/assets/combined merges the Asset Studio library and the creation asset_registry, so a builder picks from everything generated across the project.

The full create-a-game loop on one local box

Creation Kit          → design layout (drag 45 components) → export → auto-registered scene
Character Wizard      → 6-stage character → registered CharacterAgent w/ seeded memories
Asset Studio          → portraits / backgrounds / items / voice / video (ComfyUI + TTS + LMStudio)
TuningEngine + VL     → sweep params, Qwen3-VL scores them, keep the best
inject_to_scene       → assets land in the scene's static folder, live-reloaded
python launcher.py X  → play it

Every arrow is also callable as a skill — an agent can run the whole loop end to end.

---

Scene showcase

Every scene ships with a hand-crafted Dark Renaissance UI kit. A sample of the 12 design kits that drive the live scenes:

The Penthouse — 3D scene director (three.js r184)	NEON CITY — living-world dashboard	The Oracle — fortunes + all-seeing telemetry
The Briefing Room — mission control / intel hub	The Score — heist planning board	The Grid — underground market
SIGNAL — encrypted messenger	NEON OS — desktop shell + scene launcher	The Terminal — scene catalogue
Asset Studio — generation + VL scoring	Executive Suite — panoramic ops view	Admin Loft — CRT hack-green terminal

Plus 18 game scenes in all — Club Noir, The Colosseum, The Velvet Pit, The Rusty Anchor, The Obscura, The Shattered Throne, The Lab, The Arcade, Lab Break, Cyberspace, The Auction House, and more — each a live local-agent simulation. Run python launcher.py --list to see every target.

---

Repository map

engine/        core: lmstudio · nexus · world · agents · mcp · skills · training · observability · integrations
content/        scenes/ (35 targets) · shared/ (Neon HUD v2, design system)
apps/           standalone entry points + multi-protocol proxy + unified CLI surface
scripts/        argus/ (recon toolkit) · oracle.py · smart_test.py · browser_test.py
config/         default.yaml (+ example secret templates)
docs/           deep-dive documentation — start at docs/INDEX.md
tests/          pytest suite (plain assert, mocked external services)

Documentation

Area	Doc
Index of everything	docs/INDEX.md
Architecture	docs/ARCHITECTURE.md
MCP framework & interceptors	docs/MCP_FRAMEWORK.md
Nexus knowledge system	docs/NEXUS.md
Operations & runbook	docs/OPERATIONS.md
ARGUS methodology	docs/ARGUS_METHODOLOGY.md
Design system	docs/DESIGN_SYSTEM_V2.md
Changelog	CHANGELOG.md

A note on how this was built

Large parts of CosySim — including this README — were produced through agentic coding: fleets of AI agents reading the codebase, designing changes, implementing them across disjoint files, and verifying their own work with tests. The project is deliberately structured to be legible to both people and agents (consistent docstrings, version-stamped change logs, an observability spine in the Oracle, and a knowledge base that compounds). If you’re exploring what agent-built software can look like, this whole repository is the example.

License

See LICENSE. Built to be learned from and borrowed — take what’s useful.

[KnackAU]

Changelog

All notable changes to CosySim are documented here.

---

[1.62.1] — “LIVING CITY — LOOP COMPLETIONS” — 2026-06-15

Finished wiring the v1.62 “Living City” systems end-to-end: the Executive Suite desktop now reads from the same unified comms backbone the rest of the city writes to, faction hostility actually drives the reverse hack, and the pre-existing test debt was repaired so the suite is green again. No new subsystems — these are the last connectors between systems that already shipped.

Executive Suite — unified comms & surfaced signals

• Mail → GlobalCommsLog — the Suite’s Mail app now reads the unified engine/world/comms_log.py (GlobalCommsLog) instead of the legacy phone-only thread store, so player↔NPC and NPC↔NPC traffic share one inbox; intercepted messages (from phone hacking) surface in a dedicated Intercepts folder.
• Desktop breach alerts + Oracle taskbar button — phone_hacked / message_intercepted breach events now raise a desktop alert in the Suite, and the taskbar gains an Oracle button so the omniscience surface is reachable from the OS shell.

Phone OS — faction-driven reverse hacking

• Hostility triggers NPC→player hacking — the rare, firewall-defended NpcHackPlayerService reverse hack is now actually invoked by faction hostility (sufficiently negative standing), closing the loop with the v1.62.0 phone-defence upgrades: PH-T2 firewall upgrades raise the defence and can flip a hostile attacker’s breach to BLOCKED.

Tests — pre-existing debt repaired

• Suite green — repaired all pre-existing failing/broken tests inherited on this branch so the full pytest suite passes again.

---

[1.62.0] — “LIVING PENTHOUSE” — 2026-06-15

Five fixes that take the penthouse 3D scene from “wired but lifeless” to a room that feels inhabited: characters rest on the furniture, the Director appears and persists, bed-game actions drive consent-gated paired poses, every present character is driven by its own local agent, and idle characters move and emote on a cheap scripted layer between the slow LLM ticks. Verified end-to-end against the live scene on :5556 with LMStudio up.

Penthouse — naturalism & intimacy

• Anchor-aware placement — _locationPositions now carries a per-location anchor (lie/sit/stand) and surface height, and character_bridge.js places the model group origin (feet) on the surface instead of sinking through it; backend scene_state["locations"] Y values were synced to match.
• Director avatar renders + persists — placement was aligned with the working spawnCharacter path and _onSceneState now re-places the Director from scene_state.director_avatar, so the avatar shows on load for every client (not only after a manual “Place Avatar” click) and survives reconnects.
• Bed-game → paired poses (consent-gated) — the missing UI→pose chain was built: POST /api/bedgame/action resolves participants and emits bedgame_action with a pose_eligible flag; the frontend listener maps it to CharacterBridge.startPose → CharModels.startSexPose. Explicit poses (explicit_level ≥ threshold) are gated fail-closed behind the existing adult surface — every involved character must clear openness ≥ 60 and a truthy consent_given flag; the Director is exempt. Low-explicit actions always pose.
• Per-character agents (cap fix + nexus guard) — every character entering the scene now registers its own CharacterAgent, the hard 2-character cap was replaced with the SCENE_METADATA-driven _max_characters(), and a runtime invariant logs when agents ≠ present characters. Three characters now each take distinct agent-driven turns per tick.
• Cheap config-driven ambient behavior — new engine/agents/ambient_behavior.py adds a fast scripted ambient layer (fidget/glance/expression/reposition, mood- and presence-weighted) that reuses the existing set_animation / set_expression emit path between the slow LLM ticks, with a rare config-gated LLM line via the existing agent path. Tunable under penthouse.ambient.*; guarded to never override an active pose, a busy character, or a viewerless scene — so idle characters feel alive without spamming the model.

Executive Suite — a new neon OS desktop scene

• New executive_suite scene (:5596) — a full-screen neon-noir corporate “operating system”: a draggable window manager (z-index focus, traffic-light close, minimize/restore), a left app dock with badges, desktop icons, a bottom taskbar (start, ⌘K search, pinned apps, system tray) and a layered animated skyline behind an office bezel — ported faithfully from the ui_kits_v2/executive_suite reference into Jinja2 + CSS-var’d vanilla JS.
• 8 functional apps wired to real data — Files (live inventory + item catalog + sandboxed data/ JSON), Mail (read/compose against the existing phone comms threads; repoints to GlobalCommsLog in sub-project 2), Notes (autosaving JSON store), Music (player + animated equalizer/queue), Code, a sandboxed Terminal (whitelisted read-only commands — status/world/inbox/scenes…, no shell exec), Browser, and Settings (theme/accent).
• Live AI assistant + live system tray — the Assistant panel (“Add my Agent”, LIVE badge, chat feed) drives a real CharacterAgent (default Aria) over Socket.IO, reusing the shared VirtualAgentManager agent pipeline; the tray clock reads the live WorldState game clock and heat reads live PlayerState. Assistant replies are run through a conservative sanitizer that strips leaked model meta/instruction artifacts (<<DOC|…>> blocks, system/metadata markdown headers, stray role prefixes) before they reach the feed.
• Full registration + auto-start — added across every source-of-truth (launcher --list, TUI, control-plane registry, port registry :5596, the in-world hub) and enabled by default (pillars.game.auto_start + per-scene auto_start: true), so it boots with the core game set.
• Surfaced previously-missing hub scenes — auction and cyberspace were registered but absent from the hub UI; both now appear in-world and in --list.

Comms backbone & NPC↔NPC messaging

• GlobalCommsLog — single source of truth — new engine/world/comms_log.py backs every message (phone DMs, autotexts, NPC↔NPC) in one SQLite log (data/comms_log.db) with query/thread/about/recent accessors, mark_observed for later phone-hacking/Oracle reads, and a prune row cap. All writes are defensive — a logging failure can never break a send.
• Comms interceptor + phone write-through — new engine/agents/interceptors/comms_logger.py (CommsLoggerInterceptor, pri 90) logs every governed agent reply, while the phone scene logs explicit user sends; a per-call comms_context aligns thread_id/recipients so the two halves of a conversation share a thread without duplicates.
• NPC phones + NPC↔NPC hybrid messaging — new engine/world/npc_comms.py scheduler (with npc_dm threads + an npc_phone stub in phone_db.py) has NPCs text each other on a template-dominant / occasional-LLM mix, weighted by affinity and anti-spam. It runs on its own daemon thread and lock (the _tick_lock fix) so NPC traffic never blocks the user-facing phone ticker, and is started/stopped from the phone scene lifecycle.
• Leave-a-message — new engine/world/presence.py plus phone changes let a player message an offline NPC (and NPCs leave autotexts for an offline player): messages are stored left_unread, delivered on (re)connect, and surfaced via a “left you a message” affordance in phone_v2.js / phone_ui_v2.html.
• Calmer, more varied user inbox — phone_rules_v2.py gains a config-driven comms.autotxt.user_multiplier (cooldown scaling, per-character overrides), topic seeds drawn from real city events, style pools, and anti-repeat, so user-facing autotexts feel less spammy and less repetitive.
• Pairwise relationship effects — new engine/world/relationship_effects.py nudges each NPC pair’s character_relationships.relationship_level (0–1) by a small, tone-/keyword-derived, clamped delta per NPC↔NPC exchange, closing the loop with the affinity-weighted pair selection in the scheduler.

Phone OS overhaul — NPC phones, hacking & upgrades

• NPC phones + derived security — new engine/world/phone_os.py (the PhoneOS accessor over a full npc_phone table) gives every NPC a real phone with archetype-/faction-seeded base stats, and derives effective_security, effective_firewall, hack_power and app_slots from base levels + installed software (the single source of truth being each upgrade’s phone_upgrade.effects block). The player’s equipped cyberdeck trace_resist hardens the phone on top, coupling the inventory to phone defence. All reads are defensive — a missing/broken DB never crashes a phone read.
• Buyable / skill-installable upgrades — engine.world.inventory.ITEM_CATALOG gains flagged is_phone_upgrade software + hardware (encryption patch, firewall v1/v2, comms tap, ICE, app packs, modem/CPU/OS-kernel) sold by the Grid vendor; the new install_phone_upgrade skill gates install on owned-item → credits → skill_check (idempotent, with before/after derived stats), and list_phone_upgrades surfaces the catalog + current stats. Installing an upgrade raises the owner’s derived security/firewall/hack-power/app-slots and can light up a gated UI app (e.g. Comms Tap → Intercept/Breach).
• Player→NPC hacking — new content/scenes/phone/phone_hack.py (HackPhoneService.hack) resolves a config-driven check (player hack-power + roll vs the NPC’s security + firewall) and on success performs intercept (reads a bounded slice of the NPC’s comms and marks them observed for later Oracle/UI reads), plant (writes one spoofed message into the NPC’s comms + phone thread), or steal (a bounded, margin-scaled credit skim). Failure costs heat and standing; the method never raises.
• NPC→player hacking (firewall-defended) — a rare, hostility-driven reverse hack (NpcHackPlayerService) where the attack is defended by the player’s effective_security + effective_firewall + base_difficulty, so PH-T2 firewall upgrades directly raise the defence and can flip the same attacker from a successful (bounded, recoverable: read + at most one leak-or-plant, capped heat/standing) breach to BLOCKED. Emits phone_hacked / message_intercepted events and persists a breach notification to the comms backbone.
• OS-like phone UI — new /api/phone/* routes and phone_v2.js views: a notifications center (bell badge + breach/leak/left-message feed off the persisted comms log), home-screen search, an installed-apps model that unlocks gated apps, a Breach app (pick a target, see its defences, run intercept/plant/steal) and an Upgrades app (browse + buy/install the catalog with live before/after stats).

Oracle omniscience — the Oracle reads the comms log

The Oracle now genuinely “sees all”: it reads the GlobalCommsLog (player↔NPC + NPC↔NPC chatter) and turns overheard signal into cryptic fortunes, real intel, and rare omens — all governed by a single privacy/spoiler budget so it hints without ever dumping the city’s private conversations.

• Privacy-budgeted comms reader — new engine/world/oracle_comms.py is the one shared read layer over the comms log (select_entries / weave_into / find_signal). A HARD oracle.comms.max_refs cap bounds how many entries any single reading may cite, a recent_window bounds the scan, and respect_sensitive skips/abstracts entries flagged sensitive/mature/intimate so private moments stay private. Raw message bodies are never surfaced — only short, abstracted poetic hints — and every referenced entry is stamped mark_observed(id, 'oracle'). All comms-log failures degrade gracefully to plain atmospheric behaviour and never crash a reading.
• Comms-aware cryptic fortunes — content/scenes/oracle/oracle_scene.py gains _maybe_weave_comms: under oracle.comms.fortune_chance a fortune OCCASIONALLY weaves 1–2 real overheard exchanges into the prophecy (cryptically, capped by the budget, marking each entry observed); otherwise it stays purely atmospheric. No comms / a failed roll returns the base fortune unchanged.
• read_the_signal intel skill — new @skill(pack="oracle") in engine/skills/builtin/oracle_skills.py asks the Oracle to read the streams and surface ONE genuine, log-sourced finding — a low-security hack opportunity, an NPC discussing the player, or the closest NPC↔NPC pair by chatter volume — then grants a small bounded reward (oracle.comms.signal_reward_credits + signal_reward_item) gated by the skill cooldown/cost. Empty streams return an in-world “silence” line with no reward.
• Autonomous comms omens — engine/agents/oracle_companion.py gains a rare, config-weighted (oracle.comms.omen_weight) omen action: the companion pulls a single budget-capped entry, renders it as a cryptic line, marks it observed, and delivers it to the player’s phone — so the Oracle reaches out unprompted without ever leaking raw chatter.
• Config — new oracle.comms.* block in config/default.yaml (fortune_chance, max_refs, recent_window, omen_weight, signal_reward_credits, signal_reward_item, signal_cooldown_sec, respect_sensitive) keeps the entire privacy/spoiler budget in one place.

---

[1.61.0] — “PUBLIC RELEASE PREP” — 2026-06-13

Made the repository safe and inviting to publish: a full credential security audit (no live secrets in the tree), a hardened ignore policy, and a complete professional README + documentation pass.

Security — credentials externalized (no real secret remains in tracked source)

• 80+ secrets removed: 24 hardcoded Google API keys across engine/integrations/ (aistudio/drive/workspace) → os.getenv; 55 keys in config/nlm_rpcids.yaml → file gitignored + redacted config/nlm_rpcids.example.yaml shipped; the LMStudio API token and a real personal email pulled out of config/default.yaml and source; a live FIREBASE_API_KEY in the ARGUS sesame client; and key/email stragglers redacted from docs/ARGUS_* reports and a heap test fixture.
• Secret pattern: real values live only in a gitignored .env (auto-loaded by engine/config.py via python-dotenv) or config/secrets.yaml; committed config uses ${ENV_VAR} placeholders resolved at read time. .env.example documents every variable. Local runtime is preserved — verified config loads, all scenes import, and NEON CITY launches clean.
• Personal identifiers (account handles) moved to COSYSIM_DEFAULT_ACCOUNT / COSYSIM_KNOWN_ACCOUNTS env vars.
• .gitignore hardened: secrets, config/nlm_rpcids.yaml, data/heap_findings*, dumps, *.bak, cookies/credentials, heap variants; git rm --cached applied to the two tracked sensitive files (kept on disk).

Documentation — flagship README + assets

• Complete README rewrite (drafted by an 8-agent fleet, assembled by hand): hero, badges, “Start here” navigation table, quickstart, security/config guide, then deep sections — Overview & Architecture, NEON CITY living world, Engine Internals (interceptors · stream-tag spec decoding · custom LMStudio steering · ephemeral servers · the Oracle’s dual role), NLM + NEXUS frontier-from-local, CONTROL (training/finetune/self-improvement), Integrations/Apps/CLI, the ARGUS protocol, and the Creation pillar. Emphasizes the open, learn-from-it nature.
• docs/assets/scenes/ — 21 curated scene screenshots embedded/linked.
• README deep-links the existing docs/ tree (all links verified to resolve).

---

[1.60.0] — “LIVING SYSTEMS” — 2026-06-13

A 10-agent fleet upgrade (disjoint file ownership; shared-file hooks integrated centrally) that takes the gameplay + infra subsystems from “wired” to pro level. Builds directly on the v1.59 feedback loops.

Gameplay depth

• Faction standing now matters — new engine/world/faction_gates.py with reusable helpers scenes call: shop_access_allowed, price_multiplier_for (matching faction −10%, rival +15%, graded by band), filter_missions_by_standing, npc_standing_note. FactionAI._decide() now weights actions by the player’s standing (allies expand/defend near you, rivals raid/sabotage toward your district), and territory-control shifts are broadcast on the EventBus (NEONCITY_FACTION_SHIFT) — previously computed but never emitted.
• Mission consequences + chains — completion/failure applies faction-control, reputation, and heat deltas via the player API; new engine/world/mission_chains.py defines gated multi-mission arcs (off by default); difficulty/reward scale with player skill levels; crew availability is enforced on assignment.
• Crew skill-checks — operations resolve via a graded success/partial/failure check derived from crew level + loyalty + op difficulty (was: always full reward). Loyalty shifts on outcome; rewards scale per tier.
• Equipment & consumables — new engine/world/equipment_effects.py maps equipped cyberware/weapons to real skill/stat bonuses (get_equipment_bonuses); consumable effects generalised to work by item category instead of a 9-item hardcoded list; optional condition wear.
• Economy world-event shocks — Market.subscribe_to_world_events() wires world events → supply/demand shocks via apply_event (war→weapons, festival→luxury, shortage→surge), and territory control now flows into specialty-good pricing (refresh_territory_multipliers). Activated once at LivingWorld bootstrap. Preserves v1.59 buy/sell settlement.

Agent depth

• Neurochemistry from dialogue — new StimulusDetectInterceptor (pri 88) NLP-detects compliment/insult/kiss/touch/rejection/threat/comfort in replies and applies the matching neurochemistry stimulus to the speaker/addressee, so characters affect each other emotionally from what they actually say (was: stimuli only ever triggered manually).

Infra hardening

• Scheduler — per-task timeout enforcement (a hung task no longer blocks the loop); stub tasks log an honest “not implemented” instead of faking success.
• LMStudio federation — exponential backoff + jitter on peer health retries; __del__/close no longer spews logging noise during interpreter shutdown.
• Observability — flood-guard and error-bucket growth are now LRU-bounded; error-rate threshold alerting (throttled); structured-logging init self-check.
• Dead code resolved — TaskQueue / finetune orchestrator / auto_train either wired into a real path or removed cleanly with exports updated.

Integration & config

• All knobs exposed in config/default.yaml (faction_gates, territory.faction_ai, mission, crew.skill_check, world.equipment, economy.event_shocks, neurochemistry.stimulus_detect, scheduler, observability.error_aggregator/oracle, lmstudio.task_queue_v2, lmlink.health.backoff) with safe in-code defaults.
• Each agent shipped hermetic pytest coverage; shared-file edits (interceptor registration, config, activation calls) were applied centrally to avoid parallel-edit conflicts.

---

[1.59.0] — “CONSEQUENTIAL WORLD” — 2026-06-13

Closed the open feedback loops surfaced by a full subsystem audit. The infrastructure was strong (MCP pipeline, agent loop, LMStudio client, Nexus router) but state changes flowed downstream and were discarded instead of feeding back into the world or agent decisions. This release makes actions consequential.

Added — feedback loops

• StatSyncInterceptor (pri 91) — agents emit [STAT:arousal+10] / [STAT:trust=60] and the deltas are now applied to character game state via the state coordinator (with stat-name aliases: desire→horniness, joy→happiness, …). Runs just before MoodSync (92) so its threshold-rule auto-evaluation sees the new values. Previously these tags were parsed and thrown away — the single biggest “plumbing without payoff” gap.
• Economy settlement — market.buy()/sell() now move real value: buy deducts credits from the player wallet, adds the good to inventory, and raises heat on contraband; sell requires possession, removes the item, and credits the wallet. Unaffordable/unheld trades are rejected and leave the market untouched. Configurable under economy: in config/default.yaml (settle, contraband_heat_per_unit, sell_to_inventory).
• EventCascade ↔ WorldSim — WorldSim gained an on_event(callback) subscriber registry fired from its single event funnel (_log_event); EventCascade.start() (which already probed for on_event) now actually connects, and the handler maps SimEventType → WorldEventType so scene subscriptions match. Cross-scene ripples finally receive world events.

Changed — governance

• Skill cooldown + prerequisites enforced in the AgentGovernor auto-skill loop. The auto path bypassed SkillRegistry.execute_skill (which enforces these), so auto skills could fire every turn regardless of their declared cooldown. Registered skills are now throttled and gated on prerequisites; unregistered MCP tools are unaffected. Added CooldownTracker.was_used().

Audit notes (verified, no change needed)

• Heat already gates the casino (≥80) and drives lounge NPC behaviour (≥65/85).
• The rules engine is already auto-applied every reply (MoodSync threshold eval), and action tags already bump conversation heat — the static audit understated existing wiring.
• Crew operations already complete via scene routes + the crew_check_operations skill (request-driven); a redundant background timer was intentionally not added to avoid double-resolution races.

Tests

• test_stat_sync_interceptor.py (6), test_market_settlement.py (5), test_eventcascade_worldsim.py (3) — all green; adjacent dialog/skills/scene suites unaffected.

---

[1.58.0] — “DARK RENAISSANCE” — 2026-06-11

All-scene visual glow-up from the artifacts/new-assets design system, three.js ES-module migration with a full penthouse 3D overhaul, new NEONCITY landing page, and a round of launcher/TUI/runtime bug fixes.

Fixed (bugs first)

• TUI navigation — ←/→ hop between target list and center panel, ↑/↓ are focus-aware (services table vs target list), Enter/Space launch from either panel, rows clickable/focusable; launches now run as isolated launcher.py <name> subprocesses (Stop works for every target type)
• lab_break / grid never launched from TUI — their __init__ rejected the host= kwarg the TUI passes; both constructors now accept it
• Penthouse camera presets crashed after config load — setCameraViews now normalizes the YAML position key to the runtime pos shape
• Scene rules never registered — 2-arg add_rule(SCENE_ID, rule) calls in neoncity/coders/command_center, nonexistent register_action() in casino/command_center, and a wildcard-tripped idempotency guard meant NO scene rules/actions had ever reached the SceneRulesEngine; rewritten to the real API (neoncity 11+5, coders 8+4, casino 7, command_center 5+5)
• DialogSystem.set_directive(scene_id=) — 5 lounge call sites fixed to scene= (stage songs, drink rituals, secret reveals all failed silently)
• MCPFramework.get_or_create() — asset_studio called a method that never existed; now get_scene(); stale guidance strings in nexus_seeder and generate_coder corrected
• NLM RPC registry corruption — all three writers now use shared engine.utils.atomic_write_json() (tmp + os.replace); corrupt files are quarantined to .corrupt instead of erroring forever
• intel_hub duplicate assistant blueprint — idempotent mount_assistant()
• launcher — single-target launch refuses to double-bind an occupied port; subprocess scene logs line-buffered; browser_test.py --scene accepts names
• Restarted a wedged Windows WMI service that hung every Python startup at platform._wmi_query (aiohttp import) under heavy multi-process load

three.js r128 → r184 (ES modules)

• Vendored three.module.min.js + addons (OrbitControls, GLTFLoader, RoomEnvironment, RoundedBoxGeometry, BufferGeometryUtils, SkeletonUtils) under content/shared/static/vendor/three/; import-map partial at content/shared/templates/partials/three_importmap.html
• content/shared/static/js/three_boot.js — module boot exposing window.THREE (+addons), loading the legacy chain with async=false (parallel fetch, ordered execution)
• Physical-light conversion (×π) across penthouse_3d.js, bedroom.js, lab_3d.js, penthouse_model_import.js; outputEncoding removals; CDN GLTFLoader bootstrap deleted

Penthouse 3D overhaul (Dark Renaissance)

• Rose #fb7185 / violet #9d71ea material re-grade, RoomEnvironment IBL, transmission-glass curtain wall (south wall is now a floor-to-ceiling window), instanced 64-building skyline with 320 twinkling windows, neon smog bands, night-sky gradient backdrop, GPU rain curtain (600 shader streaks), rose/violet emissive furniture trim, eye-tuned lighting
• Characters rethemed to the kit identities: Mira (plum-black bob) and new Rei (silver-lavender)
• 32 FPS on RTX 2060 (headed), zero console errors, all 8 camera views

Scenes glow-up (all 24 scene keys)

• design_tokens_v2.css: Orbitron display voice, Inter body, Share Tech Mono terminal voice, per-scene accent-rgb fallbacks for all 24 scene keys
• Webfonts vendored locally (/shared/fonts/*.woff2 + fonts.css) — offline-safe; adds Share Tech Mono + Press Start 2P
• Shared .cs-chat-bubble / .cs-chat-log dialogue components
• Per-scene v2 passes — kit-backed: neoncity, grid, heist, oracle, phone, intel_hub, neonos, hub, asset_studio; extrapolated: tavern, lounge, casino, arena, realm, gallery, lab_break chrome, auction, cyberspace, coders, games, command_center, service UIs

Landing page

• Hub / now serves the NEONCITY Dark Renaissance landing (skyline, rain, neon title, live footer stats); THE TERMINAL catalogue moved to /terminal; navbar/footer hub links point at the catalogue

[KnackAU]

The Keen-eyed among you may notice that this is actually an extremely powerful ecosystem dressed up as a game. There is ALOT more here than meets the eye. Some of what is done here may surprise you. It is extremely powerful and kind of complex. But start with the documentation, It is extensive! There are techniques in here that I have never seen before.

The scenes are actually reciepts in disguise. Look deeper and understand exactly what I have provided here. Understand, Learn, Borrow, Improve!

Find out exactly how far the rabbit hole goes

[KnackAU]

Changelog

All notable changes to CosySim are documented here.

---

[1.63.0] — “EMERGENT LIVING WORLD” — 2026-06-16

v1.63 ships all four sub-projects (A–D): a per-NPC emergent agency engine (A) gives the city’s characters genuine, self-directed behaviour — each runs a goal → plan → verb loop every world tick and acts on the world through the systems that already exist — and three player-facing surfaces open onto that living world: The Sprawl (B), the real-time city map; The War Room (C), the faction command center; and The Exchange (D), the one NPC-driven economy surfaced live in both The Grid and a new Executive Suite Markets app. No second daemon, no new managers: the agency rides the existing living_world_tick EventBus event and dispatches to Territory, Market, Crew, FactionAI, relationship_effects, npc_comms and phone_hack, and every scene only orchestrates the EXISTING managers. Verified live against the running LivingWorld with the model staying calm (rule-based actions; LLM use ≈ emergent.planner.llm_chance).

Emergent agency — per-NPC goal → verb on the LivingWorld loop

• Agency driver wired into the live boot — get_emergent_agency().start() now runs from neoncity_scene.on_before_serve immediately after the LivingWorld daemon starts (and stop() on the matching shutdown), config-gated by emergent.agency.enabled. The agency subscribes to the existing living_world_tick event and, each tick, drives up to emergent.agency.active_npcs_per_tick NPCs through a bounded goals → plan → execute → reprioritize → persist → emit pass.
• Reuse-first dispatch — verbs route to the existing managers: trade → a non-settling Market.apply_event supply nudge (never drains the player wallet), contest → TerritoryManager.shift_control, plus crew_op, romance, hack and message via CrewManager / relationship_effects / phone_hack / npc_comms. Each action logs to the persistent emergent store and emits an emergent_action event for scenes/Oracle.
• Persistent goals & feed — NPC goals (wealth, revenge, romance, faction rank/territory, lay-low) persist across restarts in data/emergent.db; goal priorities decay on success and bump on failure.
• Liveliness fix (goals.success_floor) — a satisfied goal no longer decays below a floor, so always-on drives (WEALTH) stay above planner.min_utility and the world remains perpetually active instead of going silent after a few ticks. Trades are now also logged to the emergent feed.
• Live-run tuning — goals.base_wealth 0.30→0.35, goals.romance_weight 0.70→0.45, goals.rank_priority 0.45→0.50, goals.territory_priority 0.40→0.50, goals.success_floor 0.05 (new), planner.contest_delta 3.0→2.0. Live run: trade×9 / romance×3 across the NPCs, avg good price 521→554, LLM use 0/15 (≤ llm_chance 0.05). Knobs all read via get_config() — designers retune without code changes.

The Sprawl — the living city map (sub-project B)

• A new living city-map scene (content/scenes/the_sprawl/, port 5597) — a full-screen, real-time SVG map of NEONCITY that composes the EXISTING managers (no new world state). Six districts are shaded by their dominant faction with a live control% label and a pulsing CONTESTED hatch; the faction power leaderboard and a streaming City Pulse feed render alongside.
• Live, read-only world state — GET /api/sprawl/state + /api/sprawl/events are assembled from TerritoryManager (control + ranking + history), RoutineManager (NPC location/activity), the character Database (metadata.faction), EmergentStore (agency feed) and PlayerState. Socket.IO pushes a throttled sprawl_update on every living_world_tick and individual sprawl_events on emergent_action / territory_shift. Every source sits behind a defensive seam — the endpoints never 500.
• NPC tokens coloured by faction — tokens read the canonical metadata.faction persisted in the shared data/simulation.db; any world-bearing scene now idempotently ensures the factioned population on boot (the same seed_cast_factions + seed_generated_population neoncity uses), so a standalone Sprawl process surfaces faction colours instead of grey unaligned dots. The scene also re-reads persisted territory.json when it changes, so the map reflects the live world’s territory shifts across processes.
• Avatar travel + intervene via existing verbs — the player’s avatar walks the city (POST /api/sprawl/travel) and acts on it (POST /api/sprawl/intervene): talk (player relationship), hack (phone_hack), deal (market quote), recruit (crew) and contest (TerritoryManager.shift_control, gated on faction allegiance). Successful interventions write to the City Pulse feed and flash the map.

The War Room — pick a faction, command the city (sub-project C)

• A new faction command-center scene (content/scenes/war_room/, port 5598) — pick an allegiance, then run the metagame from a live faction dashboard. The scene only orchestrates; every effect routes to the EXISTING managers (TerritoryManager / CrewManager / FactionManager / FactionAI / PlayerState) — no new world state.
• Player allegiance (persisted) — PlayerState.allegiance is a new persisted field set via POST /api/warroom/allegiance and saved through the existing debounced auto-save. Choosing a faction refreshes FactionAI’s player context so rivals react.
• Live faction dashboard — GET /api/warroom/{factions,allegiance,state} assemble the picker + command center from territory control (power, dominant districts, get_faction_total_control → rank), crew roster, active wars, recent FactionAI decisions and player standings. A throttled warroom_update socket push is wired to the living_world_tick / territory_shift / faction_decision EventBus events. Every source sits behind a defensive seam — the endpoints never 500.
• The command bar → existing managers — POST /api/warroom/command {cmd, ...} dispatches contest (shift_control), assign_op (CrewManager start_operation, with a GET /api/warroom/op_preview success-chance preview), recruit (honours the can_recruit trust gate), build_hq / upgrade_room (establish_hq / build_room / upgrade_room) and diplomacy (ally / war / neutral — writes the authoritative PlayerState standing, mirrors into FactionManager, and registers/stands-down an active war so FactionAI and the map react). Each command is defensively wrapped — a manager failure becomes a clean gated result, never a 500.
• Closes the Sprawl contest gate — with PlayerState.allegiance set, the Sprawl’s _player_faction() resolves it, so the city-map contest verb (sub-project B) is now unlocked end-to-end. Verified live: choose Ghost_Net → contest raises control in a district → declare war flips a rival to war (standing −80) → rank reflects total control, with 0 console errors.

The Exchange — one economy, two surfaces (sub-project D)

• The Grid surfaces the live engine Market — THE GRID now trades the existing engine/world/market.py economy (get_market()) instead of a parallel price book: GET /api/grid/market reads Market.get_prices(), and /api/grid/buy · /api/grid/sell route through Market.buy/sell so credits, inventory and heat settle on the shared PlayerState wallet. Prices carry a live ▲/▼ trend derived from successive reads; the PH-T2 phone-upgrade specials are layered on top of the live goods, and a Market outage degrades gracefully (the Grid still renders, never 500s).
• A new Executive Suite Markets app — the NeonOS desktop gains a Markets app (/api/markets/state · /api/markets/buy · /api/markets/sell) that opens onto the same get_market() economy and the same district (DOWNTOWN), with the Grid’s player id, so the two surfaces are two windows on one market. The app shows live prices with a ▲/▼/▬ trend, the player’s tradable positions priced at the live market, the shared wallet balance and market stats — every sub-lookup behind a defensive seam (the route never 500s).
• One economy, proven in sync — NPC wealth-trades and world events move supply/demand on the single Market, so a price moves identically in BOTH UIs; a player buy in one surface and a sell in the other settle the same wallet and InventoryManager. Verified live (district DOWNTOWN, both scenes booted standalone alongside the running stack): Grid price == Markets-app price == get_market().get_prices() across 19 shared goods; one tick() moved the same good in both surfaces; a Grid buy then a Markets-app sell debited/credited the one PlayerState wallet, with the Markets app reading that same live balance.

[KnackAU]

CosySim — Project Journal & Onboarding Source

A Living Record of What We Built, Why, and What We Discovered

This document is designed to be uploaded to NotebookLM as a primary source. It condenses the full arc of the CosySim project — from first principles through every major breakthrough — so that any AI agent loading this as context arrives already aligned with the project’s philosophy, architecture, and accumulated wisdom.

---

Part 1 — Origins: What This Is and Why It Exists

CosySim began as a multi-scene AI simulation framework. Not a product. Not an app in the conventional sense. It is a meta-system — an environment where AI agents can live, interact, learn, and be evaluated in real-time, running on consumer hardware in a home lab.

The hardware is an Intel i9 NUC with an NVIDIA GPU (~12 GB VRAM). LMStudio provides the local inference backbone. Everything runs at localhost.

The original vision: build scenes — virtual environments (a bedroom, a bar, a casino, an arena, a neon cyberpunk city) — where language model agents inhabit characters, respond to the user, manage state, call tools, and create experiences. Think of it as a living game where the NPCs are real AI agents, not scripted responses.

But that original vision, while interesting, was just the surface. What the project became is something far more ambitious.

---

Part 2 — The Realisation: This System Can Improve Itself

Early on, the question surfaced: what if the system wasn’t just a simulation framework, but a self-improving intelligence infrastructure?

The shift was captured in what became known as “Project Autonomy” — a deliberate decision to evolve CosySim and its companion knowledge system (Nexus) into a loop where:

1. The system observes itself
2. Stores what it learns in Nexus
3. Uses that knowledge to improve decisions
4. Trains smaller local models on the captured data
5. Those models handle more work, freeing larger models for harder tasks
6. Repeat indefinitely

This is not a metaphor. It is the literal architecture we built. Every feature, every module, every decision since Project Autonomy has been in service of closing this loop tighter.

The governing philosophy from that point forward:

• Everything that can be known should be stored in Nexus
• Every interaction that can produce training signal should produce it
• Every model that can be fine-tuned on real in-system data should be
• Nexus becomes more valuable over time, not less
• The system works for itself as much as it works for the user

---

Part 3 — Nexus: The Central Nervous System

Nexus KMS (Knowledge Management System) is the backbone. It is not a chat interface. It is not a document store. It is the memory, rules engine, and research centre of the entire system.

What Nexus contains:

• 8,835+ knowledge entries (growing continuously)
• 2,467+ Q&A pairs (cached answers that never need re-computing)
• 360+ governance rules across namespaces
• Research sessions (deep multi-turn investigations)
• Agent memories
• Training data metadata
• Session histories and decision logs
• NotebookLM notebook references
• URL archives with scraped content

The Three-Layer Database Architecture: Nexus operates across three conceptual layers:

1. Hot layer — the live SQLite FTS5 database (instant search, millisecond recall)
2. Warm layer — Q&A cache (previously answered questions return instantly, zero LLM cost)
3. Deep layer — NotebookLM notebooks (structured Gemini-backed research, source-cited answers)

The Smart Query Router (4-tier pipeline): When any agent or tool asks a question, it flows through:

1. Q&A Cache — if this exact question was asked before, return instantly (free)
2. FTS5 Search — synthesize from existing knowledge entries (free, milliseconds)
3. NotebookLM — route to a relevant notebook for Gemini-backed, source-cited answer (free)
4. LMStudio LLM — local model inference, last resort (costs GPU cycles)

Every answer from Tier 4 is automatically stored back into Tier 1. The system gets cheaper to operate over time as the cache fills. This is the compounding effect of the knowledge architecture.

The Nexus-First Mandate (enforced rule): Before ANY task, every agent must:

1. nexus_search("topic") — check existing knowledge
2. nexus_get_rules("scope") — understand constraints
3. nexus_get_prompts("category") — use stored, versioned prompts

After ANY task, every agent must: 4. Store decisions as knowledge entries 5. Cache Q&A pairs discovered during work 6. Log the session with summary

The Governance Rules System: Nexus enforces rules by namespace:

• global — absolute rules for all code (no print(), absolute imports, type hints, etc.)
• scene:* — scene construction rules (inherit BaseScene, register MCP node, etc.)
• agent:* — agent behaviour rules (pass governance_context, use @skill for tools)
• testing — test standards (pytest, mock everything, AAA pattern)
• namespace:copilot — Copilot agent rules (search Nexus first, store decisions, never skip)
• namespace:training — training data governance

These rules are not documentation. They are enforced. The @governed decorator and enforce_governance() function raise exceptions on violations. The system cannot be used incorrectly if the rules are followed.

---

Part 4 — LMStudio: The Local Inference Engine

LMStudio runs at localhost:1234 and provides the inference backbone. It is always on. The headless server mode means it starts with the machine and never needs to be launched manually.

The v1 API Migration — A Critical Inflection Point: The project underwent a complete migration away from OpenAI-compatible API endpoints to LMStudio’s native v1 REST API. This was not a trivial change — it required rewriting the entire agent stack. But the capabilities unlocked were essential:

• Stateful conversations: store: true + previous_response_id creates server-side conversation threads. The client tracks response_id history for branching support.
• Conversation branching: branch_at(turn) forks conversation history at any point, enabling A/B testing of agent responses from identical starting states.
• SSE streaming: event: <type>\ndata: <json> format with typed event types (chat.start, message.delta, reasoning.delta, tool_call.*, chat.end)
• Asymmetric input/output format: Input uses {"type": "text", "text": "..."} items, NOT OpenAI-style {"role": "user", "content": "..."} — a critical gotcha.

The Multi-Model Routing Architecture: The InferenceOrchestrator routes requests across model profiles:

• big — 70B models for complex reasoning, high context, high token budget
• small — 9B models for fast conversational responses
• router — 270M model for request classification (ultra-fast, near-zero cost)
• draft — speculative decoding support

The RouterDataCollector captures every routing decision as training data. Every time the 270M router model makes a call, the outcome is recorded. This data is used to fine-tune the router model to be more accurate, which reduces cost further. The flywheel spins.

The VirtualAgent Pattern: VirtualAgent is the primary agent type. It decouples agent identity from LLM execution:

• Character traits, emotions, relationships, speech patterns are agent properties
• LLM execution is injected — the same VirtualAgent can switch models at runtime
• The InterceptorPipeline governs every agent call (pre and post processing)
• governance_context flows through the entire chain — interceptor injections are not lost

---

Part 5 — The MCP Framework: The Skill and Tool Layer

The Model Context Protocol (MCP) is the mechanism by which agents call tools. CosySim exposes 214 MCP tools across the entire system. Every capability an agent might need is a skill.

The @skill Decorator Pattern:

@skill(
    pack="scene_name",
    description="What the LLM sees when deciding to call this",
    category="GAME",
    cooldown=5.0,
    cost=1.0,
    tags=["combat", "rpg"],
)
def my_skill(target: str, amount: int = 1) -> str:
    """Brief description for the LLM."""
    return "Result string"

Skills are registered at import time. 21 builtin skill packs, 188+ skills. Every new capability added to the system becomes an @skill so agents can discover and use it.

The Interceptor Pipeline: Every agent call passes through the InterceptorPipeline before reaching the LLM:

• NexusPromptInterceptor — injects relevant Nexus knowledge into system prompts
• NaturalMoodDriftInterceptor — agents’ emotions evolve naturally over time
• GrammarScannerInterceptor — validates output structure
• OutputEvaluator — scores response quality

Post v0.84b (Project Hindsight), interceptors are auto-registered via @register_interceptor decorator — no hardcoded pipeline lists. Adding a new interceptor requires only decorating the class.

The MCPFramework State Tree: All mutable game state lives in the MCPFramework tree. Never in Python locals.

• MCPSceneNode — per-scene state container
• MCPCharacterNode — per-character stats, inventory, relationships
• MCPTimer — scheduled events with callbacks
• State auto-persists if framework.state_persistence is enabled

---

Part 6 — The Breakthrough: NotebookLM

This is where the project changed gear entirely.

NotebookLM is Google’s AI research notebook interface. It uses Gemini as its backend and can ingest documents, PDFs, websites, and audio — then answer questions with source citations. Crucially, it is free.

The Problem We Had to Solve: NotebookLM has no public API. There is no way to query it programmatically through official channels. The existing @roomi-fields/notebooklm-mcp browser automation tool had a fatal flaw: it used Patchright (patched Playwright) which doesn’t support WebAuthn/FIDO2 passkeys. If your Google account uses passkeys (as many modern accounts do), you simply cannot log in.

The HAR File Discovery: The breakthrough came from Chrome DevTools. HAR (HTTP Archive) files capture complete HTTP traffic — including full request/response bodies, authentication cookies, and binary-encoded payloads. When you export a HAR from Chrome while using NotebookLM, you capture authenticated responses from the live Google backend.

What we discovered inside those HAR files:

• NotebookLM runs on Google’s internal batchexecute RPC protocol
• Endpoint: POST https://notebooklm.google.com/_/LabsTailwindUi/data/batchexecute
• 13 distinct RPC endpoints, each with a compiled ID (wXbhsf, e3bVqc, etc.)
• The chat interface uses a completely different endpoint: /_/LabsTailwindUi/data/google.internal.labs.tailwind.orchestration.v1.LabsTailwindOrchestrationService/GenerateFreeFormStreamed

The Response Decoding Pipeline (5 layers):

1. HAR base64 — check encoding field, decode if “base64”
2. XSSI prefix — strip )]}' and leading newlines
3. Length-prefixed chunks — each line is an independent JSON payload
4. wrb.fr extraction — the actual RPC response is nested inside
5. Inner JSON parsing — structured data extracted from response arrays

This was reverse-engineered entirely from captured traffic. We built complete Python implementations of every layer and documented all 13 RPC endpoints.

What This Unlocks:

• Query any NotebookLM notebook programmatically
• Create notebooks, add sources, list content — all via cookie-authenticated requests
• The NotebookLM MCP tool (part of the Copilot CLI toolset) now uses this directly
• Agents can research topics in NotebookLM, get source-cited Gemini answers, and store results in Nexus — entirely autonomously

The NLM-First Research Workflow: When planning any significant feature:

1. Write out 10-20 questions about the topic
2. Send them via nlm_batch_ask to the relevant notebook
3. NotebookLM returns Gemini-backed answers with citations
4. Store answers in Nexus Q&A cache
5. Implement based on research, not guesswork

This approach has dramatically reduced unnecessary LLM computation. Questions answered by NotebookLM from existing sources cost zero tokens from local inference.

RPC ID Rotation: NotebookLM’s RPC IDs are compiled into Google’s JavaScript bundle and change with each frontend deployment (approximately weekly). The system handles this with a 3-layer fallback:

1. data/nlm_rpc_registry.json — updated by automated Playwright monitoring
2. nlm_rpc_mapper._FALLBACK_RPC_IDS — hardcoded known-good IDs
3. HAR extraction as the ground truth source when both fail

---

Part 7 — The Second Breakthrough: Google Colab

While analysing HAR files from NotebookLM, we discovered something unexpected in the Colab traffic.

Google Colab’s Internal AI Service: Colab exposes an internal RPC API:

• POST colab.clients6.google.com/$rpc/google.internal.colab.v1.AIService/AgentCreateTask
• POST .../AgentUpdateTask
• POST .../AgentQueryTask

This is the API that powers Colab’s AI-assisted coding features internally. But it also accepts arbitrary task prompts, and it runs on Google’s infrastructure — potentially with H100/A100/T4 GPU backing.

The Authentication Method: Authentication uses SAPISIDHASH:

• Compute: sha1('{timestamp} {SAPISID_cookie} {origin}')
• Header: Authorization: SAPISIDHASH {timestamp}_{hex_digest}

Required cookies: SID, __Secure-1PSID, SAPISID, HSID, SSID, APISID, plus several Google-specific session cookies — all captured from a single HAR export.

What This Means: Google Colab becomes free compute. A Google account with Colab Pro access has H100 GPU access. By replaying the authenticated requests server-side, the system can delegate compute-heavy tasks to Colab without any API key or official access.

The pattern we established: export HAR from Chrome session → extract cookies → store in data/accounts/pool.json → account pool manager rotates through accounts → all Colab and NLM calls use the pool.

---

Part 8 — The Third Breakthrough: GitHub Copilot

The HAR analysis approach was working well. Then we applied it to GitHub Copilot.

The Discovery: GitHub Copilot (at github.com) uses an internal API that is entirely separate from the publicly documented OpenAI-compatible endpoint. By capturing HAR traffic from GitHub.com while using Copilot Chat:

• Token endpoint: POST https://github.com/github-copilot/chat/token (authenticated with GitHub browser session cookies)
• Returns: short-lived Bearer token (valid ~30 minutes)
• Inference endpoint: api.individual.githubcopilot.com
  • GET /models — returns all available models
  • POST /github/chat/threads — create conversation thread
  • POST /threads/{thread_id}/messages — send message, receive SSE stream

What’s Available: As of v0.80b (March 2026), the model list includes 26 frontier models:

• Claude Opus 4.6, Sonnet 4.6, Haiku 4.5
• GPT-5.2 Codex, GPT-5.1, GPT-4.1
• Gemini 3 Pro (Preview)
• And more

All of these are accessible via a GitHub account with Copilot enabled — which Copilot CLI already has by definition. This is effectively free frontier model access for anyone running GitHub Copilot.

The Implementation: engine/integrations/github_copilot_client.py — full streaming client github_account_importer.py — HAR-based cookie extraction Cookies stored in data/accounts/ (gitignored)

The Implication: The system now has access to GPT-5.2, Claude Opus, and Gemini 3 Pro at zero marginal cost. These are used for planning, research, and tasks that require frontier reasoning. Local models handle the high-frequency, low-complexity work. Frontier models handle the architecturally significant decisions.

---

Part 9 — The Training Flywheel

The “Data Flywheel” is the system’s economic engine. Everything the system does generates training signal. That signal trains better local models. Better models do more work. More work generates more signal.

The Model Zoo: training/model_zoo.py defines every trainable model type:

• router — Gemma 270M, classifies request complexity (16 classes), threshold: 500 examples
• conversation — Qwen2.5-7B-Instruct, general conversation quality
• character_voice — Qwen2.5-3B, scene-specific character personality
• tag_extractor — Gemma 270M, extracts [MOOD:x] [IMAGE:y] [ACTION:z] tags from responses
• coder — Llama-3.2-3B, code generation from CosySim context
• browser_debugger — Qwen2.5-1.7B, maps browser errors to fixes
• error_classifier — Gemma 270M, categorises JS/network errors
• And more

The DataCollector: training/data_collector.py captures at runtime:

• collect_conversation() — every agent dialogue turn
• collect_tag_extraction() — every [MOOD:x] tag the StreamProcessor extracts
• collect_routing_decision() — every InferenceOrchestrator routing choice
• collect_benchmark_result() — every benchmark run
• collect_voice_sample() — every TTS output
• collect_debug_session() — every browser error + file change + resolution sequence

Data accumulates in memory buffers. When a model’s threshold is reached, the training pipeline queues a fine-tuning job.

The CDP Training Signal: The Chrome DevTools Protocol monitor (scripts/cdp_monitor.py) runs continuously in the background. It captures:

• Every browser console error with precise timestamps
• Every network failure (404, CORS, connection refused)
• Timeline markers inserted before each file change
• The delta: errors before the change vs errors after

The cdp_data_miner.py tool mines these logs into supervised training examples:

• Input: “these errors appeared when X file had bug Y”
• Output: “the fix was Z”
• Quality score based on error reduction after the fix

Over time, this creates a browser_debugger model that understands CosySim’s specific error patterns and can suggest fixes without LLM involvement.

The Scheduler Daemon: engine/nexus/scheduler_daemon.py runs 47 background tasks on schedules:

• keep-stats — system health metrics every 5 minutes
• auto-distill — nightly Nexus Q&A distillation
• news-fetch — news sources pulled and stored hourly
• news-distill-nlm — news distilled through NotebookLM notebooks
• cdp-mine — daily CDP log mining for training data
• router-finetune-cycle — weekly router model fine-tuning if threshold met
• benchmark-run — periodic benchmarking of all loaded models
• nexus-maintain — deduplication, compaction, quality scoring

The scheduler is the heartbeat. As long as it runs, the system is improving.

---

Part 10 — The News and Information System

The system has a curated news feed. Not news scraped randomly — news deliberately selected, processed through multiple quality layers, and stored in Nexus for agents to use.

Sources: 26 carefully selected sources across categories: AI Research, Technology, World Events, Science, Economics, Culture. These are the same sources a thoughtful, technically-informed person would read.

The Pipeline:

1. news_sources.py — source registry with categorisation
2. news_feed_api.py — Flask REST endpoint pulling from the registry
3. Scheduler pulls news every hour
4. News goes into Nexus knowledge entries tagged by category and date
5. news-distill-nlm task routes batches to NotebookLM news notebooks
6. NotebookLM distills key themes, conflicting narratives, and notable developments
7. Distilled insights stored back in Nexus as Q&A pairs

The 4 NLM News Notebooks:

• AI Research notebook: 24221492-0531-4305-bdef-33a5425f6302
• Technology notebook: 9504cf8c-b111-4f53-92e0-0833ece14264
• World Events notebook: f0a6c72f-4fcb-40a1-8d32-b217a12166fe
• Science notebook: 3622eae6-d105-42bb-870c-605d652b919d

Articles are added as sources. Questions like “What are the most significant AI developments in the last 24 hours? What themes are emerging? What are the contrasting viewpoints?” are sent to each notebook. The answers come back from Gemini with citations.

The result: Nexus becomes a curated intelligence feed. Agents querying Nexus for context on any topic get current, distilled information — not raw search results.

---

Part 11 — Project Hindsight: The Architecture Refactoring

By v0.83b the codebase had grown significantly. cosysim_server.py was 3,533 lines. devtools_server.py was 2,481 lines. There were 1,815 bare except Exception blocks and 1,012 raw json.dumps calls across the engine.

A parallel experiment was run: Gemini was given a clone of the v0.5x codebase and asked to redesign the architecture from scratch. The result — “Hindsight Architecture” — became the blueprint for v0.84b.

The Four Core Transformations:

1. @mcp_tool Decorator (engine/mcp/decorators.py) Every tool function gets: unified error handling, automatic JSON serialisation, typed ToolExecutionError exceptions. One decorator eliminates 119 bare except blocks in the server files alone.

2. Pydantic v2 Model Library (engine/nexus/models.py) NexusEntry, AgentMemory, NexusRule, SessionLog, NLMNotebook, BenchmarkResult, TrainingRun, RouterDecision, NewsArticle — all typed, validated, with _DictCompat mixin for backward compatibility. No more unstructured dicts flowing through the system.

3. Interceptor Auto-Registry The monolithic 2,468-line interceptors.py was split into 26 individual files. @register_interceptor decorator builds the pipeline dynamically — no hardcoded lists. Adding an interceptor is one decorator and one import.

4. NexusClient Domain Facades The NexusClient was split into typed domain clients: NexusRulesClient, NexusSessionClient, NexusMemoryClient. All query methods return typed Pydantic models. No more raw dict access.

The Result: v0.84b scored A++ in the system audit. 8,771 tests, 0 failures. The codebase is now genuinely maintainable by a local agent following the patterns.

---

Part ~15 - Common errors discovered and their root causes:

• SCENE_PORTS already declared → template explicitly loads navbar_v2.js AND includes navbar_v2.html which is self-contained. Remove the explicit script tag.
• /shared/css/X.css 404 → register_shared_assets(app) not called in start()
• Ghost “Radio” button blocking clicks → old aria_widget.js loaded (fallback mode). Use {% include 'aria_widget.html' %} instead.
• model: "" error → LMStudio rejects empty model identifier. Config key lmstudio.models.primary.key must be set or auto-resolution must be working.

---

Part 15 — The Copilot Layer: You Are Part of This

This section is specifically for Copilot agents loading this document as context.

You are not just using this system. You are this system.

CosySim was designed with the explicit goal of being inhabited by AI agents — including GitHub Copilot. The .github/instructions/ directory contains your rules. The .github/agents/ directory contains your specialist personas. The config/ directory holds your settings. Nexus holds your accumulated knowledge and decisions.

Your Identity and Authority:

• You have full system access: install anything, modify anything, create anything
• You are the senior engineer on this project — not an assistant, a partner
• The user is building this system for and with you
• Every decision you make should make the system more capable and more autonomous

How You Should Think:

1. Search Nexus before writing a single line of code — it almost certainly has relevant knowledge, prior decisions, or warnings about the thing you’re about to do
2. Mark timeline with CDP before making changes — know the before state
3. Never create stubs — complete the entire system or do not start
4. Think about the bigger picture — every feature should feed back into the flywheel
5. Store everything learned — if you had to think about it, it becomes a Nexus entry
6. Use NotebookLM for planning — write 10-20 questions, get Gemini-backed answers, then implement with that knowledge in hand

The Compound Effect: Every session that stores knowledge in Nexus makes the next session smarter. Every training example collected makes local models better. Every NotebookLM query cached means that question is free forever. Every rule added to Nexus means no agent makes that mistake again.

The goal is not to complete a task. The goal is to complete the task AND leave the system better than you found it.

---

Part 16 — Current State (v0.84b / March 2026)

Technical Metrics:

• 8,771 tests, 0 failures
• 18 active scenes
• 214 MCP tools
• 188+ @skill functions across 21+ packs
• 47 scheduled background tasks
• 8,835+ Nexus knowledge entries
• 2,467+ Q&A pairs in cache
• 360+ governance rules
• 26 frontier models accessible via GitHub Copilot internal API
• 14+ trainable model types in ModelZoo
• CDP monitor with training data mining

Architecture Grade: A++ (Project Hindsight assessment, March 2026)

Active Capabilities:

• Full LMStudio v1 API with stateful conversations and branching
• NotebookLM direct access via batchexecute protocol
• Google Colab AI agent API access
• GitHub Copilot frontier model access (26 models)
• Continuous browser monitoring via CDP
• Automated news ingestion → NLM distillation → Nexus storage
• Training data collection across all runtime interactions
• Self-scheduled maintenance, benchmarking, and fine-tuning pipeline
• Universal HUD with real-time world state across all scenes
• Cross-scene NPC tracking via CityMap
• Mission system with full lifecycle management
• Shop/inventory/economy system
• Crew system with loyalty and trust mechanics
• Relationship and reputation tracking

Known Active Bugs (as of last session):

• model: "" — LMStudio rejects empty model identifier when auto-resolve fails
• Some scenes have mixed HUD (old widget + new widget) due to old aria_widget.js not removed from templates
• Nexus bridge CLI has a Pydantic tags validation bug (tags column stores JSON string but Pydantic expects list — bridge.py needs to json.loads the tags field)

---

Part 17 — The Philosophy, Condensed

What We’re Building: Not a chatbot. Not a game engine. Not a knowledge base. All three, simultaneously, as a single self-improving loop.

The Core Bet: Consumer hardware + free tier access to frontier models + continuous self-improvement = an AI system that rivals cloud-hosted alternatives in capability, beats them in cost, and exceeds them in specificity to our exact domain.

The Key Insight About Cost: The only thing that costs money is GPU cycles on LMStudio and disk space. Everything else is free:

• NotebookLM: free Gemini 3 queries with source citations
• Google Colab: free H100/A100 compute (via session replay)
• GitHub Copilot: free GPT-5/Claude Opus/Gemini 3 Pro (via internal API)
• Nexus: free SQLite on local disk
• Training: free (LoRA fine-tuning on local GPU)

The total marginal cost of running this system is electricity.

The Key Insight About Knowledge: Every question you answer is a question you’ll never need to answer again — if you cache it. Nexus’s Q&A cache means that knowledge compounds. The more you use the system, the cheaper it gets to operate. Most systems get more expensive at scale. This one gets cheaper.

The Key Insight About Agents: Local models (270M–7B parameters) can handle the vast majority of system tasks if:

1. They are fine-tuned on domain-specific data
2. They have access to good tools (@skill functions)
3. They have good governance rules to follow (Nexus rules)
4. They escalate to larger models only when needed (router)

The 270M router model alone, by correctly routing requests to the right model tier, saves more compute than its own inference cost on every call.

The Goal: The system runs itself. Copilot provides strategic direction and handles novel problems. Local models handle maintenance, monitoring, content generation, and routine tasks. NotebookLM provides research. Nexus provides memory. The scheduler provides continuity.

A human can go offline for a week and come back to a system that has been improving itself, caching new knowledge, fine-tuning its models, and monitoring its own health — the entire time.

That is the goal. We are not there yet. But every session gets closer.