MosAIc: a reconfigurable surface where an LLM emits a layout, not just text

[vonargo]

I’ve been exploring an alternative to linear chat: instead of a transcript, the model emits a small JSON overlay that re-configures the UI, and the surface lays it out. The interesting parts are the mechanism and the contract and would love some technical feedback (or thrown spears).

The core mechanism is a couple dozen lines.

A surface holds views that lay out “tesserae” (typed tiles: markdown / code / table / diagram / note / tasks).

An overlay is a surface-shaped patch, merged over a base by view id:

• matching id: fields shallow assigned over the base view (set tesserae to replace tiles, omit to keep them)
• new id: appended as a new view/sidebar entry
• remove: [id] and it drops views

effective(base) = base merged with overlay, and the entire render path reads only from effective(), so one overlay reshapes the sidebar and the tiles, no imperative DOM code. The schema is deliberately tiny and it’s meant as a render target any producer (model, agent, IDE) can write to SCHEMA.md

It evolves, not regenerates (a bit simplistic right now tbh). Successive tasks are sent the current surface and emit a patch while composeOverlay() folds each into the accumulated overlay with the same id-merge, so “add a PKCE section” extends the surface instead of starting over. (Composing a patch into the accumulator yields the same result as applying it to the rendered surface, so the accumulator stays one overlay.)

Architecture is viewer-billed with no back end. Static Space, pure HTML/ES-modules, no build. The model path is BYO LLM: client side HF OAuth with Inference Providers billed to the signed-in viewer and the token never leaves the browser (no server, no owner key). The OAuth app is provisioned just from hf_oauth in the README.

Robustness. Every overlay (model or hand-edited) passes a validator that normalizes and degrades a bad emit to a message instead of breaking the render; the model path uses json_object, a balanced-brace extractor, and one corrective retry. The model is also told it has no file access, which curbs it from confidently inventing specifics about content it can’t see (though YMMV).

Open question I’d like input on: the wire format. It’s JSON today (precise, trivially validated, but models fumble escaping/brackets and it can’t stream). I wrote up the tradeoff matrix (JSON vs YAML block scalars vs markdown vs XML tags vs tool calls/json_schema) in ROADMAP.md. What have people found most reliable for getting structured UI out of a model?

MIT Licensed code and docs are in the Space tree. No signin demo via the examples: MosAIc - a Hugging Face Space by vonargo

[vonargo]

Google shipped OKF nine days ago as a portable “LLM-wiki”: a folder of markdown files with YAML frontmatter.

The reference viewer renders a bundle as a force-directed graph.

I wanted the opposite: a calm reading surface you’d actually read in.

So MosAIc now opens an OKF bundle as panels: one view per concept type, schema tables and bodies rendered, and cross-links that navigate in place (connectivity-as-reading, not a graph). No sign-in needed and the examples have “Open a sample OKF bundle.”

Two things I leaned into:

1. Provenance. OKF’s frontmatter has no author/confidence/trust field (only resource, timestamp, and a body # Citations). So each concept shows a strip surfacing what it does carry (source link, freshness) and a sourced / unsourced badge that makes the gap visible. The format’s great but the missing trust layer feels like the interesting open problem actively being discussed in the Issues.

2. It’s interactive, not just a viewer. MosAIc is BYO LLM (Inference Providers, billed to you, no backend).

Because the loaded bundle is the model’s context, you can sign in and direct a model against it.

As a test, I typed “list only the unsourced concepts” and it returned exactly the two that lack a source, as a new view.

The static viewer is now a working surface.

MIT, no backend, pure static. Curious what folks think, especially about provenance as OKF’s next layer.

MosAIc - a Hugging Face Space by vonargo

SCHEMA.md · vonargo/mosAIc at main .

[John6666]

Hmm… maybe something like this?:

---

My concise take: I would keep JSON, but I would frame the model output less as “layout JSON” and more as a small, versioned, schema-validated patch protocol over a typed surface.

The interesting part of MosAIc, to me, is not simply that the LLM emits UI. A lot of projects are already exploring “generative UI” in that broad sense. The more distinctive framing might be:

bounded generative UI over a typed work/knowledge surface, where the model proposes auditable patches and the renderer remains the authority.

That framing also gives a fairly practical answer to the wire-format question.

Direct recommendation

I would separate document format from runtime patch format:

Layer	Suggested format
portable knowledge/document bundle	Markdown + YAML frontmatter, e.g. OKF
LLM runtime output	JSON constrained by JSON Schema / structured outputs
surface mutation	JSON Patch-inspired operation list, or a small domain-specific patch protocol
human-readable tile content	Markdown
provenance / source metadata	host/loader-owned metadata, not free-form model claims

So my answer would be:

Use Markdown/YAML for knowledge documents, but use JSON Schema-constrained JSON for live model output. Then make that JSON an explicit patch proposal rather than an arbitrary layout blob.

This preserves the nice simplicity of the current overlay idea, while making the system easier to validate, preview, debug, undo, and secure.

Why not “just layout JSON”?

A raw overlay is pleasantly simple, but as the surface grows, merge semantics tend to become implicit:

• is this replacing a tile or appending one?
• can the model delete a view?
• can it move/reorder tiles?
• can it create official-looking source badges?
• can the user preview the diff?
• can the system reject one unsafe operation while keeping the rest?
• can the patch be logged and undone?

That is why I would consider evolving toward an ops[] model:

{
  "schema_version": "mosaic.patch.v1",
  "intent": "Add a PKCE explanation without replacing the existing overview.",
  "ops": [
    {
      "op": "add_view",
      "id": "auth_pkce",
      "title": "PKCE"
    },
    {
      "op": "add_tile",
      "view": "auth_pkce",
      "tile_id": "pkce_summary",
      "tile": {
        "type": "markdown",
        "content": "PKCE is an OAuth extension that protects public clients..."
      }
    }
  ]
}

This does not have to be raw RFC 6902 JSON Patch. A domain-specific patch language may be friendlier for the model and easier to validate. Internally, it can still borrow ideas from JSON Patch or JSON Merge Patch.

Renderer as authority

The main safety boundary I would keep is:

The model proposes; the renderer disposes.

A possible pipeline:

model output
  ↓
JSON parse
  ↓
schema validation
  ↓
semantic validation
  ↓
policy gate
  ↓
diff preview
  ↓
transactional apply
  ↓
patch log / undo

Structured output helps with parsing. It does not automatically make the output safe, true, useful, source-grounded, or UX-good.

So I would make the host/renderer responsible for:

• validation
• source/provenance resolution
• policy gates
• sanitization
• diff preview
• transactionality
• undo/replay
• capability restrictions

Tile capabilities

The tesserae vocabulary can be more than a rendering detail. It can be the capability boundary.

Capability tier	Examples	Policy
content	markdown, note, table, code block	usually allowed after sanitization
structured visualization	mermaid, chart, timeline	allowed with complexity limits
navigation	internal links, source refs	host-resolved only
external reference	external links	show domain clearly
input/action	forms, buttons	restricted
privileged UI	login, payment, auth, destructive actions	host-only
raw execution	script, arbitrary HTML, iframe	avoid or sandbox heavily

The LLM should probably be allowed to create explanations, tables, diagrams, and summaries. It should not be able to create authority-bearing UI.

Provenance

If MosAIc is becoming OKF-aware, I would be careful not to let the model mint trusted provenance.

Instead of:

{
  "type": "markdown",
  "content": "This came from docs/foo.md",
  "source": "docs/foo.md"
}

Prefer something like:

{
  "type": "markdown",
  "content": "A generated summary of the OAuth section.",
  "source_refs": ["okf:docs/oauth.md#section-pkce"],
  "generated": true
}

Then the host/loader resolves source_refs into trusted provenance metadata.

That avoids fake citations / fake source badges and makes it clearer which tiles are source-derived, model-generated, user-authored, or host-generated.

Practical roadmap

A staged path might be:

Stage	Goal	Change
v0	keep the prototype simple	keep current overlay merge
v1	make the contract explicit	add schema_version, stricter schema, tile capability table
v2	improve auditability	introduce ops[], diff preview, patch log, undo
v3	improve provenance	host-owned source refs, OKF source resolver, generated/source-derived badges
v4	improve safety	policy gate, destructive-op limits, external-link warnings, no generated privileged UI
v5	improve ecosystem fit	map to/from JSON Patch, MCP resources, AG-UI-style state deltas where useful

The compact design principle would be:

Keep JSON, but turn the model output into a constrained patch proposal. Use typed tesserae as the capability boundary. Let the renderer own validation, provenance, policy, preview, and application.

Terminology / search terms

These terms are not fully standardized, but they are useful for searching nearby work.

Term	Rough meaning	Useful links
Generative UI	model/agent returns UI, not only text	Vercel AI SDK, OpenUI
Declarative generative UI	model emits a declarative spec consumed by renderer	json-render, OpenUI, A2UI
Bounded generation	model generates within a vetted component/template set	Portal UX Agent
Typed composition	UI represented as typed component/template specs	Portal UX Agent
Structured outputs	model output constrained to a schema	HF Structured Outputs
JSON Patch	standard operation list for mutating JSON	RFC 6902
JSON Merge Patch	JSON-shaped partial update format	RFC 7386
State delta	incremental update to shared state	AG-UI State
Provenance-aware UI	UI distinguishes source-derived and generated content	OKF

Nearby projects / papers

Item	Type	Relevance to MosAIc
Vercel AI SDK Generative UI	framework	tool results rendered as custom UI components
json-render	framework	AI generates JSON UI constrained to developer-defined components
OpenUI	declarative UI language	compact LLM-generated UI DSL
A2UI	open project/spec direction	agent-driven declarative UI
OpenAI Apps SDK UI	app/widget layer	structured tool result rendered into ChatGPT component UI
AG-UI	agent/frontend protocol	events, state snapshots, JSON Patch state deltas
Portal UX Agent	paper	bounded generation, deterministic renderer, typed composition
PatchBoard	paper	validated JSON Patch mutations over shared structured state
OKF	knowledge format	markdown+YAML knowledge bundles for agents/tools
OWASP LLM Top 10	security guidance	prompt injection, insecure output handling, excessive agency
NCSC prompt injection note	security guidance	data/instruction separation cannot be assumed inside LLM prompts

Wire format comparison

Format	Good for	Less good for	Fit for MosAIc
JSON	validation, schema, patching, machine handling	long human-authored prose	best runtime format
YAML	human-readable metadata/config	strict model output, ambiguous parsing edge cases	good for frontmatter / OKF-like bundles
Markdown	prose, notes, long content	outer mutation protocol	good tile content format
XML-ish tags	mixed text/structure, some streaming patterns	verbosity, parser edge cases	possible, but probably not needed
Tool calls	host-owned capabilities	whole surface representation	useful around MosAIc, not necessarily inside it
JSON Schema	constrained output	semantic correctness	strong base layer

Possible operation vocabulary

A small domain-specific patch format could start with only a few operations.

Operation	Meaning
add_view	create a new view
rename_view	change a view title
remove_view	remove a view, probably restricted
add_tile	add a tile to a view
replace_tile	replace a tile
remove_tile	remove a tile, probably previewed
move_tile	move/reorder a tile
set_active_view	request navigation after patch
attach_source_ref	request a host-resolved source reference
summarize_tile	compact existing tile content

Example:

{
  "schema_version": "mosaic.patch.v1",
  "ops": [
    {
      "op": "add_view",
      "id": "format_notes",
      "title": "Format Notes"
    },
    {
      "op": "add_tile",
      "view": "format_notes",
      "tile_id": "json_schema_note",
      "tile": {
        "type": "markdown",
        "content": "JSON Schema is a good fit for constraining runtime model output."
      }
    }
  ]
}

Policy rules / safety checklist

Examples of simple but useful policy rules:

Policy	Rationale
max 2 new views per patch	avoids view explosion
max 8 new tiles per patch	avoids surface bloat
no generated login/payment/auth components	prevents fake authority
no raw HTML by default	reduces XSS/sanitization risk
external links must show domain	improves user trust
generated tiles cannot create trusted provenance	prevents fake source badges
destructive ops require preview	prevents silent data loss
source-derived tiles are read-only by default	protects imported knowledge
model cannot edit system/host UI	preserves authority boundary
remove operations are quota-limited	avoids accidental context loss
source refs are resolved by host	prevents model-minted provenance

Relevant security vocabulary:

• OWASP LLM Top 10
• prompt injection
• insecure output handling
• excessive agency
• untrusted external content
• fake provenance
• authority-bearing UI
• renderer/host as trust boundary

Context management sketch

Once surfaces grow, I would avoid sending the whole surface every turn.

Instead, maybe send a compact surface context:

{
  "active_view": {
    "id": "auth_pkce",
    "title": "PKCE",
    "tile_summaries": [
      {
        "tile_id": "pkce_summary",
        "type": "markdown",
        "summary": "Explains PKCE at a high level."
      },
      {
        "tile_id": "pkce_steps",
        "type": "table",
        "summary": "Shows the PKCE flow steps."
      }
    ]
  },
  "surface_index": [
    { "view": "overview", "title": "Overview" },
    { "view": "auth_pkce", "title": "PKCE" },
    { "view": "glossary", "title": "Glossary" }
  ],
  "available_sources": [
    {
      "source_id": "okf:docs/oauth.md#section-pkce",
      "title": "OAuth / PKCE"
    }
  ],
  "allowed_ops": ["add_tile", "replace_tile", "add_view"],
  "forbidden_ops": ["delete_view", "external_action"]
}

One-line summary

MosAIc could be framed as:

A lightweight, bounded generative UI surface where LLMs emit schema-validated, auditable patch proposals over typed tesserae, while the renderer remains the authority for validation, provenance, policy, and application.

[vonargo]

Thank you for this. It’s the most useful outside read MosAIc has gotten, and that you chose to dig in this far is the part I appreciate most.

I’m adopting your framing wholesale: “bounded generative UI over a typed surface, where the model proposes auditable changes and the renderer stays the authority.”

You’ve stated the thesis better than I had it.

Taking immediately:

• schema_version on the model output. Overdue, and you’re right that implicit merge semantics get dangerous as the surface grows.

• Host-resolved provenance (this is the one I owe you for).

  • You named the principle I was only half-enforcing. Neither the model nor the renderer may mint a source badge, only the host. MosAIc computes provenance host-side, but the overlay validator was preserving a model-supplied provenance block, and the model is shown the current surface (host badges included), so a generated tile could mirror a “sourced” badge onto its own content. It was never triggered, but “never triggered” isn’t “nonzero”
  • Fixed and tested: model output is stripped of that block at the trust boundary before it merges, so only the host can stamp a badge. Rolling it out now.

One place I’d flag a difference rather than just adopt: you have it as “the renderer stays the authority.”

I want authority to move off the renderer and onto the server, with the renderer dropping to a presenter that also can’t mint truth, and the wire contract between them as the real trust boundary. It’s your principle. I’m just moving it a bit.

What I’m deliberately not building yet: the full ops patch protocol, diff-preview, undo. My next milestone is keeping a person in continuous flow on a narrow loop, and a general patch/undo system is where that quietly turns into a general notebook app I didn’t set out to build. ops earns its place the moment each mutation needs to be individually loggable and rejectable.

More of a when, not an if.

Your sourcelist sent me reading!

PatchBoard (Zhang et al., arXiv 2605.29313) is the closest neighbor by a distance: it’s the multi-agent generalization of this exact idea, where workers propose validated JSON-Patch mutations against a shared schema and a deterministic kernel checks role-specific write contracts and commits transactionally with a replayable log. That’s “the host disposes” turned into a coordination substrate, and their ALFWorld numbers (about 85% success against 31% for a LangGraph baseline) are a strong vote for the typed patch + deterministic validation shape. AG-UI’s state deltas are the other one I’m still working through…

Thanks again. This is the kind of response that makes working in the open worth it.

[vonargo]

MosAIc 2.0 is up: the surface now shows you what relates to what you’re reading.

The new thing is the subject rail. For whichever view is active, everything else on the surface (other views, loaded OKF concepts) gets listed in the sidebar ranked by relatedness, strongest first, each row a click-through. Ranking v1 is deliberately simple: client-side token-set overlap,
instant, zero tokens, works signed out. The part I actually care about is the contract, not the scorer: the rail renders {scorer, ranked:[{kind, ref, title, score}]}, so the scorer can improve forever without the UI changing.

Signed in, there’s a small “rerank” button: one LLM call re-scores the same rows, billed to your own account like the typed tasks. The trust rule from the provenance work carries over: the model can only re-score rows the rail already computed. It can never add, remove, or retitle one -
unknown refs are dropped, scores clamped, and a bad response just keeps the overlap ranking.

The tiles also got hands: drag to reorder, corner-resize, collapse to a title bar, all persisted across reloads, including OKF concept views (full-row by default so they read like a column, but drag one to half width and two concepts sit side by side). Sticky table of contents on bigger panels. And a retheme.

Same architecture as before: static site, no backend, MIT. MosAIc Link

Open question I’m chewing on: the overlap scorer treats titles and tags as flat token sets. Has anyone found a cheap client-side ranking trick that respects structure (headings > body, tags > prose) without shipping an embedding model to the browser?

[John6666]

Nice! From what I can see, I think the design principles have translated into the feature really well. For now, on the scorer and related bits:

---

For the specific question — a cheap client-side ranking method that respects structure without shipping an embedding model — I think the simplest good answer is field-weighted lexical ranking.

Something roughly like:

title       × 4
tags        × 3
headings    × 2
tile titles × 2
body        × 1

Then add document-frequency/IDF weighting so that rare, discriminative terms matter more than words appearing everywhere.

That is essentially a small BM25F-style scorer: ordinary lexical retrieval, but with separate weights for structured fields. For the current size of MosAIc, I would probably try that before adding a larger dependency.

A useful implementation detail is that both sides are structured:

• the active view has a title, tags, headings, and body;
• each candidate view/concept has the same kinds of fields.

Most search libraries primarily boost fields on the candidate-document side. One simple way to preserve the active side’s structure too is to run several weighted searches and combine them:

final score =
    4 × search(active.title)
  + 3 × search(active.tags)
  + 2 × search(active.headings)
  + 1 × search(active.body)

The search index can independently boost candidate title/tags/headings/body. It is not mathematically pure BM25F, but it is easy to inspect, tune, and explain.

Existing parts worth considering

My rough shortlist would be:

Option	Best fit	Relevant features	Trade-off
MiniSearch	Minimal browser-side full-text search	field boosting, fuzzy/prefix search, dynamic add/remove, zero dependencies	multilingual tokenization may need customization
Orama	More complete browser-side search	BM25 by default, field boosting, typo tolerance, filters, language-specific support including Japanese	probably more machinery than the rail currently needs
Fuse.js Token Search	Fuzzy matching and typo tolerance	weighted keys, token-level fuzzy matching, BM25-style IDF, Unicode-aware tokenizer	it is still fuzzy-search-oriented rather than a conventional full-text engine
wink-bm25-text-search	Explicit, inspectable ranking	BM25, numerical field weights, separate text-preparation pipelines	lower-level; tokenization and normalization remain application concerns
FlexSearch	Larger or performance-sensitive browser indexes	multi-field search, workers, persistence options, partial matching	more configuration and a less transparent scoring surface
Lunr	Mature traditional browser search	fields, boosts, pipelines, language plugins	older design; less compelling than MiniSearch/Orama for a new small feature
Pagefind	Build-time search for static document sites	multilingual indexing and section weighting	not a natural fit for runtime-loaded OKF bundles and dynamically changing views

My practical recommendation would be:

1. Keep the custom scorer first, because the corpus is small and the ranking contract is intentionally simple.
2. Add field weighting and IDF-like rare-token weighting.
3. If owning the scorer becomes distracting, try MiniSearch as the smallest conventional replacement.
4. Consider Orama if multilingual search, larger bundles, filtering, or persistence become first-class requirements.
5. Consider Fuse.js Token Search if typo tolerance is more important than conventional lexical ranking.
6. Consider wink-bm25-text-search if keeping the weighting formula explicit and inspectable is the priority.

I would probably avoid embeddings until there is evidence that lexical recall is the limiting factor. For navigation within one bundle/surface, titles, tags, headings, IDs, and repeated domain terms often provide a surprisingly strong lexical signal.

The reranking contract looks good

The local scorer producing:

{
  "scorer": "token-overlap-v1",
  "ranked": [
    {
      "kind": "view",
      "ref": "oauth",
      "title": "OAuth",
      "score": 0.82
    }
  ]
}

and the LLM being allowed to modify only the scores of known refs is a clean boundary.

It keeps the responsibilities separate:

• local retrieval controls candidate identity and recall;
• optional LLM reranking improves ordering;
• the model cannot mint links, titles, concepts, or authority;
• malformed output degrades to the local result.

That is a nice concrete application of the earlier “model proposes, host disposes” principle.

One small tokenizer caveat

If non-English or acronym-heavy material is in scope, I would revisit an ASCII-only tokenizer such as:

/[a-z0-9]{3,}/g

It excludes:

• Japanese, Chinese, Korean, Cyrillic, Arabic, etc.;
• accented Latin text;
• common two-character technical terms such as AI, UI, UX, HF, JS, DB, and OS.

If the intended corpus is strictly English prose, the multilingual part may not matter, although the acronym case probably still does.

A lightweight improvement is:

• a Unicode-aware fallback regex;
• Intl.Segmenter when available for languages without whitespace-delimited words;
• application-specific preservation of technical tokens such as C++, node.js, OAuth2, paths, IDs, and hashtags.

Fuse.js’s token-search documentation is a useful reference here: its default tokenizer accepts Unicode letters/numbers, but it recommends Intl.Segmenter for actual CJK/Thai word segmentation.

So the compact version of my answer is:

Start with a field-weighted lexical scorer, add IDF-like weighting, and keep the current host-owned candidate contract. MiniSearch is probably the closest small drop-in; Orama is the fuller multilingual/BM25 option; Fuse.js Token Search is interesting when fuzzy matching matters; wink-bm25-text-search is attractive when explicit scoring control matters.

Possible minimal scorer sketch

This is intentionally simple rather than a full search-engine implementation:

const FIELD_WEIGHTS = {
  title: 4,
  tags: 3,
  headings: 2,
  tileTitles: 2,
  body: 1
};

function weightedOverlap(active, candidate, documentFrequency, documentCount) {
  let score = 0;
  let possible = 0;

  for (const [field, fieldWeight] of Object.entries(FIELD_WEIGHTS)) {
    const queryTerms = new Set(active[field] ?? []);
    const candidateTerms = new Set(candidate[field] ?? []);

    for (const term of queryTerms) {
      const idf = Math.log(
        1 + (documentCount + 1) / ((documentFrequency.get(term) ?? 0) + 1)
      );

      possible += fieldWeight * idf;

      if (candidateTerms.has(term)) {
        score += fieldWeight * idf;
      }
    }
  }

  return possible > 0 ? score / possible : 0;
}

Possible refinements, only if needed:

• separate weights for active fields and candidate fields;
• exact title-match bonus;
• tag-to-tag bonus;
• heading-to-title bonus;
• length normalization for body fields;
• penalties for generic/very frequent terms;
• small fuzzy bonus for typos;
• stable tie-breaker using the original local order.

More detailed library notes

MiniSearch

MiniSearch is probably the most straightforward small dependency here.

It:

• runs in the browser;
• has no external dependencies;
• indexes selected document fields;
• supports per-field boosts;
• supports prefix and fuzzy matching;
• allows documents to be added and removed dynamically.

That last property fits runtime-loaded views and OKF concepts better than a build-time static index.

Orama

Orama is the most complete option in this group.

Relevant features include:

• browser-side operation;
• BM25 as the default lexical scoring algorithm;
• field boosting;
• typo tolerance, filtering, facets, and persistence-related plugins;
• language-specific tokenizers and explicit documentation for Japanese and Chinese.

It may be excessive for a few dozen rail entries, but it becomes more attractive if the rail evolves into general OKF/surface search.

Fuse.js Token Search

Fuse.js Token Search is newer and more relevant than traditional single-pattern Fuse usage.

It:

• splits a multi-word query into tokens;
• fuzzy-matches each token independently;
• applies BM25-style IDF weighting;
• supports weighted keys;
• uses a Unicode-aware default tokenizer;
• allows a custom tokenizer, including one based on Intl.Segmenter.

It seems particularly suitable if users will type imperfect queries. For automatic view-to-view relatedness, its fuzzy behavior may be unnecessary or should be kept conservative.

wink-bm25-text-search

wink-bm25-text-search maps almost literally onto the question.

It supports:

• BM25 ranking;
• numerical weights for fields;
• independent text-preparation pipelines for document fields and queries;
• browser and Node usage.

It is less of a ready-made UI search package and more of a ranking primitive, which may actually be desirable if transparency matters.

FlexSearch

FlexSearch is worth knowing about if the corpus or performance requirements grow.

It supports:

• browser and Node operation;
• multi-field/document search;
• workers;
• partial matching;
• storage/persistence configurations.

I would benchmark it rather than assume it is necessary: the current rail is small enough that clarity may matter more than throughput.

Lunr and Pagefind

Lunr remains a mature browser-side inverted-index library with fields, boosts, pipelines, and language plugins.

Pagefind supports multilingual static-site search and section weighting, but its index is normally produced from built HTML. That is excellent for documentation sites, but less aligned with MosAIc’s runtime-loaded and mutable surface.

Tokenization sketch

A browser-native tokenizer can use Intl.Segmenter where available and fall back to a Unicode regex:

function tokenize(text, locale = undefined) {
  const normalized = String(text ?? '').normalize('NFKC').toLocaleLowerCase(locale);

  if (typeof Intl?.Segmenter === 'function') {
    const segmenter = new Intl.Segmenter(locale, { granularity: 'word' });

    return [...segmenter.segment(normalized)]
      .filter((part) => part.isWordLike)
      .map((part) => part.segment);
  }

  return normalized.match(/[\p{L}\p{M}\p{N}_]+/gu) ?? [];
}

Depending on the corpus, it may also be worth preserving technical tokens before segmentation:

C++
C#
node.js
OAuth2
RFC-6902
foo/bar.md
#provenance

A generic natural-language tokenizer may otherwise split or discard exactly the identifiers that distinguish concepts within a technical knowledge bundle.

Search terms for further investigation

• field-weighted full-text search
• BM25F structured document ranking
• browser-side BM25
• lexical reranking
• client-side inverted index
• field boosting
• weighted keys
• Unicode tokenization
• CJK word segmentation
• Intl.Segmenter
• query likelihood vs document field weighting
• candidate generation and reranking
• lexical retrieval followed by LLM reranking

[vonargo]

FIrst, I must once again thank you for the incredible engagement. Your posts really help improve the work. Doubly thanks for the survey!

Adopting the core recommendation as scorer v2: field-weighted lexical with IDF. Your point that BOTH sides are structured is the bit I hadn’t thought through - I was only going to boost candidate fields, and running weighted queries per active-side field and combining them is cleaner and stays inspectable. The contract makes this a drop-in: token-overlap-v1 becomes field-weighted-idf-v2 and the rail never knows. Keeping it custom for now for exactly the reason you give (the corpus is a few dozen entries and I want to be able to read my own ranking math), with MiniSearch as the named escape hatch if owning it starts to distract.

Full concession on the tokenizer. The [a-z0-9]{3,} regex was me being lazy, and the acronym case bites today - AI, UI, HF, OS are in half my test bundles and they’re all invisible to the scorer. And non-Latin isn’t hypothetical for me either: some of the material I want on this surface is Cyrillic. Taking your sketch roughly as-is: NFKC + Intl.Segmenter with the Unicode-regex fallback, plus preserving technical tokens before segmentation (node.js, OAuth2, paths) since those are exactly the terms that distinguish concepts in a technical bundle.

One data point in favor of the contract you called clean: it already survived a scorer swap in the wild. On my governed fork of this engine I run the same rail against a local embedding model, and moving between that and the overlap scorer changed nothing but the scorer label - “swap the scorer, never repaint” held in practice, not just in the design doc. That said, your “avoid embeddings until lexical recall is the limiting factor” matches what I saw: on a small surface the lexical signal from titles and tags was most of the value, and the embeddings only started earning their keep as the corpus grew.

“Model proposes, host disposes” is going in the docs, credited to you.

v2 (field weights + IDF + the tokenizer fix) will land in a point release - I’ll note it here when it does.

[vonargo]

Scorer v2 is field-weighted lexical with IDF per your outline: both sides structured (title x4 / tags x3 / headings x2 / body x1), BM25-flavored IDF so rare terms win, cosine-normalized - roughly 60 commented lines, zero dependencies. MiniSearch stays the named escape hatch if owning it ever distracts.

Tokenizer fixed too: NFKC, technical tokens preserved whole (node.js, OAuth2, paths), then Intl.Segmenter with a Unicode-regex fallback. AI/UI/HF/OS score now, and Cyrillic tokenizes instead of vanishing.

Two things you might find satisfying:

• The contract held again in the wild: overlap v1 → lexical v2 and the rail changed nothing but its scorer label. That’s three scorer swaps now with zero repaints.
• v2 immediately proved itself on the demo bundle - the top rail row changed from a coincidental panel to the actually-related concept.

One honest judgment call to disclose: real cosine-IDF scores run small (~0.03), which made the strength bars nearly invisible. Bars are now rank-relative (top row = full bar) with the raw score in the tooltip so the display scales and the tooltip tells the truth.

Your scorer design and “model proposes, host disposes” are credited in the README with a link back to this thread. Most useful review this project has had - thank you.