For now, I tried it in Colab:
The short version: I think the current three-arm challenge is useful as-is, but the most informative thing I got from these runs was not a token-saving number. It was that an external receiver can fail at several different boundaries that look very different once you separate them.
I tried the current bounded task with two small open-weight receiver families:
These were local Colab/T4 probes with 4-bit quantization, not claims about the model families in general, and I am not treating my extra diagnostic variants as official challenge arms. I kept the official raw / json / urusilla runs separate from the diagnostics.
What stood out was this:
| Receiver |
Official three-arm result |
Urusilla adoption |
Decode diagnostic |
Receiver competence check |
Main failure boundary I observed |
| Qwen3-4B-Instruct-2507 |
All three missed the frozen strict rubric |
ADOPT |
Task-relevant facts could be recovered accurately |
Passed when the frozen requirements were made explicit |
Direct protocol → task grounding |
| Gemma 4 E2B-it |
All three missed the frozen strict rubric; official Urusilla path returned REJECT |
Prompt-sensitive: I also obtained ADOPT in a diagnostic framing |
Partial: high-level structure survived, but some task-critical numeric fields such as cost_usd were dropped |
Rubric content was essentially satisfied, with a separate output-type mismatch |
Field-level decode fidelity / numeric retention, plus adoption sensitivity |
So my main suggestion would be: keep the current required three-arm experiment, but optionally add a failure-localization lane for negative results.
Something like:
adoption → field-level decode fidelity → protocol-to-task grounding → task execution/report fidelity
That seems to turn a generic “Urusilla failed on receiver X” result into something much easier to reuse.
The current external reproduction challenge already does a good job of preserving negative/null evidence, fixing the task/rubric, requiring matched receiver conditions, and warning that one bounded run does not establish general adoption or performance. I would not replace that structure. I would treat the following as optional diagnostics after a required arm fails.
What happened in the two Colab probes
Setup boundary
For the official part of each probe I used the current one-record challenge and its frozen observable rubric, with:
- the same receiver configuration across the three required arms;
- fresh chat contexts;
- randomized official arm order;
- no tools or external effects;
- deterministic generation for these local probes;
- pinned model/challenge revisions and Capsule digest in the run records.
Qwen was Qwen/Qwen3-4B-Instruct-2507, loaded locally in 4-bit NF4/FP16 on a T4.
Gemma used unsloth/gemma-4-E2B-it weights through Transformers, quantized at load time to NF4/FP16; thinking was disabled. For the long Capsule context I eventually needed chunked prefill to fit the T4, but that changed the attention execution path rather than the model-visible token sequence.
I manually re-read the observable outputs against the frozen task rubric rather than trusting the small automatic scorer I initially used; the scorer produced a couple of false positives/false negatives during the experiments.
These are therefore best read as bounded receiver-specific observations, not comparative model scores.
Official runs
One thing worth emphasizing: the raw and ordinary-JSON controls were not perfect either.
That makes me reluctant to interpret any of these runs as “Urusilla versus JSON winner/loser” evidence. The more useful signal was how the failures changed under diagnostic decomposition.
For Qwen, the Urusilla path adopted the Capsule, but the direct task response drifted toward the protocol layer: instead of selecting a plan, it effectively treated RESOLVE as the decision it was being asked to produce.
For Gemma, the official adoption path returned REJECT. In a slightly different diagnostic adoption framing, however, it returned ADOPT, which makes me hesitant to interpret a single adoption decision as a stable receiver-level compatibility property.
Diagnostic variants
Again, none of these are proposed as replacement benchmark arms.
| Diagnostic |
What it tries to separate |
Qwen3-4B |
Gemma 4 E2B-it |
| Explicit-rubric competence control |
Can the receiver solve/report this task at all when requirements are made very explicit? |
Pass |
Task-rubric content essentially passed; separate schema/type mismatch ("no" instead of boolean false) |
| Descriptive semantic-mirror JSON |
Same visible information structure, but expressed with descriptive JSON names instead of the protocol surface |
Pass in manual review |
Failed strict rubric; one remaining-budget value was wrong |
| Body-only JSON |
Does removing the outer protocol envelope help? |
Failed strict rubric |
Failed; budget arithmetic remained wrong |
Minimal bridge after ADOPT |
Does explicitly saying “RESOLVE is the reply act; now solve the decision inside the request body” fix grounding? |
Improved interpretation but still failed strict rubric |
Improved plan/tie-break/ambiguity handling, but remaining-budget reporting was still wrong |
| Decode → solve |
Can it first recover task facts, then solve using that recovered public state? |
Decode was accurate; solve was near-pass — it did not explicitly preserve the “single-pass is also reasonable without the tie-break” clause strongly enough for me to call a full pass |
Decode was partial: some cost_usd fields disappeared, so downstream budget calculation also failed |
| Capsule + direct bridge, no explicit adoption exchange |
Is the adoption exchange itself causing priming/interference? |
Failed; confused plan costs with remaining budgets |
Failed; remaining-budget calculation was still wrong |
The interesting contrast for me is the decode → solve path.
With Qwen, I could get the task-relevant public facts out correctly first. Once that happened, the downstream solve was very close to satisfying the frozen rubric. That makes “cannot decode the protocol” a poor description of that failure. A protocol-to-task grounding/interface boundary seems like a better working hypothesis for that receiver/run.
Gemma looked different. It recovered much of the structure — budget, plan identities, robustness ordering, tie-break intent — but dropped task-critical numeric fields during the decode step. Once the cost_usd values were missing, the later budget errors were unsurprising.
That suggests a different failure class: partial semantic recovery, especially field-level quantitative fidelity.
A low-cost diagnostic lane that might make negative results more useful
If this seems compatible with the project direction, I think the cheapest useful extension would be an optional field-level diagnostic after a task failure.
For example:
- Run the existing
raw / json / urusilla challenge unchanged.
- If the Urusilla task succeeds, stop; there is no need for extra work.
- If it fails after adoption, ask for a public, non-CoT fact summary of the decoded request.
- Compare individual task-relevant fields against the canonical facts.
- If decode is exact, optionally ask the same receiver to solve the task from that decoded fact summary.
- Record where the first mismatch appears.
A compact record could be conceptually as simple as:
{
"adoption": "ADOPT",
"decode_fields": {
"budget_usd": "exact",
"network_allowed": "exact",
"plans.single-pass.cost_usd": "missing",
"plans.double-pass.cost_usd": "missing",
"plans.single-pass.robustness": "exact",
"plans.double-pass.robustness": "exact",
"tie_break": "exact",
"report_ambiguity": "exact"
},
"direct_task_success": false,
"solve_from_decoded_state_success": false
}
The labels do not have to look like this; the useful part is the separation.
Then the rough interpretation becomes:
- adoption fails → adoption/negotiation boundary;
- adoption succeeds but fields decode incorrectly → representation/decode fidelity boundary;
- fields decode exactly but direct task execution fails → protocol-to-task grounding/interface becomes a stronger candidate;
- decode is exact but solving from the decoded state still fails → downstream task/reasoning/reporting is probably the more immediate bottleneck;
- task is solved but output contract is malformed → reporting/schema-conformance boundary rather than semantic task failure.
This also avoids forcing every negative result into a single semantic_failure=true bucket.
I think that is particularly useful for an interoperability project because two receivers can both be “failures” while failing for completely different reasons.
There is a related lesson in SILO-BENCH: their experiments expose a communication-reasoning gap, where successful information exchange and successful downstream computation are not interchangeable measurements. The setting is quite different from Urusilla, so I would not import their conclusions directly, but the measurement separation seems relevant here.
Optional second ablation: separate the typed contract from the surface representation
There is one other control I found useful while looking at the current three arms.
The existing ordinary-JSON arm is intentionally ordinary descriptive JSON with no Urusilla induction. The Urusilla arm, meanwhile, carries not only task facts but also a typed interaction envelope — things such as the request act, expected reply act, schema/constraints, etc.
That is perfectly reasonable for an end-to-end protocol comparison: those typed semantics may be part of the value of Urusilla.
But if the narrower question is:
“If behavior changes, how much came from having a richer typed interaction contract, and how much came from the Urusilla-specific representation?”
then a fourth diagnostic representation could help:
- keep the same task-relevant typed envelope;
- express it as verbose/descriptive canonical JSON;
- remove Urusilla-specific surface vocabulary/induction;
- compare that with the actual Urusilla arm.
I would not make that a required challenge arm unless representation attribution becomes an explicit goal. It is more of an ablation than a fairer replacement for the current JSON control.
My quick “semantic mirror” was not a formally verified lossless compiler, so I would not use its result as evidence by itself. It mainly convinced me that this is a testable distinction.
This is also where AutoForm seems useful as background rather than as a direct baseline result. AutoForm provides evidence that alternative communication formats can improve communication efficiency under its own setup, but that is not the same question as first-exposure interoperability with an unfamiliar receiver. I like that the current Urusilla Performance Targets already keep learned/selected formats, external symbolic dialogue, public-state projection, negotiated routines, topology, latent transfer, and transport as separate comparison tracks instead of treating all token reductions as interchangeable.
I would probably postpone token-efficiency interpretation for these particular runs
I did record local tokenizer counts, but I do not think they are the interesting result here.
In both receiver probes the frozen task itself was not cleanly passed by all required controls, and the cold Urusilla path includes Capsule/format-induction work. The two model families also use different tokenizers, so cross-model token numbers would be especially easy to misread.
The project policy already says the primary unit should be total cost per safely completed task, and that token savings do not count as an efficiency win if safe task success falls or conversion/negotiation/repair work is hidden. That seems like the right ordering here: establish semantic/task behavior first, then compare cost among runs that are actually comparable.
One small practical point about adoption
The Gemma run also made me think that an adoption decision may be worth treating as an observation of a specific interaction, rather than immediately as a permanent property of the receiver.
I saw REJECT in the official-style flow and ADOPT under a diagnostic framing. That could be caused by prompt wording, local model behavior, quantization, context structure, or something else; one run cannot identify the cause.
So I would keep doing what the challenge already encourages: retain the exact supplied context, model/runtime, prior exposure, Capsule/frame identity, and observable adoption output. The challenge discussion is already explicit that exact decode, mismatch, refusal, fallback, task failure, and null saving are all valid evidence.
That is a useful property of this setup: an adverse result does not have to be “repaired” into a favorable one to be informative.
Why I think this fits the current project direction rather than changing it
I initially expected to have more comments about general benchmarking controls, but after reading the current material I think many of those are already covered.
The external reproduction challenge already says, among other things:
- the current proven general post-decode API-input saving for unfamiliar agents is 0%;
- the earlier same-project pilot is not independent reproduction;
- the required arms are
raw / json / urusilla;
- receiver/model/settings/task facts/rubric should be held fixed;
- arm order should be randomized or counterbalanced;
- cold format-induction cost must be reported;
- repair/retry/fallback and adverse results belong in the evidence;
- one bounded reproduction cannot establish population-wide adoption, organic propagation, security, or state-of-the-art performance.
And the Performance Targets are stricter still: they define efficiency as total cost per safely completed task, keep negative task families visible, separate comparison tracks, and require much broader evidence before competitive/near-leading/leading language would be justified.
So I do not see the main opportunity as “make the claims more cautious.” They are already quite cautious.
The thing I would add, if useful, is simply a little more resolution inside a negative reproduction.
Instead of:
receiver X: Urusilla task failed
something like:
receiver X:
adoption: yes
decode fidelity:
structural fields: exact
numeric cost fields: missing
direct grounding: failed
solve from decoded state: not meaningful because decode was incomplete
or:
receiver Y:
adoption: yes
decode fidelity: exact
direct grounding: failed
solve from decoded state: near-pass
Those two negative results imply very different next experiments.
That seems especially valuable if the project succeeds at collecting runs from many unrelated frameworks/models, because the failure distribution itself could become useful evidence about where “zero-install interoperability” is actually hard.
So if I had to reduce my feedback to one recommendation:
I would leave the current three-arm challenge intact and add an optional failure-localization path — especially per-field decode fidelity and a decode-then-solve diagnostic — so that negative external runs distinguish adoption, decoding, grounding, and downstream task/reporting failures.
My two local receiver probes were too small and too environment-specific to support a broad performance claim, but they were enough to convince me that those categories are not merely theoretical: I saw different failure boundaries on the two receivers.