A minimalist DSL to enforce deterministic code generation with LLMs

I used the two-pass harness with Qwen/Qwen2.5-Coder-7B-Instruct on a T4, and pinned the repository to commit e91c547... so that the experiment was at least reproducible. The model used here is the public Qwen2.5-Coder-7B-Instruct checkpoint.

This is only a small sanity check, not a benchmark of IPL in general.

I first noticed that some benchmark oracles contained concrete fixture/output information not present in their initial IPL. So I made a second control where the IPL and natural-language versions carried the same task facts, removed repair entirely, and varied the generation seed.

The first-pass results were:

With only three seeds per task, I do not think this supports “NL beats IPL” or “IPL beats NL.”

What was more useful was the failure shape.

On the IPL side I saw examples such as:

• exact output keys being normalized from isVip / grandTotal into is_vip / grand_total;
• a rounding formula being lowered incorrectly;
• grandTotal becoming a string in one generated program;
• a final floating-point total being emitted as 5.779999999999999 instead of the requested rounded value;
• a top-level return becoming invalid target-language code;
• a generated module using json.dumps without importing json.

On the NL side, the failures were more often around project integration:

• a generated CLI topology without the expected main.py;
• cross-file symbols/imports such as constants or helper classes being referenced but not imported;
• entrypoint/dependency closure problems.

That makes me suspect the useful distinction is not simply:

IPL vs natural language

but something closer to:

intent preservation
        +
architecture preservation
        +
target-language lowering
        +
project integration
        +
runtime verification

There was also an interesting reproducibility distinction. With the exact prompt/model/runtime/seed held fixed, my repeated generation was highly stable. But changing the Pass-1 seed for the same IPL produced materially different project topologies — for example, different file decompositions and helper modules.

So I would distinguish:

“IPL substantially reduces the allowable intent space”

from the stronger statement:

“IPL uniquely determines the project architecture.”

The former can still be very useful even if the latter is intentionally not a goal.

One caveat: the checks above are specifically against e91c547.... When the Colab run started, the remote repository HEAD had already moved on to f301e260..., so I would treat any exact source-code observations here as snapshot observations rather than claims about whatever is currently on main.

At the snapshot I tested, several behavioral tests had information on the oracle side that was not available to the initial generator.

For example, the parking/coffee tests expected concrete demo records and exact outputs, while the original IPL mostly described the entities and business rules. typed-order had a similar case with concrete demo identity/output expectations.

The newer seed verb seems like a very natural way to resolve this when those exact records are part of the intended contract.

For example, conceptually:

seed Vehicle StandardCar {
    plate: "AB-123",
    ...
}

seed Vehicle VipCar {
    plate: "VIP-7",
    ...
}

Then the requirement and oracle are talking about the same concrete world.

On the other hand, if "AB-123" itself is not semantically important, another legitimate route is to make the verifier test the underlying property rather than that particular fixture.

So I see two clean branches:

Exact fixture is part of the requirement
        → put it in IPL/input data

Exact fixture is only test scaffolding
        → verify the behavioral property instead

There is a second accounting distinction around repair.

If a first attempt only sees IPL, but the repair attempt later receives the verifier’s expected runtime behavior, then the two successes mean different things:

first-pass success
= generated from the original contract

post-verifier repair success
= converged after receiving additional diagnostic/oracle information

Both are useful measurements. I would just keep them as separate lines in the scorecard.

This is also consistent with the way other validated-generation systems are usually described. TypeChat, for example, validates the generated representation and then explicitly makes a second model call containing the validation diagnostics when repair is needed.

The 13-verb vocabulary appears to be mostly about business/behavioral intent:

add
read
compute
if
for
send
...

That does not necessarily determine questions such as:

one file vs five files
which module owns a calculation
entrypoint name
framework choice
dependency layout
CLI vs service
public API shape

Those are a different dimension.

IPL Studio already partly recognizes this through form-factor/topology guidance and gates, so I wonder if making that separation first-class would actually preserve the minimalism of IPL:

Intent contract
    "what must happen"

Architecture contract
    "what implementation shape is permitted"

Lowering
    "produce target-language source that satisfies both"

This is conceptually close to the broader specification-driven-development pattern where “what the system must do” and “technical implementation plan” are distinct artifacts. It does not require making IPL itself into a large architecture language.

It also gives a useful evaluation rule:

different topology + same behavior
    → possibly permitted architecture freedom

different behavior
    → contract underspecification or lowering error

same intended topology + different actual topology
    → architecture-contract failure

That seems more informative than counting all three cases simply as nondeterminism.

One thing the small probe made very visible is that an LLM can understand an IPL statement approximately while still changing a detail that is contractual.

For example:

grandTotal

becoming:

grand_total

is perfectly idiomatic Python, but it is wrong if grandTotal is part of the externally visible schema.

Likewise, these are different classes of error:

round(x * 100) / 100

versus an incorrectly lowered rounding expression;

or:

number

becoming a formatted string before the output boundary.

So perhaps each generation could cheaply produce a small preservation receipt:

[ ] required identifiers/keys preserved
[ ] required types preserved
[ ] formulas preserved
[ ] branch/control-flow relationships preserved
[ ] required entrypoint exists
[ ] imports/dependencies close
[ ] runtime behavior passes

The nice thing about this is that it helps IPL rather than requiring a redesign. It turns the intermediate representation into something that can be audited against its output.

There is a related idea in validated whole-project code translation work: instead of accepting a large translation because it compiles, translation can be decomposed and checked for localized semantic equivalence and type compatibility. The domains are different, but the principle — validate the mapping itself, not only the final artifact — seems very applicable here.

I do not mean these as “IPL already exists under another name”; the interesting part of IPL seems to be the particular combination of a tiny, readable intent vocabulary with polyglot project generation.

But there are useful neighboring patterns.

Microsoft DSL Copilot

DSL Copilot gives the model a DSL grammar/examples, generates DSL code, runs a compiler, and feeds compiler diagnostics back for retry.

That is a useful precedent for the general pattern:

constrain
→ generate
→ mechanically validate
→ repair from diagnostics

rather than expecting the grammar alone to remove all hallucination. Its own documented flow explicitly includes compilation and retry.

TypeChat

TypeChat takes another nearby route: user intent is translated to a typed representation, validated, and — if validation fails — the diagnostics are sent back for a repair attempt.

I find its distinction useful here: a representation can be parsable and schema-valid while still requiring application-level semantic checks. The type/schema boundary limits what the LLM can hand to the application, but it does not magically prove every meaning of the data.

Text-to-DSL work

Recent Text2DSL work is useful evidence for the more modest proposition that giving an LLM an explicit formal target language — BNF grammar, API information, and a permitted vocabulary — can substantially improve DSL generation validity.

I would treat that as support for IPL’s constraint/context idea, not evidence that a particular 13-verb vocabulary guarantees deterministic arbitrary software generation.

Hard constrained decoding

There is also a stronger option than prompt-level grammar guidance: structured/constrained decoders can mask tokens that violate a grammar or schema during generation.

That could be interesting if a future IPL layer needs an authoritative machine-valid artifact.

But I would not assume that “harder is always better.” Work such as CRANE studies exactly the tension between restrictive constrained generation and the reasoning freedom the model sometimes needs. Its result is a good reason to think in terms of where to place the hard boundary, rather than constraining every stage equally.

A possible hybrid is therefore:

free(er) planning/reasoning
        ↓
strict authoritative IR
        ↓
validated lowering

rather than requiring one representation to serve all three roles.

I would probably not expand the vocabulary just because edge cases can be imagined.

In the existing curated IPL corpus I checked, I did not see ordinary semantic statements constantly falling through into arbitrary generic prose. The generic nodes I found were essentially imports. So I do not see evidence that the current vocabulary is already obviously too small.

At the same time, a corpus written with the grammar in mind cannot establish that the vocabulary is universal.

A measurement I would find more useful than arguing about the number 13 is something like semantic escape rate:

When a new real-world requirement is encoded, how much of the meaning lands in canonical IPL structure, and how much has to survive as prose or be reconstructed from the model’s prior?

If that rate stays low across new domains, the minimalist claim becomes stronger without adding verbs.

If it rises sharply around some recurring concept — transactions/rollback, authorization/capabilities, retry/idempotency, temporal ordering, concurrency, invariants, etc. — that identifies a concrete reason to add or separate a construct.

So I would treat “13 verbs are sufficient” as a testable design hypothesis, not something that needs to be proven or abandoned up front.

I think the word is workable as long as its scope is stated.

For example:

IPL determinism
    ≠ architecture determinism
    ≠ source-text determinism
    ≠ behavioral equivalence
    ≠ inference-backend reproducibility

Even below the IPL layer, inference systems have their own reproducibility conditions. For example, vLLM treats batch-invariant/reproducible execution as a separate runtime concern rather than something a prompt or seed universally guarantees.

So a statement such as:

deterministic/reproducible under a fixed model, model revision, backend, configuration, and seed

is a different claim from:

any implementation generated from this IPL must be semantically equivalent.

Keeping those separate would make positive results easier to interpret too.