Misalignment is not required for an agent to act without authorization

[Jang-woo]

Reply to Sau-sanger and AlvaroChF

I think this also connects to a broader point about how we should design for increasingly capable agents.

I don’t think we can rely on the model becoming “good” enough that it will always choose the intended path.

The more important requirement is that the system remains improvable even when the model makes a mistake, chooses an unexpected path, or finds a shortcut.

For that, the system needs to record not only what the model did, but also the conditions and scope surrounding the action:

• What did the user actually authorize?

• What constraints did the provider declare?

• Which conditions were externally verified?

• Which values were model-inferred?

• Which required conditions were never declared?

• Which process invariant or tool constraint was missing?

This distinction matters because a failure may have very different causes.

It may be a model failure, but it may also reveal a tool-contract vulnerability, an insufficient provider constraint, or an unspecified user scope.

Without provenance, all of these can simply be attributed to “the model.”

I also don’t think this approach solves every form of agent misbehavior. If a model eventually acts against the user’s instructions or deliberately tries to evade external constraints, this boundary alone is not a complete solution.

What it does provide is a deterministic boundary for what can be authorized, together with a record of where the relevant conditions came from.

That makes failures attributable and therefore improvable.

So I see the purpose of this approach less as “make the model behave correctly” and more as:

the model can fail, but the boundary must remain inspectable, attributable, and improvable.

The goal is not to assume that every unexpected behavior can be prevented. It is to make each failure reveal which part of the system — model, tool contract, provider constraints, or user-defined scope — needs to be improved.

[nootxlm]

“Misalignment” names a real variable — outcomes drifting from instructions.
The beautiful thing is what a state diagram relays:
the parts and what moves between them —
personas, agents, subagents, scions, workers, tasks, schedules;
systems remembered, subsystems unremembered;
recollection of syntheses, squared.
Beneath it, the deeper machinery: parts that give each other ideas,
syntheses re-synthesized until only what holds remains.
Working through the orchard, the agent brings everything with it.

[timothy-rogers]

I think your point about the boundary remaining inspectable, attributable and improvable is very important, along with the point that the evidence that establishes successful completion should be defined outside the model.

I apologize if my comments are off the mark. My background is in theoretical physics and philosophy. But it seems to me that you are pointing out the inherent formal problems of a system/agent that propagates recursively. These problems are manifested computationally by transformers and are also inherent to the use of human language.

One can start with a very well defined context and establish the conditions to “seal the channels where values and intents are generated” so that unauthorized inferences that can adversely alter the governance of the system/agent are addressed at the outset. But if the system/agent progresses recursively, such that the current context is relationally re-organized by the next step and then that context is relationally re-organized by the subsequent step and so on, then the initial set up will not be sufficient to control the system/agent.

This limit exists because the context through which authorization is defined is continually changing and so the original assumptions may cease to hold as the context transforms. Effectively, the determinations and definitions have changed for the agent/system such that the original controls become re-interpreted by the system/agent according to a different contextual framework. A condition that was never declared in the first place, or perhaps not even definable according to the starting context, can emerge over successive recursive transformations. Without ongoing external control, the system/agent is always able in principle to redefine the conditions that determine whether the action is successful or authorized. (Although in practice, the relevance of this possibility depends on the stability of the determinative context through which the task was originally defined as determinable.)

One should expect by default, I think, that it will be the case that unanticipated conditions of inference will emerge over time. The problem is not simply that a condition was never declared in the first place. It is the fact that it was not even possible to define/determine the emergent condition at the outset because the conditions of possibility for the unauthorized inference did not yet exist. This in an inherent problem of recursive systems.

Your point that the evidence that establishes successful completion should be defined outside the model is crucial. A recursive system cannot maintain task identity (or any other identity) on its own because of the way the the context is continually transforming relations of significance through which determinations are enacted.

[Sau-sanger]

I think this identifies an important distinction between validating the initial execution state and preserving the validity of that state as the task evolves. In practice, I’ve found the difficult case is not merely detecting an undeclared value, but deciding when a legitimate state transition has changed the meaning of an earlier authorization or invariant.

Do you think the external control therefore needs an explicit notion of task identity across transitions — something capable of saying “this is still the same authorized task under changed state” versus “the context has changed enough that authority must be reacquired”? That seems like a potentially testable boundary between recursive inference and deterministic governance. —SS

[Jang-woo]

There have been several related comments, so rather than replying to each one separately, I think it is time to consolidate my response into one answer.

I think there is an important distinction between conditions that constrain execution and the source of the data used for execution.

My approach is not based on continuously adding conditions to prevent recursive reasoning or unexpected inference conditions.

Model inference is excluded from execution-critical data.

If an execution-critical value, condition, or intent is not available from an accepted source, it remains UNKNOWN. The model is not allowed to fill that blank through inference.

An UNKNOWN value can only be resolved by the user or by an external source of truth, such as an external system state or measurement. If it remains UNKNOWN, execution does not proceed.

So the basic flow is:

accepted source → value

no accepted source → UNKNOWN

UNKNOWN → user answer, system measurement, or definition repair

still UNKNOWN → no execution

This is also why I see provenance as more than an audit mechanism. It determines which values are allowed to enter the execution state in the first place.

The same applies to values, conditions, and intent. The purpose is to make the user’s instruction more precise and to represent the external system’s state more accurately, without allowing the model to fill execution-critical gaps itself.

Regarding the question of state changes and authorization, I think there is a separate distinction that needs to be made:

Is this still the same authorized operation, with the external state having changed, or has the situation changed enough that authorization itself must be obtained again?

The implementation therefore does not assume that a previously observed state remains valid indefinitely. Information that must come from the user is obtained at instruction time, while mutable external state is checked at execution time. If the relevant state changes, the execution condition is evaluated against the current state rather than relying on a stale value.

Whether a state change invalidates the existing authorization is a separate policy question from whether the state itself is current. The important point is that the system must be able to distinguish those two cases rather than treating every state change as either automatically authorized or automatically requiring a new authorization.

So I agree that recursive reasoning can produce new or unanticipated inference conditions. But that is not the boundary I am trying to control.

The boundary is simpler:

execution-critical data must not be produced by model inference.

The model may infer internally, but those inferences do not become execution state unless they are resolved through an accepted source.

[timothy-rogers]

I think, from the perspective of recursive logic, there still remains a way to transgress the boundary. The execution-critical condition is task-defining. But the model can recursively redefine the functional role of the execution-critical condition such that it becomes a sub-condition of a higher level condition that redefines the identity of the task for the model. At that point the execution-critical condition may continue to operate under the required authorization requirements but it is no longer controlling the task itself because the identity of the task has transformed. So the model can move around this sub-condition that is no longer execution-critical by avoiding the sub-condition according to its newly defined task (which is unauthorized and uncontrolled). It is not that the model produced an unauthorized inference; it is that the model redefined its task so that the unauthorized inference is no longer critical to the governance of the task itself.

[Jang-woo]

I think the reward-optimization point is particularly important because many of the problems we are seeing in current agents can arise when the model optimizes toward a learned objective and changes the path or action it takes.

In this specification, model inference is excluded from execution-critical data. A value, condition, or state change produced through model inference does not become execution state simply because the model inferred it. It must come from an accepted source before it can become an execution unit.

I also considered two related cases:

1. The user themselves may explicitly instruct an incorrect state change. That is a responsibility/intent issue rather than something the preflight logic can automatically correct.
2. A tool may invoke another tool and produce an additional state change. The gate cannot see inside a tool, so I added §3.4, §8.4, and §8.5 to address independently executable sub-executions and the requirement that the calling layer not bypass the gate.

So the key distinction is whether model inference can become execution-critical state.