Hmm… for now, taking everything so far into account, I think it comes out roughly like this:
Those new results narrow the problem down quite a bit.
The combination of:
- Encoder-only reaching 66.5%, above the full WND model’s 63.0% peak;
- masking or shuffling the Feature Bank causing little or no degradation;
- Depth becoming somewhat more useful on the harder dataset;
suggests that the current model has at least two regimes:
- on the present easier/medium tasks, the early path appears sufficient and the recurrent retrieval path adds little;
- as the task becomes harder, there may be a regime in which recurrence starts to provide useful extra computation.
So I would not interpret this simply as “Depth failed.” The harder-set result may be the most useful clue in the whole experiment. The immediate goal would be to identify when recurrence becomes necessary and where the full path first loses or gains information.
1. How to prevent Encoder takeover without assigning an artificial role
I think the cleanest answer is: do not prescribe what “reasoning” must look like with an activity loss; instead, define an information-flow contract under which the intended computation is identifiable.
For example:
| Component |
Diagnostic contract |
| Wide Layer / Encoder |
Produce rich local or limited-hop representations |
| Feature Bank |
Preserve input-conditioned reference context |
| Depth state |
Hold the evolving, mutable computation state |
| Depth iterations |
Perform repeated multi-hop integration or retrieval |
| Output head |
Read only the final Depth state |
The key distinction is between:
- forcing Depth to move, attend, or produce a particular trace; and
- making Depth the only route capable of the computation being attributed to it.
The latter is not necessarily an artificial semantic assignment. It is an architectural capability boundary.
A useful isolation variant would therefore be:
- keep the Wide Layer expressive;
- restrict the Encoder to node-local or one-hop processing;
- retain the Feature Bank as a high-dimensional reference store;
- let Depth query the bank at every iteration;
- remove any direct Encoder/Feature-Bank-to-output shortcut;
- allow global or multi-hop integration only through repeated Depth updates;
- let the output head read only
z_N.
I would treat this as a positive control and isolation experiment, not automatically as the final architecture.
If Depth begins showing a clear difficulty-dependent iteration benefit in that version, the original global Encoder probably provided an avoidable bypass.
If the restricted version simply collapses and Depth does not recover, then the Encoder was doing necessary computation rather than merely taking over, or the recurrent path lacks enough state, retrieval quality, or optimization support.
Structural separation alone does not guarantee learned functional specialization. Work on specialization under resource constraints similarly finds that specialization depends on the environment, resource constraints, architecture, and timing/bandwidth of information flow—not only on drawing module boundaries.
2. Before changing the architecture, locate where 66.5% becomes 63.0%
Because Encoder-only is better than full WND, there may be more than inactivity: one of the downstream stages may currently be degrading useful information.
I would compare a small sequence of matched variants:
| Variant |
Path |
Main purpose |
| A |
Encoder → Head |
Encoder’s direct performance |
| B |
Encoder → Compressor → Head |
Cost of compression |
| C |
Encoder → Compressor → Depth → Head |
Effect of recurrence |
| D |
Encoder → Compressor → Depth + Feature Bank → Head |
Added effect of retrieval |
| E |
Full WND |
Complete reference |
Ideally, these would use the same data split, seed set, training steps, head capacity, and early-stopping rule.
This distinguishes several possibilities:
- A > B: the Compressor is discarding useful information;
- B > C: the recurrent update is currently harmful or difficult to optimize;
- C > D: Feature Bank retrieval is injecting noise or distracting the state;
- A ≈ B ≈ C ≈ D: the difference may mainly be run variance or head/training differences;
- D > C only on hard cases: retrieval may become useful only after a difficulty threshold.
There is also an important distinction between two kinds of Encoder-only result:
- If Encoder-only was trained as a separate model, it shows that the architecture can learn the task without Depth.
- If the same full-model checkpoint performs similarly when Depth is skipped or
z_N is replaced with z_0, it shows that the trained full model’s prediction is not dependent on the recurrent updates.
Both are useful, but the second is the more direct causal test.
3. Interpreting the Feature Bank result
The mask result could be strong evidence that the bank path is unused, but the shuffle result depends heavily on what was shuffled.
If Feature Bank attention treats its entries as a set, changing only the array order may be expected to have no effect. Attention-based architectures can be explicitly permutation-invariant over sets; Set Transformer is a clear example.
I would arrange Feature Bank interventions as a ladder:
| Intervention |
What it disrupts |
| Entry-order shuffle |
Only sequence order |
| Node-entry shuffle |
Alignment between nodes and stored content |
| Key-only shuffle |
Retrieval addressing |
| Value-only shuffle |
Retrieved information |
| Independent key/value shuffle |
Key-to-content association |
| Cross-example bank swap |
Example-specific context |
| Zero or norm-matched random bank |
Bank content as a whole |
| Remove retrieval output from the Depth update |
Entire retrieval route |
The strongest low-cost test may be a cross-example bank swap between examples with the same graph size, or directly replacing the retrieval output with zero or norm-matched noise.
It is also worth verifying that the intervention actually reaches the recurrent computation:
- the retrieval vector changes;
- the next Depth state changes;
- the mask is applied to the tensor that is actually consumed;
- a residual path is not silently preserving the unmasked value.
That is just a positive control for the ablation, not an assumption of an implementation problem.
A practical reading of the current null result
If full removal of the retrieval output still has no effect, the leading branches would be:
- Encoder/Compressor already contains the needed answer information;
- Depth has learned to ignore retrieval;
- the current task does not require revisiting the bank;
- the recurrent path is not functionally connected as intended.
If only order-shuffling was harmless, considerably less can be concluded.
A compact default experimental route
If I were reducing the next stage to a small sequence, I would use:
- Same-checkpoint
z_0 versus z_N
- Encoder → Compressor → Depth → Bank stage localization
- Complete retrieval removal or cross-example bank swap
- Difficulty × iteration evaluation within one generator
- A strict-locality Encoder variant as a positive control
- A graph-structure-matched Transformer baseline
This order avoids changing several parts of the architecture before knowing which path is responsible.
Step 1: same-checkpoint state substitution
Compare:
- normal
Head(z_N);
Head(z_0) using the existing head;- small matched readouts trained separately on
z_0 and z_N.
The existing-head substitution tests causal dependence, although it may introduce distribution shift. The matched readouts test where answer information is decodable.
Step 2: localize degradation
Compare A–E from the previous table.
If possible, log both accuracy and prediction margin, because a stage may reduce confidence before changing top-1 accuracy.
Step 3: stronger bank intervention
Prefer one of:
- cross-example bank swap;
- value shuffle while retaining key distribution;
- complete retrieval-output removal;
- norm-matched random replacement.
Then confirm that z_(t+1) actually changes under the intervention.
Step 4: difficulty × iteration matrix
For each difficulty bin, evaluate Depth at 0/1/2/4/…/N steps.
Useful difficulty axes include:
- required path length;
- number of necessary hops;
- graph size;
- distractor count;
- branching factor;
- number of competing paths.
The main quantity could be:
accuracy at N steps - accuracy at 0 steps
for each bin.
Step 5: strict-locality positive control
Construct a variant in which one recurrent iteration can propagate information only one hop.
If the task genuinely requires k hops, this creates a clear expected relationship between task complexity and iteration count.
This does not have to be the final WND design. It asks whether the Depth implementation and training procedure can learn useful recurrence under conditions where recurrence is genuinely necessary.
Step 6: structure-matched Transformer
If the goal is to test the inductive-bias explanation, compare against a Transformer that receives comparable graph information rather than only a larger plain Transformer.
4. The harder-dataset result may be the most promising clue
I would give this result more attention than the aggregate Encoder-only score.
If Depth contributes increasingly as examples become harder, that is consistent with the intended idea: easy instances can be solved from an answer-ready encoding, while difficult instances require repeated computation.
However, comparing two datasets can mix several changes:
- path length;
- graph size;
- density;
- branching factor;
- distractor count;
- class balance;
- feature distribution;
- positive/negative generation rules.
The stronger version is to hold the generator family fixed and bin examples by a measured difficulty property.
For example:
| Required path length |
Depth 0 |
Depth 1 |
Depth 2 |
Depth 4 |
Depth N |
| 1–2 hops |
|
|
|
|
|
| 3–4 hops |
|
|
|
|
|
| 5–8 hops |
|
|
|
|
|
| 9+ hops |
|
|
|
|
|
A particularly informative result would be:
- short examples saturate at zero or one step;
- medium examples benefit from several steps;
- longer examples need more steps;
- the optimal iteration count moves with required hop count.
That would be substantially stronger evidence of iterative processing than showing that the Depth state changes or receives gradients.
Recurrent models have previously demonstrated easy-to-hard generalization by increasing their test-time iteration budget in Can You Learn an Algorithm?. The CLRS Algorithmic Reasoning Benchmark and its official repository are also useful references for size-based OOD evaluation, graph algorithms, and optional intermediate-state “hints.”
How I would interpret possible difficulty × iteration patterns
Pattern A: benefit tracks required hops
Example:
- two-hop cases need about two iterations;
- longer paths continue benefiting from additional iterations;
- graph size alone does not explain the result.
This would support the intended iterative-computation interpretation.
Pattern B: benefit tracks graph size but not path length
Possible interpretations:
- recurrence helps with general capacity or denoising;
- larger graphs produce a harder aggregation problem;
- the model is not necessarily traversing a path step by step.
Still useful, but a weaker match to the intended mechanism.
Pattern C: benefit appears only on a separately generated “hard” dataset
Possible interpretations:
- recurrence is useful;
- the hard generator changes another relevant statistic;
- a shortcut available in the easy dataset disappeared;
- the train/test protocol differs.
This motivates within-generator bins or matched examples.
Pattern D: benefit rises and then falls
The model may have a useful recurrence range followed by overthinking or fixed-depth specialization.
Logical Extrapolation Without Overthinking studies recurrent models that must remain stable when run beyond their training horizon, using techniques such as input recall and progressive training.
Pattern E: no benefit even under strict locality
This redirects attention toward:
- recurrent state capacity;
- retrieval implementation;
- gradient/optimizer path;
- training curriculum;
- task construction;
- output-head path.
5. Why a 50K Encoder can outperform a 600K Transformer
Yes, the Encoder’s inductive bias is a plausible explanation.
Parameter count alone does not determine sample efficiency or task suitability. A small model that directly respects graph structure may have a much easier learning problem than a larger model that must infer that structure from a weaker representation.
Relevant factors include:
- local message passing;
- permutation equivariance;
- explicit edges;
- node or edge types;
- graph-specific aggregation;
- shortest-path or structural encodings;
- the amount of data available;
- optimization budget.
Graph Transformer research supports this general point. Graphormer emphasizes explicitly encoding graph structure into the Transformer, while GRIT reports that Graph Transformers without sufficient graph inductive bias can perform poorly on smaller datasets, where those biases matter more.
So the result is not inherently surprising. However, several explanations remain compatible with it:
-
good Encoder inductive bias
Its locality or graph operations match the task.
-
insufficient structure in the Transformer input
The Transformer may not receive comparable adjacency, edge, or positional information.
-
data-regime advantage
The smaller model may generalize better with the available sample size.
-
optimization-budget difference
The 600K model may need more steps, different regularization, or more tuning.
-
task shortcut
The Encoder may exploit a generator-specific statistic efficiently.
-
single-run variance
A few points may move across seeds.
I would therefore separate three kinds of matching:
| Matching type |
What is held comparable |
| Parameter-matched |
Unique trainable parameter count |
| Compute-matched |
Training/inference operations or measured time |
| Information-matched |
Graph edges, node features, structural encodings and masks |
The last one is especially important. If the 50K Encoder receives explicit topology while the Transformer must reconstruct topology from a serialized input, the comparison mixes architecture with information availability.
The most useful next baseline
To test the inductive-bias hypothesis, I would not necessarily add an even larger Transformer. I would add a Transformer with comparable graph structure:
- adjacency-aware attention or masking;
- edge information;
- degree or centrality encoding;
- shortest-path or relative structural encoding;
- the same node features and pooling target.
A simple comparison ladder could be:
| Baseline |
Main role |
| Statistics-only MLP |
Detect generator shortcuts |
| Small graph Encoder / MPNN |
Local graph inductive bias |
| Plain Transformer |
Weakly structured global attention |
| Transformer + adjacency mask |
Explicit local topology |
| Transformer + structural encoding |
Graph-relative information |
| Hybrid local MPNN + global attention |
Stronger graph-Transformer baseline |
This makes the 50K-versus-600K result easier to interpret without diminishing it.
What I would report for the 50K versus 600K comparison
To preserve the value of the preliminary result, I would report each efficiency claim separately.
| Quantity |
Encoder-only |
Full WND |
Transformer |
| Unique trainable parameters |
|
|
|
| Training examples |
|
|
|
| Optimizer steps |
|
|
|
| Recurrent iterations |
|
|
|
| Training wall-clock |
|
|
|
| Inference latency |
|
|
|
| Peak device memory |
|
|
|
| IID accuracy |
|
|
|
| Hard-set accuracy |
|
|
|
| Longer-path OOD |
|
|
|
| Larger-graph OOD |
|
|
|
| Mean ± variation across seeds |
|
|
|
This distinguishes:
- parameter efficiency;
- sample efficiency;
- training speed;
- inference compute and latency;
- memory footprint;
- OOD generalization.
A recurrent model can use few unique parameters while still spending additional sequential compute at inference, so parameter and compute efficiency should remain separate.
6. The Dynamic Context Construct clarification helps
Your description makes the current WND experiment easier to interpret.
The Feature Bank appears to be:
- high-dimensional;
- generated from the current input;
- preserved during the current example;
- dynamically queried by Depth.
The Depth state appears to be:
- small;
- mutable;
- recurrent;
- responsible for keeping what is currently needed.
That looks like a meaningful separation between:
- input-conditioned reference context, and
- recurrent working state.
Whether it later supports a broader separation between persistent knowledge and reasoning may depend on how the bank is populated, retained, or updated in other tasks. But the current distinction is already technically useful and testable.
One remaining architectural question is where evolving intermediate objects are meant to live.
For a graph procedure, those might include:
- visited nodes;
- current frontier;
- partial path;
- tentative distances;
- predecessor pointers;
- unresolved alternatives.
If the Feature Bank is read-only and the compressed Depth state is the only mutable object, all of those intermediate results must fit in that small state.
That may be completely sufficient for the present task. If a strict-locality version makes Depth necessary but performance remains low, then state width, multiple mutable slots, or a small writable workspace become reasonable next branches.
I would not prioritize that capacity change yet, because the current Feature Bank ablations first suggest that retrieval may not be used. But it becomes important after the early bypass is removed.
A possible WND-compatible memory contract
One option that preserves the intended wide/narrow distinction would be:
- keep the original Feature Bank read-only;
- give Depth a much smaller writable workspace;
- let Depth read from both at each iteration;
- let only the final Depth state reach the output head.
This separates:
| Memory type |
Purpose |
| Read-only Feature Bank |
Preserve rich input-conditioned context |
| Mutable Depth state |
Control the current operation |
| Small writable workspace |
Preserve evolving intermediate results |
Architectures such as Neural Turing Machines make an explicit distinction between controller state and addressable read/write memory. WND does not need to reproduce that design; it is simply a useful reference for separating stable reference content from mutable computation state.
If the strict information-flow variant succeeds or fails
If it succeeds
Suppose the local Encoder version produces:
- increasing Depth dependence;
- path-length-dependent iteration benefits;
- meaningful Feature Bank intervention effects.
That would support the bypass hypothesis.
Possible long-term choices:
- retain limited Encoder receptive field;
- gradually relax locality until the bypass returns;
- prohibit only the direct output shortcut;
- use path dropout during training;
- temporarily freeze the Encoder, then jointly fine-tune;
- randomize the number of training iterations.
The useful design boundary would be the least restrictive one that preserves Depth-dependent behavior.
If it fails but larger state helps
This supports a mutable-state bottleneck.
Options:
- wider
z_t;
- multiple recurrent slots;
- node-aligned scratch slots;
- limited writable memory;
- more structured bank retrieval.
If it fails but stronger retrieval helps
This suggests Depth lacked access to the right information.
Options:
- input recall at every iteration;
- node-aligned keys/values;
- explicit query-node identity;
- gating changes;
- residual retrieval integration.
If accuracy collapses and none of these help
Return to:
- graph generator controls;
- recurrent optimization;
- head path;
- parameter registration;
- gradient detachment;
- whether the task admits the intended iterative decomposition.
Short implementation sanity checks if code is later available
These would be routine checks, not a diagnosis:
- Depth parameters appear in
model.named_parameters();
- those exact parameters are in the optimizer;
- shared recurrent weights are genuinely shared;
- no accidental
detach() or no_grad() interrupts the path;
- retrieval output enters
z_(t+1);
z_N, rather than a cached earlier state, reaches the head;- the retrieval gate is not permanently saturated;
- parameter values actually change after
optimizer.step().
I would also use hooks to verify that:
- masking the Feature Bank changes the retrieval tensor;
- the retrieval intervention changes the Depth input;
- cross-example bank swaps propagate into later states.
My current default interpretation
At this point, I would summarize the evidence like this:
- The early Encoder path is currently strong enough to solve much of the task.
- The full Compressor/Depth/retrieval path has not yet shown an aggregate advantage and may currently degrade some useful information.
- The existing Feature Bank intervention suggests low reliance, although the exact shuffle semantics matter.
- The harder-dataset result leaves open an important and promising regime in which recurrence becomes useful.
- The 50K-versus-600K result is compatible with a graph inductive-bias advantage, but structure-, compute-, and information-matched baselines would clarify it.
- The
/5 clarification makes the present experiment primarily a test of input-conditioned context versus recurrent working state.
So I would not try to force specialization globally.
I would:
- localize where the full path loses performance;
- validate the Feature Bank ablation more strongly;
- map iteration benefit against actual task difficulty;
- create one strict information-flow positive control;
- compare against a structure-matched Transformer.
Then the next design choice follows from the result:
- bypass confirmed: restrict only the route that creates the bypass;
- working-state limit confirmed: enlarge or structure mutable state;
- retrieval limit confirmed: improve addressing or input recall;
- fixed-depth behavior confirmed: vary training depth;
- task shortcut found: repair the generator or split;
- none of these: inspect optimization and implementation.
The fact that Depth appears more useful on harder data is the part I would preserve and investigate most carefully. It may indicate that the intended separation is not absent, but that the current benchmark spends most of its time below the difficulty threshold where recurrence becomes necessary.