I analyzed hidden-state dynamics across 7 open-weight LLMs and found recurring functional patterns. Looking for feedback

Research
1

I’ve spent the last few months trying to answer a question that initially looked much simpler than it actually is:

What actually happens inside an LLM while it is generating a response?

Most work evaluates language models through their outputs (benchmarks, perplexity, reasoning scores…). I decided to look at something different: the evolution of the hidden representations themselves.

I built a runtime framework that records hidden states layer-by-layer during inference and started running the same experiments across multiple open-weight models (GPT-2, DistilGPT2, OPT-125M, Qwen2.5-0.5B-Instruct, TinyLlama, Phi-1.5 and Llama-3.2-1B).

I expected a relatively straightforward result.

Instead, every new experiment generated a new question.

Some of the observations so far are:

• Hidden-state trajectories are not random. They exhibit reproducible internal dynamical regimes across architectures.

• Functional proxy states (syntax-like processing, decision-like behavior and output stabilization) can be detected consistently enough to cluster models according to their internal dynamics rather than simply their parameter count.

• These functional signatures remain reasonably stable across different prompt families, although not perfectly, suggesting that prompt content modulates the dynamics without completely changing the internal organization.

• Linear probes can decode several functional categories directly from hidden representations with surprisingly high accuracy.

At that point the obvious question became:

Are we just overfitting labels?

So I started adding progressively stronger negative controls.

First:

  • label permutation.

Then:

  • random Gaussian representations.

Then:

  • feature permutation.

Finally:

  • orthogonal rotations of the hidden space.

The results became much more interesting.

Random labels collapse the decoding performance.

Random Gaussian representations also collapse it.

Feature permutation destroys most of the signal.

However…

Orthogonal rotations preserve almost all decoding performance.

This strongly suggests that the relevant information is not encoded in individual neurons or embedding dimensions.

Instead, it appears to be encoded in the relative geometry of the representation.

That was not the result I expected.

Another unexpected finding concerns depth.

Initially I was looking for something like “syntax layers” or “semantic layers”.

The data doesn’t really support such a simple picture.

Instead, the same functional signatures seem capable of appearing at different absolute layers depending on the architecture.

This led me to think less in terms of fixed layers and more in terms of functional regimes evolving through computation.

At this stage I am not claiming to have discovered a universal law of transformers.

These are empirical observations obtained on a limited set of open-weight models.

What I do believe is that they raise interesting questions about how computation is actually organized inside modern LLMs.

I’d really appreciate feedback from people working on:

  • mechanistic interpretability

  • representation learning

  • probing methods

  • transformer internals

  • geometry of representations

In particular I’d like your opinion on three questions:

  1. Which control experiment would you absolutely require before taking these observations seriously?

  2. Have you seen previous work showing comparable evidence that functional information is primarily encoded in representation geometry rather than individual dimensions?

  3. If you were extending this project, what would be your next experiment?

I’m not affiliated with a research lab—this is an independent research project. I’m sharing it because I would genuinely value critical feedback more than validation.

If there’s enough interest, I’m happy to share the methodology, code, and experimental reports.

2

Hmm… maybe something like this for now?:

I do not read this as a one-step “accept or reject” result. I would mostly treat it as a useful measurement map, and then separate a few layers so that future readers can tell which part is already measured, which part is a proxy-labeling choice, which part is a probe result, and which part would need later causal testing.

Directly answering your three questions first:

  1. The control I would want first: metadata-only and covariate baselines. Before adding more models, I would check how much of each proxy label can be predicted from generation step, token position, response length, prompt family, entropy, top-1 probability, top-1/top-2 margin, punctuation/EOS proximity, and chat-template boundaries. Then I would run prompt-family-held-out and model-family-held-out splits.

  2. Closest related-work buckets: I would look under probing controls/selectivity, MDL probing, representation-similarity methods such as CKA/RSA/Procrustes, and later causal-intervention tools such as activation patching, ablation, or steering. I would treat these as vocabulary/tool buckets rather than exact precedents.

  3. Next experiment: I would extend sideways before scaling up: same proxy labels and same models, but across metadata baselines, residualized states, extraction-point variants, held-out prompt/model families, and fixed-vs-retrained rotation probes. That would make the result much easier to interpret.

None of this is meant as a knock on the observation. I think the result is interesting precisely because it looks like it can be made more reusable with a few fairly concrete reporting and audit choices.

I also think the most useful current wording is probably something like:

Under this extraction / labeling / split pipeline, several proxy states are linearly recoverable from hidden-state trajectories, and the rotation/permutation controls suggest that the signal is not simply tied to individual coordinates.

That is already useful. I would just keep it separate from the stronger claim:

The model causally uses these functional regimes during generation.

That stronger version may be reachable later, but it needs a different family of tests.

1. Measurement contract: I would be very explicit about what “hidden state” means here

For replication, I would spell out the hidden-state extraction point very explicitly. This is not an objection; it just prevents future readers from comparing slightly different objects.

Things I would report:

Detail Why it matters
forward() vs generate() Hidden-state shapes and semantics can differ between a direct forward pass and generation output.
output_hidden_states=True Makes clear that all layer states were requested.
return_dict_in_generate=True Needed when collecting optional generation outputs through generate().
Prompt-token states vs generated-token states Input/prompt states and newly generated token states may need different collection paths.
Prefix decision-state vs generated-token state The state used to choose the next token is not necessarily the same object as the generated token’s later full-forward hidden state.
hidden_states[-1] vs last_hidden_state These are not always interchangeable; some models apply normalization or post-processing before returning last_hidden_state.
Hook point Residual stream, attention output, MLP output, final norm output, etc. are different objects.
Final norm before/after Especially important for decoder-only models using RMSNorm/LayerNorm variants.
Layer indexing Some APIs include embeddings as index 0, so “layer N” can be off by one across implementations.
use_cache During autoregressive generation, cache behavior changes what is recomputed and what is returned.
Serialized prompt / chat template Chat templates and special tokens can create position/role artifacts.

Useful references for this part:

A small sanity run made this concrete for me: generation step 0 and later generation steps can have different sequence dimensions in the returned hidden-state tuples. That is enough to make me think the extraction contract should be part of the result table, not just an implementation detail.

I would want a table like:

Model API path Token object analyzed Layer object Norm position Pooling
GPT-2 forward() / generate() prompt-final / generated-token / prefix decision-state / post-hoc token-state residual / final hidden pre/post final norm last-token / mean / per-token
Qwen2.5 … … … … …
Llama-3.2 … … … … …

That table would save future readers a lot of ambiguity.

2. Proxy-label notes: what the labels might be tracking

I like the idea of using process labels such as “syntax-like,” “decision-like,” and “stabilization,” but I would keep them as proxy/process labels until each one is linked to a more specific behavioral or mechanistic quantity.

A useful sanity check is: how much of each label is recoverable before hidden states enter the analysis?

Possible non-hidden-state predictors:

Possible predictor Why it matters
Generation step Later tokens may look more “stabilized” by construction.
Token position Syntax-like labels may correlate with early/mid/late position.
Response length Short vs long completions can create different trajectory shapes.
Prompt family Different tasks may induce different label distributions.
Entropy Low entropy can look like commitment or stabilization.
Top-1 probability High top-1 confidence may mimic “decision.”
Top-1/top-2 margin Large margin may mimic “commitment.”
Punctuation / newline / EOS proximity Formatting tokens can create artificial phase changes.
Chat-template role boundary System/user/assistant role tokens can introduce strong structural markers.

I would not frame this as “the labels are bad.” Proxy labels are often necessary in this kind of work. I would frame it as:

What exactly is each proxy tracking?

A metadata-only baseline helps answer that.

For example:

Test If it works well If it fails
Predict proxy label from generation step only Label may be partly temporal/positional. Good: hidden states likely add nontrivial information.
Predict proxy label from entropy/margin only Label may be mostly confidence/commitment. Good: hidden states likely contain something beyond surface uncertainty.
Predict proxy label from prompt family only Label may be task-format dependent. Good: signal may be less tied to prompt category.
Predict proxy label after residualizing these covariates Stronger representation claim. Result can be renamed as covariate-linked rather than discarded.

A recent nearby failure mode is the preprint “Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States”. The setup is not the same, but the caution is relevant: high probe accuracy can sometimes reflect task format / source identity / option count / response length rather than the intended computational category.

That paper is not a reason to dismiss your result. It is a useful template for asking: “what simple covariates can already explain the separation?”

In a small Qwen2.5-0.5B sanity check, this kind of issue showed up very clearly:

Proxy-like label Metadata result Interpretation
phase-like early/mid/late label generation-step metadata was already strong phase-like labels need step/position baselines
confidence/stabilization-like label entropy/margin metadata was already very strong stabilization-like labels need entropy/margin baselines
prompt/task family hidden states decoded it strongly prompt-family-held-out splits should be first-class
token surface category surface metadata was trivial, hidden states also carried it strongly syntax/format-like labels can mix with token-surface information

That does not imply your labels behave the same way. It just makes the baseline worth reporting.

3. Probe-result notes: decodability, selectivity, and cheap baselines

I would label the current probe result as a decodability result. That is useful by itself.

The next question is not just “is accuracy high?” but:

  • How selective is the probe?
  • How much data does it need?
  • Does it still work on held-out prompt families?
  • Does it still work on held-out model families?
  • Does it beat metadata-only baselines?
  • Does it still work after simple covariates are removed or matched?
  • Does it survive a different hidden-state extraction point?

Relevant probing references:

A compact probe report could include:

Item Why useful
Probe type Linear / logistic / ridge / MLP changes interpretation.
Regularization Prevents capacity from silently doing the work.
Split unit Token-level random splits can leak prompt-specific structure.
Prompt-family-held-out split Tests generalization beyond prompt format.
Model-family-held-out split Tests cross-architecture transfer.
Layer-selection protocol Avoids selecting the best layer on the test set.
Seed variance Shows whether the probe result is stable.
Bootstrap over prompts More informative than only reporting one point estimate.
Metadata-only baseline Shows what is recoverable without hidden states.
Residualized-hidden-state probe Shows what remains after simple covariates.

A useful wording might be:

The probe result shows that the labels are recoverable from the representation. The control/selectivity/MDL checks would tell us how specific and cheap that recovery is.

One extra check I would add is a continuous regression sanity check:

Regression target Why useful
hidden state → entropy Shows whether “stabilization” may be mediated by uncertainty.
hidden state → top-1 probability Shows whether confidence is directly recoverable.
hidden state → top-1/top-2 margin Shows whether commitment/competition is recoverable.

If hidden states predict entropy/margin well, then entropy/margin-derived proxy labels will naturally be probe-decodable. That is still interesting, but it should be named as such.

4. Geometry / rotation notes

The orthogonal-rotation result is one of the most interesting parts, but it becomes much easier to interpret if the probe protocol is named explicitly.

I would ask one direct clarification:

After applying the orthogonal rotation, was the probe retrained, kept fixed, or transformed together with the representation?

Those cases mean different things:

Rotation protocol How I would read it
Probe retrained after rotation Preserved performance is partly expected because orthogonal rotations preserve distances, angles, and linear separability. Still useful evidence against raw coordinate identity, but not surprising by itself.
Original probe kept fixed Stronger test of whether the original readout is tied to the original coordinate basis.
Probe weights co-transformed with the representation Mostly a coordinate-system sanity check: the same linear readout is being expressed in the rotated basis.
Different rotations for train/test Potentially very different claim; should be specified carefully.
Same rotation train/test Cleaner test of basis dependence.
Layer-wise independent rotations Tests each layer’s coordinate basis separately.
One global rotation Tests a different kind of global coordinate dependence.

A small sanity check made this distinction very concrete for me:

Protocol Typical behavior
original probe / original features baseline accuracy
fixed original probe / rotated features can collapse strongly
retrained probe / rotated features can recover the original accuracy
co-transformed probe / rotated features can recover the original accuracy exactly or nearly exactly

So I would not phrase the rotation result simply as “the information is geometric” unless the protocol is specified. I would phrase it more narrowly:

The result is consistent with the information being recoverable from relational structure in the representation rather than from fixed coordinate identities alone.

For related vocabulary:

So I would use CKA/RSA/Procrustes as descriptive tools, not as proof by themselves. If you add them, I would include random/shuffled/debiased controls and report sample-size sensitivity.

5. Transfer and robustness checks

For me, the most informative extension would be a robustness table, not just a bigger model list.

The table I would want:

Check What it tests
Random split Basic decodability under the current distribution.
Prompt-family-held-out Whether the proxy labels transfer across task/prompt types.
Model-family-held-out Whether the readout transfers across architecture/model families.
Template-held-out Whether chat formatting or role tokens are doing the work.
Length-bin-held-out Whether response length drives the label.
Entropy-bin-held-out Whether uncertainty/confidence drives the label.
Top-margin-bin-held-out Whether commitment/competition drives the label.
Extraction-point variant Whether the effect depends on a specific hook/API object.
Bootstrap over prompts/seeds Whether the effect is stable rather than a small-panel artifact.

I would not require every result to survive every split. The point is to name the result more precisely.

For example:

Outcome Possible interpretation
Survives random split only Distribution-specific decodability result.
Survives prompt-family-held-out More likely to reflect a reusable process label.
Survives model-family-held-out Stronger cross-model representation claim.
Collapses under entropy/margin controls Proxy may be mostly confidence/commitment geometry.
Collapses under template-held-out Proxy may be chat-format or role-boundary dependent.
Changes under extraction point Measurement is implementation/hook dependent.

This is especially important because prompt/task family itself can be strongly decodable from hidden states. In a small sanity run, prompt-family classification was much easier from hidden states than many of the process labels. That makes prompt-family-held-out evaluation very useful: it tells readers whether a “functional” probe is actually learning a task-family separation.

Again, not every result has to survive every split. A split table is not only a filter; it is a naming tool.

6. Extraction-point variant: decision-state vs post-hoc token-state

One specific distinction I would add is:

Object Meaning
Prefix decision-state The state around the prefix position used while predicting/emitting the next token during generation.
Generated-token state during generation The state returned by the generation API for newly produced tokens.
Post-hoc generated-token state The generated token’s hidden state from a full forward pass over the completed sequence.
Middle-layer token state Same token, but earlier/later internal layer.
Final-layer token state Same token, final hidden layer.

These are related, but they are not the same measurement object.

A small sanity check on Qwen2.5-0.5B made this distinction visible: the cosine similarity between a generation decision-state and the later post-hoc final-layer token-state was only moderate, not near-identical. Middle-layer and final-layer token states also lived on very different scales.

So I would avoid writing only:

hidden states were recorded

and instead write something like:

final-layer hidden states were recorded at <specific token position / API path / hook point>, with prompt-prefix states and generated-token states handled as follows…

That one sentence would make the analysis much easier to reproduce.

7. Norms, outliers, and boring geometry checks

I would also add a few boring geometry checks before interpreting clusters or trajectories:

Check Why useful
Hidden-state norm by label/prompt family Clusters can partly reflect norm differences.
Norm-normalized probes Tests whether direction, not norm, carries the signal.
Top outlier-dimension ablation Tests whether a few large channels dominate.
Whitening / PCA sensitivity Tests dependence on covariance structure.
Middle-layer vs final-layer comparison Tests whether the effect is layer-stage specific.
Prompt-template removal/change Tests special-token and chat-role effects.

In a small sanity check, final-layer states and middle-layer states had very different norm scales, while dropping top absolute-mean dimensions did not by itself erase most probe results. I would not over-read that, but it suggests norm/outlier controls are useful reporting items rather than guaranteed explanations.

Related background that may be relevant:

I would use these as cautionary references, not as claims that this is what explains your result.

8. Answering your geometry question more directly

Your second question was whether there is comparable evidence that functional information is primarily encoded in representation geometry rather than individual dimensions.

I would answer carefully:

  • There is a lot of nearby work showing that useful variables can be recovered from directions, subspaces, population-level activations, or representation geometry.
  • There is also a lot of work warning that linear decodability and visual/geometric separation can be confounded by format, dataset, probe capacity, or other covariates.
  • I would therefore not look for one exact precedent. I would treat this as sitting at the intersection of probing, representation similarity, and later causal intervention.

Useful buckets:

Bucket Why relevant Links
Probing controls/selectivity Helps interpret high probe accuracy. Hewitt & Liang
MDL / ease of extraction Asks whether labels are cheaply recoverable, not merely recoverable. Voita & Titov
Format-confound probing Close warning case for hidden-state probes. Task format vs reasoning mode
Representation similarity Vocabulary for comparing layers/models. CKA
CKA caveats Avoid over-reading high similarity in high-dimensional / low-sample regimes. Correcting biased CKA
Population-level representation work Related framing: information in directions/subspaces rather than single neurons. Representation Engineering
Causal follow-up Moves from “readable” to “behaviorally involved.” Activation patching best practices
Tooling Makes activation extraction/patching easier in later phases. TransformerLens, nnsight activation patching tutorial

I would phrase your current geometry claim as:

The controls suggest that the signal is not merely a fixed-dimension lookup. It appears to depend on relational structure preserved under orthogonal transformations.

Then I would leave stronger claims for the transfer/intervention stage.

9. If I were extending this project next

I would not scale up first. I would first make a compact audit matrix on the current setup.

Something like:

Axis Variants
Input baseline metadata-only, step-only, entropy-only, margin-only, position-only
Representation raw hidden state, norm-normalized hidden state, residualized hidden state, whitened hidden state
Token object prefix decision-state, generated-token state, post-hoc token-state
Split random, prompt-family-held-out, model-family-held-out, template-held-out
Extraction forward(), generate(), hook, pre-final-norm, post-final-norm
Rotation fixed probe, retrained probe, co-transformed probe
Statistics bootstrap over prompts/seeds, layer-wise confidence intervals

A minimal table in the report could look like:

Test If signal survives If signal collapses
Metadata-only baseline Hidden states add nontrivial information. Proxy label may be mostly metadata/format.
Entropy/margin control Signal is not just confidence geometry. Rename as confidence/commitment-linked.
Prompt-family-held-out More reusable process label. Prompt-family dependent result.
Model-family-held-out Stronger cross-model claim. Model-family-specific readout.
Extraction-point variant Measurement robust to API/hook choice. Hook/API-dependent observation.
Fixed vs retrained rotation Clarifies coordinate vs geometry claim. Rotation result needs narrower wording.
Bootstrap over prompts/seeds Stronger stability claim. Small-panel / sample-sensitive observation.

This kind of audit is useful either way:

  • If the signal survives, the result becomes much harder to dismiss.
  • If it collapses under a simple baseline, that is still progress because it tells you what the proxy was actually tracking.

This is similar in spirit to what happened in the GD_ratio thread: the temperature audit did not make the whole project worthless; it gave the result a better name and separated the raw claim from the residual question.

10. Later causal extension, only if useful

I would not require causal intervention for the current descriptive post. But if the project later moves from “recoverable proxy state” to “functional regime used by the model,” then the next family of tools would be:

  • activation patching
  • ablation
  • direction removal
  • activation steering
  • clean/corrupt prompt pairs
  • random-direction and label-shuffled controls

Caution: activation patching is not automatic proof. The paper “Towards Best Practices of Activation Patching in Language Models” is useful because it shows that patching conclusions can depend strongly on metrics, corruption methods, and other methodological choices.

Possible tool references:

I would keep this as a later stage, after the extraction/proxy/probe/transfer audits are clearer.

A compact way to summarize the whole thing:

I would treat the current result as a potentially useful decodability and geometry observation. The next step is to make the measurement contract explicit, then test what the proxy labels are tracking with metadata baselines, held-out prompt/model families, and extraction-point variants. If the signal survives those, it becomes much stronger. If it does not, the project still improves because the proxy gets a more accurate name.

That is the main reason I think this is worth pursuing: the result does not need to be defended in its strongest form to be useful. It just needs a clear map of which checks support which level of claim.

3

This s a really interesting direction. The rotation result especially stands out it suggests the signal is likely in the overall structure of the representation space, not individual neurons.

The biggest next step would be causal tests: probing shows information exists, but not whether it drives the model’s behavior. Things like activation patching or targeted interventions would help confirm that.

Also worth testing:

  • untrained models as a baseline
  • cross-model comparisons
  • whether state transitions carry more information than individual hidden states

The idea of dynamic functional regimes rather than fixed syntax semantic layers seems like a promising way to think about transformer internals.

4

Thank you for taking the time to write such a detailed review. This is exactly the kind of feedback I was hoping for.

I agree with your central point: decodability, proxy labeling, and causal function should remain clearly separated. At the moment I only claim that certain properties are linearly decodable from hidden-state trajectories under a specific extraction pipeline. Whether the model causally uses those representations is a different question that requires intervention-based experiments.

Interestingly, several of the directions you mention have become the focus of my current work over the last few days.

I extended the framework from simple probe accuracy toward a more complete characterization of internal dynamics, including:

  • Functional Capacity (how many independent properties are simultaneously linearly decodable from the same layer),

  • Functional Density (how that capacity is distributed across depth),

  • Functional Entropy and Effective Capacity,

  • Dynamic trajectory metrics (curvature, commitment, transition entropy, D-state occupancy, etc.),

  • Architecture-level comparisons across GPT-2, OPT, Qwen, Phi, TinyLlama and Llama families.

One unexpected result is that prompt ambiguity does not simply increase instability. Instead, it appears to modify the geometry of trajectories in architecture-dependent ways. In several models, ambiguity significantly changes curvature, cosine similarity, transition entropy or commitment without necessarily increasing D-state occupancy. That shifted my attention from “how many unstable states exist?” toward “where do these dynamic states appear along the trajectory and what role do they play?”

Your suggestion about metadata baselines is also well taken. I haven’t completed that part yet, but it is now high on the roadmap. Holding out prompt families and model families, residualizing obvious covariates, and comparing extraction points should help determine whether these dynamic signatures remain after controlling for simpler explanations.

I also agree that causal interventions are the natural next stage. Once the descriptive framework is stable, I would like to test whether perturbing specific trajectory regions or geometric structures actually changes downstream generation. That would allow us to move from observation toward causal evidence.

Thanks again for the thoughtful comments. They helped refine the roadmap considerably, and I appreciate that your suggestions focus on strengthening the measurement framework rather than simply arguing for or against the current interpretation.

5

Thanks, I really appreciate the thoughtful feedback.

I completely agree that probing and causality are two different questions. At this stage, my work is about measuring the organization of hidden-state trajectories, not claiming that the decoded properties are themselves causal. Causal interventions (activation patching, steering or targeted perturbations) are definitely where I’d like to go next once the descriptive framework is stable.

Interestingly, some of the experiments you suggested are already becoming part of the roadmap. I recently expanded the work beyond simple probing into what I call functional organization: measuring functional capacity, functional density, functional entropy, and dynamic trajectory metrics such as curvature, commitment and state transitions across several Transformer families (GPT-2, OPT, Qwen, Phi, TinyLlama and Llama).

One result that surprised me is that increasing prompt ambiguity doesn’t simply make models “more unstable.” Instead, different architectures respond differently, and the effect seems to be expressed more through trajectory geometry than through a simple increase in unstable states. That shifted my focus toward understanding the topology of these internal trajectories rather than looking only at individual layers.

I also like your suggestion about transitions carrying more information than isolated hidden states. In fact, that’s becoming one of the central ideas of the framework: instead of asking “where is information stored?”, I’m increasingly asking “how does information move through the network?” The trajectory itself may be the fundamental object, with individual hidden states being snapshots of a larger dynamic process.

The untrained-model baseline is also a great idea. If the same geometric organization appears before training, that would point toward architectural effects; if it only emerges after training, it would suggest that these structures are learned. That’s definitely an experiment I’d like to include.

Thanks again for the excellent suggestions they fit remarkably well with the direction the project is now taking.