Cross-architectural runtime probability dynamics in transformer LLMs — two clusters not explained by parameter count

Research
1

I want to share a finding from a measurement framework I’ve been working
on, because the result is counterintuitive enough that I think it might
interest people thinking about architectural differences between
transformer LLMs.

The setup

I measured the runtime geometry of probability distributions across
eight open-source attention-based transformers ranging from 70M to 1.3B
parameters: Pythia-70M, DistilGPT-2, GPT-2, OPT-125M, Pythia-160M,
Qwen2.5-0.5B, TinyLlama-1.1B, and Phi-1.5.

For each (token, layer) point during inference, the framework computes
geometric properties of the probability distribution over the vocabulary:
entropy, concentration on the top candidates, competition between the
leading and runner-up tokens, dispersion above a 1% threshold. From these
metrics, a bicephalic operator separates two distinct geometric tensions
that probability distributions can carry, which I label G (concentration
pole) and D (competition pole). The ratio between mean G and mean D, what
I call the GD_ratio, becomes a per-model signature.

What I found

The eight models do not vary continuously on the GD_ratio. They partition
into two clusters with no overlap and roughly an order of magnitude of
gap between them:

GPT-2 GD_ratio 2.458

Phi-1.5 GD_ratio 1.764
DistilGPT-2 GD_ratio 1.577

Qwen-0.5B GD_ratio 0.079
OPT-125M GD_ratio 0.074
Pythia-70M GD_ratio 0.059
Pythia-160M GD_ratio 0.039
TinyLlama-1.1B GD_ratio 0.021

The cluster split appears on three independent components of the operator:
the GD_ratio itself, the mean G alone, and the mean D alone. The
separation is not an artifact of one metric.

The interesting part is what does not explain the clustering. Parameter
count does not. GPT-2 has 124M parameters and is in the upper cluster.
OPT-125M has 125M parameters and is in the lower cluster. Phi-1.5 has
1.3B parameters and sits with GPT-2. TinyLlama-1.1B has roughly the same
size as Phi-1.5 and sits with OPT.

What might explain it (hypothesis only)

The most parsimonious pattern I can see is that the upper cluster shares
characteristics of training corpus curation. Phi-1.5 was trained on
heavily curated synthetic data. GPT-2 and DistilGPT-2 share the original
GPT-2 WebText distribution and tokenizer, which had its own filtering
protocol. The lower cluster spans more heterogeneous training corpora,
including older (OPT, Pythia) and newer (Qwen, TinyLlama) architectures
trained on relatively unfiltered web text.

I want to be careful here: this is a hypothesis, not a finding. I do not
have an experimental setup that isolates training corpus from architecture
choices. The hypothesis is consistent with the data but cannot be
established by it.

Why this might matter

If the two-cluster structure generalizes, any tool or analysis that
implicitly assumes a single dynamic profile across transformer models
will produce inconsistent results depending on which cluster the target
model falls into. This includes calibration techniques, uncertainty
estimation methods, and probably some interpretability approaches that
were tuned on one architectural family and may not transfer cleanly to
the other.

Other observations in the same study

A few things worth noting briefly:

  • The framework also defines a five-state taxonomy of dynamic regimes
    (stable, hidden turbulence, surface branching, committed, full
    bifurcation). The full bifurcation state turns out to be consistently
    transient across architectures: on three primary models tested in depth,
    its self-transition probability is 0.023 (GPT-2) or exactly 0.000
    (OPT-125M, Qwen-0.5B). Models pass through this regime, they do not
    settle into it.

  • Three models tested under controlled hidden-state perturbation respond
    in qualitatively different ways. GPT-2 absorbs the perturbation with
    state percentages shifting by less than 1.5 points. OPT-125M converts
    the perturbation into surface dispersion (branching state rises +12.5
    points). Qwen-0.5B destabilizes its dominant state (stable state drops
    -18.8 points). Three architectural perturbation signatures, same input
    noise.

  • One model (Phi-1.5) produces an anomalous taxonomy distribution under
    the standard threshold rule. I report it openly as needing dedicated
    investigation rather than smoothing it over.

What I’m not claiming

The panel is eight models, all under 1.3B parameters. The two-cluster
structure could collapse, stretch, or restructure when extended to 7B+
models. I have not validated on non-transformer architectures within
this study. The work is single-author and has not been independently
replicated. The training-corpus hypothesis is offered, not established.

I included explicit “limited findings” and “rejected claims” sections in
the paper, listing five things in each category that initial intuitions
suggested but that the data either partially support or actively reject.
I treat this as central to the framework’s credibility.

Why I’m posting

I would be interested in hearing whether anyone working with larger or
more architecturally diverse models has observed similar partitioning
phenomena in their own measurements, whether on attention, hidden states,
gradients, or any other intermediate quantity. The two-cluster structure
felt unexpected enough that I want to understand whether it is a
transformer-wide phenomenon, an artifact of the parameter range I tested,
or something specific to the particular operator I defined.

I would also be interested in alternative interpretations of the cluster
split beyond the training-corpus hypothesis. Possible candidates I have
considered but cannot test from this panel alone: pre-norm vs post-norm
architecture, tokenizer differences, attention head configurations,
intermediate layer dimensionality, positional encoding choices.

Where the details are

Full methodology, all tables, the explicit limitations section, and the
list of rejected claims are in the preprint on Zenodo:

Happy to discuss the operator definition, the threshold methodology, the
cluster finding, or any concern about the panel size and statistical
robustness.

2

Really like this one - and what makes it worth engaging with seriously is the discipline, not just the result. An explicit rejected-claims list and “hypothesis, not finding” on the corpus story is the posture that turns a cluster plot into something credible. So in the same spirit, here are the controls I’d want to see before believing GD_ratio is measuring dynamics rather than something more boring - plus one concrete test you can run from the architecture side.

(Working from your writeup + the abstract, so apologies if some of this is already handled in the preprint.)

1. Rule out logit temperature first - this is the cheap one I’d run before anything else. Your operator reads the geometry of the softmax distribution: entropy, top-k concentration, top-1/top-2 competition. All of those are dominated by the scale of the logits entering the softmax, and that scale is set by things unrelated to “dynamics”: final-LN gain, tied vs untied embeddings, weight decay, label smoothing. A model trained to be more confident has sharper distributions → high G, low D → high GD_ratio, with no trajectory behavior involved. That’s actually a clean mechanism for your training-corpus hypothesis (curated-data models tend to run lower-entropy), but it means GD_ratio could just be a per-model temperature reading. Test: fit one scalar temperature T per model on a held-out set to match a common mean entropy, recompute G and D on the calibrated logits, and see if the two clusters survive. If they collapse, you’ve measured calibration. If they hold, you’ve got something real on top of calibration. Either way it’s a stronger paper.

2. Make the operator vocab-invariant before cross-model comparison. Entropy and “dispersion above a 1% threshold” both scale with vocabulary size, and your panel runs 32k (TinyLlama) to ~152k (Qwen). It’s not a clean confound - OPT and Pythia sit at ~50k and still land in the lower cluster, so size alone isn’t the story - but a raw entropy / threshold-count still isn’t comparable across those vocabularies. Normalizing entropy by log(V), or computing the geometry over the top-p (say 0.99) mass instead of the full tail, removes tokenizer granularity as a hidden variable so the split can’t be partly a vocab artifact.

3. The perturbation experiment needs magnitude-matching and a null. This is the part closest to work we do (inject into hidden state, measure the downstream distribution), so two things we’ve been burned by: (a) “same input noise” across models is different relative perturbation if their hidden-state norms differ - and activation RMS varies a lot across models, and across layers within a model. Before concluding “architecture-specific response,” scale the injected noise to each model’s per-layer activation RMS; otherwise GPT-2 “absorbing” the noise might just mean GPT-2 carries larger activations. (b) Add a zero / identity-perturbation null that must produce no change - it proves the measurement path itself isn’t manufacturing the effect. With those two controls, “three reproducible perturbation signatures” becomes very hard to argue with.

4. Put error bars on the gap. The “order of magnitude, no overlap” split is the headline and it currently rests on one GD_ratio point per model. Bootstrapping GD_ratio over prompts/sequences (resample, recompute, CI per model) lets you state non-overlap quantitatively instead of visually - on n=8, single-author, that’s the difference between “suggestive” and “hard to dismiss.”

5. A concrete test from the large-model side. You asked whether this holds past 1.3B. One thing we see clearly on a 12B (Gemma) is that a few architectural choices impose output geometry independent of training. Gemma applies final-logit softcapping - a tanh cap on the logits - which structurally bounds exactly the concentration/competition geometry your G and D measure. Prediction: a softcapped model will cluster by the cap, not by its corpus. So dropping Gemma (or any softcapped model) into the panel is a clean way to separate “architecture-imposed logit geometry” from “training dynamics.” More generally, past 7B the confound surface grows - softcapping, QK-norm, attention-sink / massive-activation effects - all of which move your operator without touching anything you’d call dynamics. If the two clusters survive those, the structure is real; if it reorganizes around them, that’s the finding.

None of this is a knock - it’s an interesting lens and the honest-limitations framing is exactly why it’s worth pushing on. If I had to order it: temperature-normalization first, because it’s cheap and it either kills the result or makes it much stronger.

3

This is exceptional feedback. Thank you for taking the time to dissect the operator and the methodology with such rigor. You’ve pinpointed the exact confounds that keep me up at night, particularly regarding calibration and activation norms.

I want to address your points directly, as they align perfectly with my own internal audits (some of which are in the “Rejected Claims” section of the preprint, but deserve more prominence):

1. The Temperature/Calibration Confound (Priority #1)
You are absolutely right. GD_ratio could simply be a proxy for per-model “confidence temperature” driven by training objectives (e.g., label smoothing, final-LN gain).

  • Action: I will run the scalar temperature fitting test you suggested. If the clusters collapse after entropy-matching, then GD_ratio is indeed a calibration metric, not a dynamic one. If they hold, it proves there is a structural geometric difference beyond simple sharpness. This is the cheapest and most decisive test. I’ll update the repo with these results.

2. Vocabulary Invariance
Good catch on the vocab size disparity (32k vs 152k).

  • Current Mitigation: My operator uses relative thresholds (top-k competition, dispersion above 1% of the mass), which helps, but raw entropy is definitely biased by VV.

  • Action: I will recompute using entropy normalized by log⁡(V)log(V) and also try the “top-p mass geometry” approach you mentioned. This should isolate the shape of the distribution from its granularity.

3. Perturbation Magnitude & Nulls
This is a critical methodological flaw in many interpretability studies.

  • Action: I currently inject fixed Gaussian noise. I will switch to RMS-normalized noise (scaled to each layer’s activation RMS) to ensure equitable perturbation strength. I will also add the zero-perturbation null to baseline the measurement pipeline’s stability. This will strengthen the claim that the “three signatures” are architectural, not artifacts of scale.

4. Error Bars & Bootstrapping
Agreed. Visual separation on n=8n=8 is suggestive, not definitive.

  • Action: I will bootstrap the GD_ratio over prompts and seeds to generate Confidence Intervals (95% CI) for each model. This will quantify the “no overlap” claim statistically.

5. The Large Model Test (Gemma & Softcapping)
This is a brilliant concrete test. Gemma’s logit softcapping is a known architectural constraint that directly bounds concentration metrics.

  • Hypothesis: If Gemma clusters with the “Low GD” group solely due to softcapping, it confirms that architecture can override training corpus signals.

  • Action: I am currently limited by compute for full 12B runs, but I can run inference on smaller softcapped variants or use existing hidden-state logs if available. Alternatively, I can simulate softcapping on my current panel to see if it forces them into the lower cluster.

Next Steps:
I’m prioritizing the Temperature Normalization and RMS-scaled Perturbation audits this week. These are high-leverage controls that will either validate the core hypothesis or refine it into a more precise statement about calibration vs. dynamics.

I’ll post an update here once the bootstrapped CIs and temperature-calibrated ratios are ready. Thanks again for pushing the rigor bar higher this is exactly how robust science gets done.

Best,
Jean-Denis

4

Thanks for this — and you were right.

Ran the temperature normalization test you suggested as priority 1. The
result is a partial falsification of the strongest interpretation of the
finding.

Setup: fitted a per-model scalar temperature T to match a common target
entropy across the panel (target = 3.01), recomputed geometry on the
calibrated logits, recomputed clustering.

What happened: two models migrate between clusters after calibration.
GPT-2 and Phi-1.5 both move. The raw clustering structure is therefore
substantially driven by effective logit temperature, not purely by
runtime dynamics.

What this falsifies: “Raw GD_ratio directly measures model dynamics
independently of calibration.” This interpretation is rejected by the
test you proposed.

What may still hold: residual structure after calibration. Two clusters
still appear post-calibration, but their composition changes — so the
question becomes “is there a calibration-independent component to the
clustering, and if so what does it measure?” That requires the additional
controls you listed: vocab normalization (log V), top-p truncation,
perturbation magnitude-matched to per-layer activation RMS, bootstrap CIs
on the gap. None of those are done yet.

The published version of the V20 preprint (deposited yesterday) overstates
the dynamical interpretation of the raw GD_ratio. I am preparing a
revision that incorporates the temperature audit explicitly. The
falsification is going into the rejected-claims section. The remaining
structure question is moving into limited-findings with the controls you
specified as the protocol for elevation.

The attention-based findings from companion work (different observation
level than logits) are not affected by this confound and remain the
direction I will continue to develop.

Genuinely useful review. The “temperature first because it kills the
result or makes it much stronger” framing was the right priority.

5

This is the rare part - you ran a test that partially killed your own headline result, and you’re putting it in rejected-claims and revising the preprint over it. That’s the whole game, and most people quietly don’t. Respect.

A few thoughts now that temperature is ruled in as a primary driver and the live question is the residual:

Name what a residual component would actually be measuring. Once mean entropy is matched across the panel, two models at the same entropy can still arrange that probability differently - a sharp top-1 with a thin tail vs a softer top-2 with a fatter tail carry identical entropy but different shape. So a calibration-independent GD component, if it survives, isn’t “dynamics” - it’s distribution shape at fixed uncertainty (tail mass / top-1–top-2 margin / participation ratio at matched entropy). Framing the surviving claim that way keeps it from drifting back toward the “dynamics” reading the audit just rejected.

Scalar-T only matches the mean - go per-token / per-bin. One temperature per model equalizes average entropy, but token-level entropy varies a lot within a model. If you stratify by local-entropy bin (or temperature-match per token) and the cluster structure still appears within bins, that’s a far harder-to-dismiss residual than a mean-matched one. If it vanishes under per-bin matching, the residual was just calibration at finer grain.

The migration pattern is data, not noise. You found GPT-2 and Phi move - report the post-calibration membership explicitly (who clusters with whom now). If the residual clusters re-form along a different axis - tokenizer family, pre/post-norm, attention-head config - that is the answer to “what does the residual measure.” The direction of migration is more informative than the fact of it.

Bootstrap is the gate, not a nicety, at n=8. With two of eight already migrating, “two clusters still appear post-calibration” could be a small-sample artifact. A bootstrap-over-prompts CI on the residual gap is what decides whether there’s a residual structure to explain at all - I’d run that as the elevation gate before the vocab/perturbation controls, because it’s the cheapest and the answer is binary.

One caution on the attention companion work. Moving to attention as “a different observation level unaffected by this confound” is reasonable, but attention carries its own confounds - attention-sink / massive-activation tokens (the BOS-sink effect), head redundancy - and those are themselves training/architecture dependent. So an “architecture signature” in attention could be an attention-sink signature. The same discipline that just paid off on the logit side (find the boring mechanism first) is worth running there before it becomes the new headline.

Either way: falsify-first, publish the negative, revise - that’s exactly why this framework will end up trustworthy where flashier ones don’t. Good work.

6

Any progress? Are you still continuing down the rabbit hole?

7

Thanks for checking back and yes, I definitely kept going down the rabbit hole.

Your previous comments ended up changing the direction of the project in a meaningful way.

Following your suggestion, I treated logit calibration as a potential confound rather than assuming GD_ratio was measuring intrinsic dynamics. I then ran a series of follow-up audits:

  • Temperature normalization audit (V22.5): after matching entropy across models, a substantial part of the original structure changed. Temperature is indeed a major driver.

  • Vocabulary-invariant audit (V22.6): normalizing for vocabulary size and tokenizer granularity only partially altered the structure. The result was PARTIAL, suggesting vocabulary explains some, but not all, of the signal.

  • Bootstrap confidence audit (V22.7): only part of the pairwise differences remained statistically robust, so I downgraded the strength of the original claim rather than strengthening it.

  • Residual structure audit (V22.8b): with the pipeline corrected, the remaining signal is still strongly associated with effective temperature, so I’ve revised the interpretation again.

At this point I no longer describe GD_ratio as a direct measure of “internal dynamics.” A more accurate interpretation is that it captures aspects of output distribution geometry, with temperature acting as an important confounding factor.

The interesting part now is the residual after controlling for those obvious mechanisms. Rather than trying to defend the original interpretation, I’m using the negative results to refine the framework.

Your suggestion about per-token / entropy-bin calibration is still on my list. I think that’s probably the next meaningful control because it asks whether any residual survives after local calibration instead of only matching the global mean.

I’m also moving beyond logits toward hidden-state and attention analyses, but with the same philosophy you recommended: identify the boring explanations first (calibration, tokenizer effects, attention sinks, architectural constraints) before making stronger claims.

So yes the rabbit hole got deeper, but also narrower. The framework is becoming less ambitious in its claims and, I hope, more trustworthy because of that. Thanks again for pushing on the weak points rather than just the positive results.