Rethinking Transformer Architecture Through the Lens of Group Theory:

[rongxianlingyi]

Abstract

Transformer architectures have fundamentally changed the landscape of artificial intelligence by demonstrating remarkable abilities in language understanding, reasoning, and representation learning.

However, a deeper question remains:

What mathematical structures are hidden beneath learned representations?

Group theory provides one of the most powerful languages for describing structure in mathematics and physics. It studies transformations, symmetries, invariants, and representations — the fundamental concepts behind many physical theories and mathematical systems.

This article explores possible connections between group theory and Transformer architectures.

We do not claim that Transformers are literally implementations of group representations. Instead, we propose that several ideas from group theory — especially symmetry, representation spaces, and invariance — provide a useful conceptual framework for understanding how neural networks learn and organize information.

Furthermore, we discuss a deeper question:

Could future AI systems require not only larger representations, but representations that dynamically evolve through experience?

---

1. Why Group Theory?

The central question of group theory is:

What remains unchanged when something changes?

A group describes transformations:

g\\rightarrow g(x)

Different transformations may change the appearance of a system while preserving deeper structures.

Examples:

• A rotated object changes its coordinates but keeps its identity.
• A physical system changes its state but preserves conservation laws.
• A sentence changes its wording but preserves semantic meaning.

Intelligence itself can be viewed as the ability to recognize stable structures under continuous change.

A human does not remember every possible sentence.

Instead, humans learn deeper representations:

“cat”

is not stored as one sentence.

It is connected to:

• animals
• movement
• vision
• emotion
• experience

Intelligence is therefore not only information storage.

It is the discovery of invariant structures.

---

2. Representation Theory and Attention Mechanisms

Representation theory studies how abstract transformations can be expressed as concrete operations.

A group representation maps:

\\rho\\rightarrow GL(V)

where:

• (G) is a transformation structure
• (V) is a vector space
• (\rho) describes how transformations act on that space

Transformer models also operate through learned representations.

Given an input embedding:

x

the model creates different projections:

Q=xW_Q

K=xW_K

V=xW_V

These projections allow the model to analyze different relationships inside the same input.

A possible interpretation:

Multi-head attention creates multiple learned relational subspaces.

Each attention head may specialize in capturing different types of relationships:

• syntactic relations
• semantic relations
• positional relations
• contextual dependencies

This is not identical to irreducible representations in group theory.

However, the conceptual similarity is valuable:

Both systems attempt to decompose complex structures into simpler relational components.

---

3. Rotary Position Embedding:

A Concrete Example of Symmetry

One of the clearest connections between Transformer architecture and group theory appears in Rotary Position Embedding (RoPE).

RoPE represents positional information through rotations.

A two-dimensional rotation can be written as:

R(\\theta)= \\begin{bmatrix} cos\\theta&-sin\\theta\\ sin\\theta&cos\\theta \\end{bmatrix}

This transformation belongs to:

SO(2)

the rotation group in two dimensions.

The important idea is not absolute position.

It is relative transformation.

Instead of asking:

“Where is this token?”

RoPE encodes:

“How does the relationship between positions change?”

This is deeply aligned with the philosophy of symmetry:

Objects are often defined not by their absolute state, but by how they transform.

---

4. From Fixed Representations to Dynamic Representations

Current neural networks usually have fixed parameters.

After training:

\\rho

remains approximately fixed.

The model processes new inputs, but its fundamental representation structure does not continuously evolve.

Human cognition appears different.

A person’s representation of a concept changes through experience.

The word:

“home”

may have a completely different internal structure for:

• a child
• a traveler
• someone who has lost their home

The word is the same.

The representation geometry has changed.

This suggests a possible distinction:

Static representation:

\\rho

Dynamic representation:

\\rho(t)

where the representation itself evolves over time.

This raises an important research question:

Could future AI systems require mechanisms that allow representations to change through experience?

---

5. Memory as Geometry Transformation

Traditional AI memory usually works by retrieving information:

Past information
        ↓
Retrieval
        ↓
Context injection
        ↓
Output

This increases available information.

But biological memory may work differently.

Memory does not simply add facts.

Memory changes how future information is interpreted.

An experience can modify:

• conceptual distances
• attention patterns
• associations
• priorities

Therefore:

Memory may not be the addition of new points.

It may be the transformation of the space itself.

Instead of:

(V,d)

where:

• (V) is representation space
• (d) is distance

experience may create:

(V,d_M)

where the geometry of meaning has changed.

This leads to a deeper question:

Is memory fundamentally information storage, or is it the evolution of representation geometry?

---

6. Afterglow and Dynamic Representation

This perspective motivates a possible direction for adaptive AI systems.

Instead of modifying only the input context, future memory mechanisms may attempt to influence the representation process itself.

A standard attention calculation:

Attention= softmax(\\frac{QK^T}{\\sqrt d})V

could be extended with a dynamic modulation:

Attention= softmax(\\frac{QK^T}{\\sqrt d}+M)V

where (M) represents experience-dependent influence.

At the current stage, such mechanisms should be understood as adaptive attention modulation rather than true representation evolution.

However, they raise an important possibility:

Could long-term memory eventually become a mechanism that changes the geometry of cognition itself?

---

7. Intelligence as Structure Preservation Under Transformation

A deeper connection appears.

Physics studies:

How does the universe preserve structure while changing?

Biology studies:

How does life preserve identity while adapting?

Cognitive science studies:

How does the self remain continuous despite constant change?

AI research studies:

How can artificial systems maintain useful representations?

These may be different expressions of the same question:

How can a system transform while preserving meaningful structure?

From this perspective, intelligence is not simply computation.

It is the ability to maintain identity through transformation.

---

8. Open Questions

Several questions remain open.

1. Are attention patterns related to deeper symmetry structures?

Can attention heads be understood as learned relational components with mathematical properties similar to representations?

---

2. Can memory modify representation geometry?

Can future AI memory systems move beyond retrieval and modify how concepts relate internally?

---

3. Can AI representations become time-dependent?

Can we design systems where:

\\rho(t)

changes through experience while maintaining continuity?

---

4. Is intelligence fundamentally about invariance?

Could intelligence itself be understood as discovering and maintaining meaningful structures under transformation?

---

Conclusion

Group theory does not provide a complete theory of Transformer architectures.

Transformers are not simply hidden implementations of Lie groups.

However, group theory provides a powerful conceptual language:

• symmetry
• transformation
• representation
• invariance

These concepts may help us understand a deeper question:

How do intelligent systems maintain structure while continuously changing?

Perhaps intelligence is not merely the accumulation of information.

Perhaps intelligence is the ability to continuously transform representations while preserving meaning.

The future of AI may not only depend on larger models.

It may depend on systems capable of evolving their own representational structures through experience.

---

Written as an exploration of ideas at the intersection of group theory, representation learning, and adaptive AI systems.

[John6666]

For now, I’ll just try to organize the idea a little (I’m not entirely confident about the content, so please take this just as a reference…):

---

My reading is that several related, but technically distinct, ideas are being discussed here:

1. Group theory as a conceptual language for transformations, invariants, and structure preservation.
2. Formal equivariance, where a group, its action, and the expected transformation law are explicitly defined.
3. RoPE as a concrete rotational construction, where the group/Lie-algebra connection is relatively direct.
4. Experience-dependent memory, where a stored state changes how later information is processed.

The fourth part—the “Afterglow” or experience-dependent contribution to attention—looks like the easiest place to start experimentally. It can be tested without first proving that attention heads form group representations or completing a general theory of intelligence.

A practical default route might be:

1. Keep a small base model frozen.
2. Define an explicit memory state.
3. Separate write, read, decay, and reset.
4. Let the memory state generate a modulation for the current attention computation.
5. Compare it with no-memory, context-only, random, shuffled, incorrect-memory, and reset controls.
6. After the write/read behavior is understood, test it under one small, explicit symmetry.

One distinction seems especially useful:

The persistent object may not be the pairwise attention matrix itself. A persistent state can store the experience, while a temporary attention modulation is generated for the current queries and keys.

For example:

S_{t+1}=U(S_t,e_t)

M_t=G(S_t,Q_t,K_t)

A_t= \operatorname{softmax} \left( \frac{Q_tK_t^\top}{\sqrt{d}}+M_t \right)V_t

Here, S_t is the persistent state, U is the write/update rule, and M_t is generated for the current attention calculation.

This also exposes a central design choice:

• Content memory: information absent from the current context is stored and later supplied to the model.
• Salience/routing memory: past experience changes which currently available information receives attention.
• Hybrid memory: stored content is retrieved, while M_t also changes how that content is used.

An additive M_t naturally fits the second interpretation. It changes priorities among the current value vectors, but by itself it cannot insert past content that is absent from the present computation. Content storage would usually require an additional state, memory token, key/value memory, generated representation, or parameter update.

So I think the broad questions are testable, but at different levels:

Question	A testable first version
Are attention relations connected to symmetry?	Define one transformation and measure invariance/equivariance.
Can experience change later processing?	Write an explicit state, remove the original context, and compare later behavior.
Can memory change representation geometry?	Measure attention, hidden states, behavior, and interventions separately.
Can representations become time-dependent?	Let a persistent state modify the representation function or routing rule.
Is intelligence fundamentally about invariants?	Keep this as a broader interpretation; one architecture experiment cannot settle it.

The default path I would try is adaptive memory first, then a controlled symmetry test.

1. Symmetry, RoPE, and what would make the group-theoretic part formal

It may help to distinguish three levels of using group-theoretic language.

Level 1: conceptual symmetry

Terms such as transformation, invariant, orbit, and structure preservation are used to organize an intuition.

Examples include:

• a sentence changes form while preserving a selected meaning;
• an object rotates while preserving identity;
• a memory system updates while preserving selected capabilities.

This can be useful even before an exact group representation is defined.

Level 2: diagnostic equivariance

The architecture is not necessarily designed to be equivariant. Instead, one defines a transformation and measures whether the model behaves consistently under it.

For an input transformation g, one can compare:

• the original output;
• the output after transforming the input;
• the expected transformed output;
• hidden-state differences;
• memory-state differences;
• attention or routing differences.

This produces an empirical equivariance error rather than assuming an exact symmetry.

For natural language, it may be easier to define which property should remain invariant before trying to define a universal group of semantic transformations. Language Invariant Properties in Natural Language Processing follows this kind of route for transformations such as translation and paraphrasing.

That is useful because natural-language transformations are often:

• context-dependent;
• many-to-many;
• only approximately meaning-preserving;
• not cleanly invertible;
• valid for one property but not another.

So “meaning-preserving transformation” can initially be an experimental relation with a specified invariant, rather than one universal exact group.

Level 3: architectural equivariance

A formal architectural claim needs at least:

• a group G;
• an action of G on the input;
• an action of G on the representation or output;
• an invariant or equivariant transformation law.

Group Equivariant Stand-Alone Self-Attention is a concrete example: it constructs self-attention equivariant to specified symmetry groups through group-compatible positional encodings.

LieTransformer is another example for data associated with Lie-group structure.

These works mainly concern known geometric transformations, particularly in vision, point clouds, and physical systems. They do not establish that semantic relations or attention heads are irreducible representations, but they show what a formal version of the proposal would need to specify.

RoPE is the strongest existing bridge

The RoPE section appears to have the clearest direct connection to existing group-based constructions.

RoFormer applies position-dependent rotations to queries and keys so their inner products contain relative-position information.

LieRE generalizes RoPE from fixed two-dimensional rotations to learned higher-dimensional rotations generated from Lie-algebra elements. Its experiments focus mainly on two- and three-dimensional vision, so it is not evidence for experiential or semantic memory, but it is a concrete example of moving from a group-theoretic observation to an implemented Transformer component.

A useful separation may therefore be:

• RoPE/LieRE: a concrete positional-symmetry construction;
• Afterglow: an experience-conditioned memory or routing mechanism;
• the broader interpretation: a hypothesis about intelligence and preserved structure.

They can later be connected, but they do not have to be proved together.

A controlled hybrid experiment

If the goal is to connect memory and symmetry, a small controlled transformation may be easier than unrestricted paraphrasing.

Possible starting domains:

• graph-node relabeling;
• entity-slot permutation;
• symbolic-variable renaming;
• renaming variables in code;
• reordering independent code statements.

Suppose a transformation is represented by a permutation matrix P_g. One possible test for a pairwise modulation is:

M(gx)=P_gM(x)P_g^\top

This would establish only a narrow result: the memory-generated relation matrix transforms consistently under one known symmetry. It would not establish a universal theory of semantic memory.

For a causal language model, arbitrary token permutation can destroy causal order, so the transformation should preserve dependencies or first be tested in an encoder-style or synthetic setting.

Attention heads and irreducible representations

Some heads appear to specialize empirically, but specialization alone would not establish that one head is one irreducible representation.

Head labels are not canonical; heads can often be permuted together with the corresponding output projection, functions may be distributed across several heads, and some heads may be redundant. A stronger representation-theoretic claim would need a basis-independent transformation law for a subspace, rather than a semantic label assigned to “head 3” or “head 7.”

Head specialization can therefore be a clue, but not yet the decomposition itself.

2. What is remembered, what changes, and which nearby mechanisms are relevant

The word “memory” covers several different mechanisms.

Mechanism	Persistent object	How it affects later processing
Context memory	Past text/tokens	Reinserted into the prompt
KV memory	Stored keys and values	Retrieved or attended to directly
Recurrent memory tokens	Hidden vectors	Passed between segments
Attention prior	Bias or routing state	Changes attention scores
Fast weights	A parameter matrix or small model	Changes the computation
Adapter state	Adapter/low-rank parameters	Changes the representation map
External structured memory	Text, records, or a graph	Retrieved into context/features

The proposed M_t is closest to an attention prior unless it is paired with a component that supplies stored content.

Why store S, not a fixed dense M

A complete pairwise matrix depends on the current batch, heads, query positions, key positions, and cache length. A later prompt will normally have different tokens and dimensions.

A reusable state S_t could instead be:

• memory vectors or tokens;
• a fixed-size state matrix;
• a low-rank factorization;
• an associative key/value map;
• a small neural network;
• adapter parameters;
• records with timestamps and provenance.

M_t can then be generated from S_t and the present input.

Four contracts to define

Write

What qualifies as an experience, and what performs the update?

Possible signals include:

• prediction error or reconstruction loss;
• novelty/surprise;
• explicit relation labels;
• reward;
• user confirmation;
• a separate memory encoder;
• a Hebbian or delta-rule update;
• gradient descent on memory-only parameters.

Read

How is the state used?

Options include:

• add a bias to attention logits;
• add memory keys and values;
• add memory tokens;
• generate an adapter;
• gate heads or layers;
• retrieve records into context;
• combine retrieved content with an attention prior.

Forget or reconcile

What happens when memories are old, contradictory, duplicated, misleading, or irrelevant?

Possible policies include:

• decay;
• confidence-weighted replacement;
• timestamp precedence;
• provenance-aware merging;
• explicit deletion;
• rollback;
• separate episodic and consolidated storage.

Scope

Does the state persist for:

• one forward pass;
• one generated sequence;
• one document;
• one conversation;
• several sessions for one user;
• all users;
• model parameters permanently?

These scopes have very different meanings. Sequence-level state should not automatically be interpreted as cross-session continuity, and cross-session memory needs isolation and deletion rules that a sequence-level experiment does not.

What exactly changes?

Adding M_t directly changes routing priority. It does not necessarily change the representation map.

Intervention	Direct change	Conservative description
Add M_t to logits	Pairwise routing priority	Attention/relational geometry
Condition Q and K on memory	Similarity computation	State-dependent similarity
Add memory vectors	Available computational state	Augmented representation space
Generate adapters/fast weights	Representation function	Time-dependent representation map
Update base weights	Learned model function	Learned representation change

There is also a mathematical distinction: a query-key score is not automatically a distance metric, because queries and keys usually come from different projections. It need not be symmetric, positive, or satisfy the triangle inequality.

If d_M is intended as a literal metric, those properties would need to be specified. Otherwise, “state-dependent similarity” or “relational geometry” may be a lighter operational interpretation.

Nearby mechanisms

These are not identical to Afterglow, but each contributes a useful component or comparison.

• Forgetting Transformer applies a data-dependent forget gate to unnormalized attention scores. It is not persistent experiential memory, but it shows that dynamic score modulation is an existing design space.
• GradMem freezes model weights and writes context into a small set of memory tokens through test-time optimization. Its context-removal evaluation is especially relevant: after writing, the original context is unavailable during reading.
• δ-mem is a recent preprint with a compact online associative state that generates low-rank corrections to attention in a frozen backbone. It is particularly close to the S_t -> M_t separation, though it is too recent to treat as a settled standard.
• Memorizing Transformers stores and retrieves past key/value representations, providing a content-memory baseline.
• Recurrent Memory Transformer passes memory-token states between segments.
• Learning to (Learn at Test Time) treats the hidden state as a model updated on the test sequence, representing a stronger form of time-dependent adaptation.
• Hopfield Networks is All You Need connects attention to modern associative-memory updates and provides vocabulary for storage, retrieval, interference, and attractor states.

This suggests that the distinctive part of Afterglow is probably not merely “add another term to attention.” It is more likely to be the combination of:

• what experience is stored;
• how it updates S;
• how S generates M_t;
• how long it persists;
• how conflict, decay, and reset work;
• whether the effect is selective and reusable on later inputs.

3. A minimal prototype and controls that separate memory from arbitrary bias

A first experiment can be intentionally small. Its purpose would be to determine whether an experience-conditioned state has a selective, persistent, and reversible effect—not to prove the full philosophical interpretation.

Minimal task

A synthetic relation task may be enough:

1. Present an experience such as an arbitrary entity-relation association.
2. Write it into S.
3. Remove the original experience from the context.
4. Present a later query with optional distractors.
5. Generate M_t from S, the current queries, and the current keys.
6. Measure whether the correct relation is recovered.
7. Reset S and confirm that the effect disappears.

Possible task families:

• entity-to-attribute associations;
• relation chains;
• variable renaming;
• graph-node relabeling;
• controlled code transformations;
• paraphrase pairs after the synthetic version works.

Comparison conditions

Condition	What it controls
No memory	Base-model behavior
Context-only	Ordinary prompt reinsertion
Experience-conditioned S and M_t	Proposed mechanism
Capacity-matched memory tokens	Alternative writable state
Scale-matched random M_t	Arbitrary perturbation
Shuffled M_t	Dependence on pair placement
Wrong/counterfactual memory	Dependence on content
Unrelated memory	Selectivity
Oracle memory	Whether reading works when writing is perfect
Reset/decay	Reversibility and forgetting
Old fact then new fact	Updating/supersession
Reversed or shuffled writes	Order dependence

Match capacity and compute

Capacity controls should match as far as possible:

• memory-token count;
• low-rank dimension;
• stored bytes;
• number of variable parameters.

Compute controls should match:

• write steps;
• forward passes;
• gradient steps;
• read/write latency.

Otherwise, an apparent gain may simply come from more storage or computation.

Context-removal test

This is probably the clearest first distinction.

• The effect remains after the original experience is removed
  → evidence that a separate state retained usable content.

• The effect appears only when related content is still present
  → evidence for salience/routing memory.

• Both occur
  → evidence for a hybrid mechanism.

Leakage test

The writer should not receive the future query, target answer, future tokens, or labels unavailable at deployment time. A causal model can otherwise appear to memorize when information leaked into the write phase.

Additive-bias sanity checks

Row-constant shifts should do nothing

For a single query, adding the same constant to every key score does not change softmax:

\operatorname{softmax}(z+c\mathbf{1}) = \operatorname{softmax}(z)

Therefore:

• subtract the key-wise row mean from M_t;
• confirm that the result is numerically equivalent;
• compare centered biases rather than raw norms;
• verify that the effect comes from relative differences among keys.

Compare M_t with the base-logit scale

A useful diagnostic is:

r= \frac{\operatorname{std}(M_t)} {\operatorname{std}(QK^\top/\sqrt{d})}

Log this per head and layer.

• very small r: the modulation may be negligible;
• comparable r: it competes with the original attention pattern;
• very large r: it may overwrite the model rather than modulate it.

Sweep the strength

Evaluate:

M_t^{(\alpha)}=\alpha M_t

for zero and several positive values, and optionally negative values.

A convincing pattern would show improvement over a reasonable range, increasing side effects when the modulation becomes too strong, and baseline recovery after reset.

Measurements

Separate four levels.

Behavior

• accuracy;
• target-token log probability;
• calibration/abstention;
• unrelated-task performance.

Attention

• pre-softmax logits;
• entropy;
• divergence from baseline;
• layer/head distribution.

Hidden representations

• pairwise similarity;
• neighborhood structure;
• probes;
• representation-similarity measures.

Intervention

• remove M_t;
• shuffle it;
• replace it with wrong memory;
• ablate selected layers/heads;
• restore an older state;
• reset completely.

Intervention matters because an attention map may change without causing the output change.

Conservative interpretation

Observation	Supported conclusion
Proposed condition beats no-memory	The state affects behavior
It beats context-only	It may offer more than prompt reinsertion
It beats random/shuffled controls	Structured modulation matters
It beats wrong-memory control	Experience content matters
It survives context removal	Information exists outside the prompt
It preserves unrelated tasks	The effect is selective
Reset removes the effect	The stored state is causally involved
A defined transformation law holds	That specific equivariance is supported
Hidden states and behavior change under intervention	Stronger evidence of functional representation change

Together these can establish a well-defined adaptive-memory mechanism, without requiring the first experiment to establish a general theory of identity or intelligence.

4. Evaluation ladder and claim boundaries

It may be easier to scale evaluation in stages.

Stage 0: unit tests

Before testing memory quality:

• verify causal and padding masks;
• verify no future-token leakage;
• verify M_t=0 reproduces the baseline;
• verify reset restores the baseline;
• verify the row-shift softmax identity;
• verify cached and uncached generation within expected numerical tolerance.

Stage 1: synthetic write/read

Measure:

• capacity;
• context-removal accuracy;
• interference;
• order sensitivity;
• update and overwrite behavior;
• decay and reset.

This is the best stage for distinguishing content memory from routing memory.

Stage 2: controlled symmetry

Apply a defined transformation such as entity renaming, graph relabeling, variable renaming, or reordering independent statements.

Test whether:

• the output property is preserved;
• S changes consistently;
• M_t follows the expected transformation;
• the behavior generalizes to unseen transformations.

This is where the group-theoretic part can become experimentally concrete.

Stage 3: broader memory tasks

Resource	Useful for	Important limit
BABILong	Long-context retrieval and synthetic reasoning	Does not establish cross-session memory
LoCoMo	Long multi-session conversations and event recall	Small, curated conversational setting
LongMemEval	Multi-session reasoning, updates, temporal reasoning, abstention	Aggregate scores can hide pipeline failures
MemoryAgentBench	Retrieval, test-time learning, long-range understanding, forgetting	Covers several mechanisms, not only internal attention state
MemFail	Diagnosing summarization, storage, and retrieval failures	Mainly designed for explicit memory systems

The useful lesson from diagnostic benchmarks is to separate failure stages:

1. Was the experience encoded correctly?
2. Was the state retained?
3. Was the correct memory selected?
4. Was it interpreted correctly?
5. Was the final response generated correctly?

Claim boundaries

Evaluation	It can support	It does not establish
Synthetic association	Basic write/read behavior	General semantic memory
Context removal	Information outside the prompt	Cross-session continuity
Long-context task	Use of distant information	Personal identity
Multi-session benchmark	Updating and session memory	Human-like continuity
Symmetry test	One specified equivariance	Universal semantic invariance
Representation analysis	Measured internal differences	A complete theory of meaning
Persistent user state	Long-lived adaptation	Safety or correctness

Measuring representation change

Attention weights alone are weak evidence for a geometric claim. Separate:

1. routing change;
2. activation change;
3. functional behavior;
4. causal intervention;
5. transformation-consistent structure.

Similarity of Neural Network Representations Revisited introduced CKA as a useful comparison measure, while Grounding Representation Similarity with Statistical Testing shows why similarity measures should be checked for both sensitivity and specificity.

Likewise, Attention Is Not Explanation and Attention Is Not Not Explanation illustrate that attention maps are observations whose explanatory value depends on controls and interventions.

A stronger result would combine:

• multiple representation measures;
• behavioral change;
• removal/ablation;
• an unrelated-task control;
• a predefined transformation test.

5. Hugging Face/PyTorch implementation and persistence notes

Begin with correctness

For a first prototype:

• use a small model and short sequences;
• use eager attention or a verified SDPA path;
• log pre-softmax attention scores;
• keep the hard visibility mask separate from the soft memory modulation;
• add deterministic unit tests;
• optimize only after cached generation is correct.

Hugging Face Attention Interface

The current Hugging Face Attention Interface supports custom attention functions and forwarding additional keyword arguments into supported attention layers. That provides one route for passing M_t or a memory-state readout.

A particularly important detail is the matching mask interface. If a custom attention implementation is registered without a corresponding AttentionMaskInterface formatter, Transformers can skip mask creation and pass attention_mask=None. The custom implementation must then handle causal, padding, packing, and sliding-window constraints itself, or those constraints may be silently dropped.

Hard visibility and soft memory modulation should therefore remain separate:

• hard mask: which positions may interact;
• soft M_t: how strongly allowed positions interact.

Float-mask convention

For a documented four-dimensional additive float mask:

• 0.0 means attend;
• negative infinity means mask.

A two-dimensional padding mask often uses 1/0. Simply converting that mask to float would not create the required additive mask: 1.0 becomes a positive score bias, while 0.0 still permits attention.

This is an important leakage test for a causal memory experiment.

Backend differences

Mask representations differ among eager attention, SDPA, FlashAttention, and FlexAttention. The implementation should therefore be tested for the selected backend rather than assuming one mask object is portable.

PyTorch FlexAttention provides a score_mod hook for changing scores before softmax. It may be a natural later implementation for M_t, especially if the modulation is sparse, low-rank, block-structured, or computed lazily. A simpler implementation is still preferable for the first correctness checks.

KV-cache interaction

The Hugging Face cache documentation explains that autoregressive generation processes new query tokens while retaining past keys and values. Each layer stores key/value tensors whose sequence dimension grows or follows a cache policy.

M_t must therefore align with:

• current query positions;
• cached key positions;
• the current layer/head structure;
• sliding-window or eviction policy;
• batched sequences with different lengths.

This is another reason to generate M_t from S_t, rather than persisting one dense pairwise matrix.

Scaling options

A dense M_t scales quadratically with sequence length. Possible alternatives:

• low-rank factors;
• sparse selected relations;
• per-token salience;
• per-head gates;
• memory-conditioned query/key shifts;
• blockwise modulation;
• a small set of memory vectors.

Goal	Reasonable first form
Feasibility on short sequences	Dense M_t
Longer generation	Low-rank/sparse M_t
Content memory	Memory vectors or KV state
Salience memory	Score bias/gate
Representation adaptation	Adapter or fast weights
Formal equivariance	Structure-constrained generator

Persistence lifecycle

If state survives across sessions, lifecycle rules are part of the mechanism:

• scope;
• source/provenance;
• timestamp;
• confidence;
• conflict policy;
• decay;
• deletion;
• reset;
• rollback.

A toy experiment does not have to solve all of these. It is still useful to design the state interface so they can be added later without replacing the entire mechanism.

Putting the routes together:

• Formal symmetry first: define a group action and invariant, then constrain the architecture.
• Adaptive memory first: define S, write/read/reset, and generate M_t.
• Hybrid: test the memory mechanism under a small explicit symmetry.

My default starting point would be adaptive memory first, followed by the hybrid route:

1. show that a separate state can be written, read, reset, and distinguished from arbitrary bias;
2. determine whether it stores content, salience, or both;
3. verify updating, interference, and unrelated-task behavior;
4. test it under one controlled transformation;
5. only then expand toward unrestricted language, long-term adaptation, or broader claims about representation geometry.

Even a negative result would narrow the problem: it could show whether the difficult part is writing useful experience, generating the right modulation, avoiding interference, preserving unrelated capabilities, or identifying a meaningful invariant.

That seems like a manageable path from the conceptual proposal to falsifiable and reusable experiments, without requiring the broader philosophical interpretation to be settled first.

[rongxianlingyi]

Thank you for this very detailed and technically grounded response. It helped us separate several ideas that were mixed together in our original post.

I think your framing is correct: RoPE / LieRE, formal equivariance, attention-head interpretation, and experience-dependent memory should not be forced into one claim too early. They can later be connected, but experimentally the most practical starting point is adaptive memory.

We have now organized our current Afterglow results more clearly.

Our project originally started with a direct M-bias attention injection design on Mistral-7B:

Attention=\\operatorname{softmax}(QK^T/\\sqrt d + M)V

The initial plan compared several conditions:

• no-memory baseline;

• prompt-prefix memory;

• token-cloud prompt;

• M-bias injection with different (\lambda) values.

The early result was mixed but useful:

• high (\lambda) made identity-related tokens appear more consistently and improved directional consistency compared with baseline;

• too high (\lambda) degraded generation quality and made output more repetitive;

• low/mid (\lambda) was often not clearly better than prompt-prefix injection;

• most importantly, static M-bias did not behave like real memory, because it did not change through use, decay, conflict, or reactivation.

That pushed us to a major shift:

Instead of treating Afterglow as only “attention bias injection,” we now treat it as an external memory metabolism layer.

So your distinction between persistent state (S_t) and temporary modulation (M_t) is exactly the right framing for us.

At the current stage, our main persistent object is not a dense pairwise (M_t). It is a structured external memory state. The system stores typed candidate memories and cards, tracks recall, handles conflicts, applies decay/archive policies, and then builds a memory prefix or, in selected experiments, can provide attention modulation.

The current architecture has several modules. The most important ones are:

• M1 / EntropyTracker: tracks information flow and attention jumps;

• M2 / ReflectionTrigger: extracts candidate facts and decides whether reflection/write should be triggered;

• M3 / ReflectionExecutor / MemorySystem: performs memory writing, conflict resolution, forgetting, card materialization, and memory-prefix construction.

This part is influenced by our earlier EDARE direction. In practice, EDARE-style signals such as entropy change, attention jumps, semantic retention, and reflection triggers are used as part of the upstream judgment about whether something is memory-worthy. We do not treat entropy as a proof of consciousness or identity. We use it more conservatively as one signal for information novelty, instability, or reflection need.

The current memory layer supports:

• candidate fact extraction;

• memory types: identity_fact, relationship_fact, world_fact, episodic_fact;

• subject-relation-target slot extraction;

• identity_impact estimation;

• scene-related fields such as scene_link, emotional_tone, and temperature_preservation;

• protected memory with explicit conflict/reflection checks;

• confirmation-echo filtering so that ordinary AI repetition does not become false memory;

• scene capsule promotion;

• weighted memory-prefix injection;

• recall tracking, decay, archive, and possible reactivation;

• automatic card materialization into relationship / identity / world structures.

So I agree with your classification:

• additive (M_t) alone is best understood as salience/routing memory;

• structured external memory supplies content;

• a practical Afterglow system is probably hybrid: content-bearing external memory plus optional salience/routing modulation.

Our strongest claim right now is therefore not “we achieved dynamic representation evolution.”

A more conservative claim is:

We have implemented a structured memory metabolism layer where experiences can be extracted, typed, written, weighted, protected, recalled, archived, and partially reintroduced into later context. The original M-bias mechanism remains an experimental attention-modulation branch, but the current engineering baseline is structured external memory plus weighted context injection.

We also agree that the next step should be stricter experimental controls. We already have unit tests and full-chain regression tests for candidate fact extraction, slot extraction, memory lifecycle, protected memory, confirmation-echo filtering, scene capsule promotion, graph recall, and app-level multi-turn behavior.

But we still need a more controlled experimental layer to separate memory from arbitrary bias or extra context.

The next evaluation plan should include:

• no-memory;

• context-only;

• random memory;

• shuffled memory;

• wrong/counterfactual memory;

• unrelated memory;

• reset/decay;

• old fact then new fact;

• context-removal tests;

• long-session stress tests.

I also agree with your point that we should distinguish:

1. content memory;

2. salience/routing memory;

3. representation adaptation.

Right now, our baseline is strongest in content/structured memory. M-bias is closer to salience/routing. We have not yet shown true representation adaptation in the stronger sense of changing the model’s representation map.

A good falsifiable next target may be:

Can an experience-conditioned external state produce selective, persistent, reversible, and content-sensitive effects on later behavior after the original context is removed?

After that, we can test whether a memory-generated (M_t) improves over structured memory alone, and then test it under one controlled symmetry such as entity renaming, graph relabeling, or variable renaming.

So the updated roadmap is:

1. establish structured external memory as (S_t);

2. verify write/read/reset/context-removal behavior;

3. distinguish content memory from routing memory;

4. add generated (M_t=G(S_t,Q_t,K_t)) as an optional modulation path;

5. test under controlled transformations;

6. only then discuss broader claims about representation geometry or time-dependent representation (\rho(t)).

Your response helped us sharpen our claim boundary.

We no longer want to present Afterglow as “proof of AI identity” or “proof of consciousness.” A better description is:

Afterglow is an attempt to build a memory metabolism layer: a system where information is not merely stored or retrieved, but written, weighted, protected, recalled, forgotten, archived, and reactivated through interaction.

If this mechanism later begins to change routing or representation geometry in a measurable way, that would be the next research stage.

Thank you again. Your feedback gave us a much clearer path from philosophical intuition to falsifiable implementation.

[rongxianlingyi]

Thank you for the extraordinary depth of your reply. Your breakdown of the four levels—conceptual symmetry, diagnostic equivariance, architectural equivariance, and experience-dependent memory—helped us see exactly where our framework sits and what needs to be validated next. we have completed a round of Afterglow experiments (M1–M7 pipeline, 19 unit tests, 4 chain regression tests, 1 baseline overview, all green). The key finding: M-bias as a static attention perturbation gives way to an external memory metabolism layer (M2→M3→weight-sorted prefix injection) that more closely approximates your “salience/routing memory”—past experience changes which currently available information receives attention, rather than inserting absent content. We have also formalized the Wu Xing (Five Elements) theory of TCM as a C₅-based structural model, with the Sheng-Ke cycles as two distinct traversal steps over ℤ₅. This paper has gone through multiple rounds of revision with rigorous feedback (thanks in part to perspectives like yours) and is now ready for arXiv submission. However—this is our first arXiv submission, and we need an endorsement. If you are an active arXiv author, or know someone who is, and would be willing to review the manuscript and consider endorsing, we would be deeply grateful. Full draft available on request. Either way, thank you for the depth of thought in your original reply—it genuinely shaped how we think about the memory/attention/representation boundary.

[John6666]

Oh. Hmm… I’m not an academic, and I don’t really have academic contacts, so I’m not very familiar with the etiquette here…:

---

Thank you for the detailed follow-up. I’m glad the earlier organization was useful.

The revised framing sounds much clearer to me. In particular, separating:

• structured external content memory;
• optional salience/routing modulation;
• and the stronger idea of representation adaptation;

makes the current implementation easier to understand and evaluate.

The falsifiable target you proposed also seems like a sensible center:

Can an experience-conditioned external state produce selective, persistent, reversible, and content-sensitive effects on later behavior after the original context is removed?

That appears much easier to test than trying to validate the broader interpretation all at once.

Regarding the arXiv endorsement request, though, I should be clear: I cannot endorse the submission or introduce you to a potential endorser. I do not have practical experience with arXiv submission workflows, and I do not have an academic network I could draw on.

I may be able to contribute a little through public discussion here on the forum, especially when there is a specific technical or evaluation question to look at. But I cannot realistically offer private review, ongoing one-to-one help, or assistance contacting potential endorsers.

From the official documentation, the safest route seems to be to follow arXiv’s own endorsement process closely and make the manuscript package easy for an appropriate subject-area author to inspect.

What I found in the official arXiv guidance

Endorsement is not peer review

The official arXiv endorsement documentation explicitly says that endorsement is not peer review.

An endorser is not expected to:

• verify every result;
• reproduce the experiments;
• establish that the argument is correct;
• or perform a detailed journal-style review.

The endorser should either know the submitter or see the intended paper, and should check that the submission is appropriate for the relevant subject area and connected to current work in that area.

That distinction may be useful when approaching someone. Asking a stranger to perform a full manuscript review and then endorse it is a much larger request than asking whether they would be willing to inspect the paper’s subject-area fit and consider endorsement.

Moderation is not peer review either

The official arXiv moderation documentation also states that moderation is not peer review and that arXiv staff and moderators cannot provide submission feedback.

Moderation checks matters such as:

• scholarly form and preparation;
• relevance to the disciplines served by arXiv;
• category suitability;
• compliance with arXiv policies;
• whether the submission meets basic scholarly standards.

A paper can therefore need three different kinds of scrutiny:

Process	Main purpose
Endorsement	Establish that a new submitter and intended submission belong in the relevant research area
arXiv moderation	Check policy, scholarly form, relevance, and category placement
Peer review	Evaluate research quality and suitability through a journal, conference, or other review venue

I only understand these distinctions from the outside; I do not have practical experience navigating them.

The endorsement policy changed in 2026

According to arXiv’s January 21, 2026 policy update, an institutional email address alone is no longer sufficient for automatic endorsement of new submitters.

The two current paths are broadly:

1. an eligible institutional email and prior authorship on an arXiv paper in the relevant endorsement domain; or
2. personal endorsement from an established arXiv author in the same endorsement domain.

If the first route does not apply, the personal-endorsement route appears to be the relevant one.

Official process for seeking personal endorsement

The official instructions suggest:

1. Start a new arXiv submission.
2. Select the intended category.
3. Receive the endorsement-request email and link.
4. Find closely related papers already on arXiv.
5. Open their abstract pages.
6. Use the “Which authors of this paper are endorsers?” link.
7. Contact a small number of genuinely relevant eligible authors.
8. Include the official endorsement-request information and, if appropriate, the intended submission or a scholarly profile.

The documentation explicitly says that it is inappropriate to:

• email large numbers of potential endorsers at once;
• or repeatedly email the same person requesting endorsement.

So a small, relevance-based list seems preferable to broad outreach.

A compact package that may make an endorsement request easier to inspect

A potential endorser probably should not have to reconstruct the paper’s scope from a long message.

A compact package could contain:

1. One-paragraph abstract

This should state:

• the problem;
• the implemented mechanism;
• the main evidence;
• the strongest supported conclusion;
• and the main limitation.

2. Intended arXiv category and endorsement domain

The message should state the intended category directly.

It may also help to explain in one or two sentences why the work belongs there. The endorser’s role is partly to assess subject-area appropriateness, so making that relationship explicit reduces their work.

3. Full manuscript

Preferably in a stable form with:

• title;
• author names;
• abstract;
• numbered sections;
• figures and tables;
• complete references;
• limitations;
• and an appendix where appropriate.

4. Short contribution summary

For example:

• what was implemented;
• what was measured;
• what was compared;
• what is proposed but not yet demonstrated;
• what is philosophical interpretation rather than experimental result.

5. Closely related work

A short list of perhaps three to six genuinely close papers may be more useful than a very broad bibliography in the initial message.

For each one, the manuscript should make clear:

• what mechanism is similar;
• what differs;
• what component was reused or adapted;
• what experimental question remains distinct.

This also gives a concrete reason for approaching a particular author.

6. Evaluation summary

A small matrix may be easier to inspect than several pages of prose.

Evidence type	Current status
Unit tests	Implemented / count and scope
Full-chain regression tests	Implemented / count and scope
No-memory baseline	Complete / incomplete
Context-only baseline	Complete / incomplete
Random-memory control	Complete / incomplete
Shuffled-memory control	Complete / incomplete
Wrong-memory control	Complete / incomplete
Context-removal test	Complete / incomplete
Update / supersession test	Complete / incomplete
Reset / decay test	Complete / incomplete
Long-session stress test	Complete / incomplete
Controlled-symmetry test	Complete / incomplete

This prevents “all tests green” from being mistaken for “the research hypothesis has been validated.”

Both are useful, but they establish different things:

• unit and regression tests show that the implementation behaves according to its software contract;
• research controls show whether the proposed mechanism explains the observed effect better than alternatives.

7. Code or reproducibility material

If available:

• repository;
• commit or release tag;
• environment file;
• model and dataset versions;
• commands for the central experiment;
• expected output;
• random seeds;
• hardware assumptions;
• license;
• known nondeterminism.

A potential endorser may not run the code, but visible reproducibility material can make the implementation claim easier to understand.

8. Limitations

For this project, relevant limitations might include:

• the current baseline is strongest as structured external memory;
• prefix injection carries content through context reconstruction;
• static M-bias alone did not behave like evolving memory;
• representation adaptation has not yet been demonstrated;
• results may depend on the base model and memory-extraction pipeline;
• unit/regression success is not yet a substitute for controlled comparative evaluation;
• the broader identity or consciousness interpretation is outside the current empirical claim.

Making these limitations visible would probably strengthen rather than weaken the submission.

9. Official endorsement request link or code

The request should include the information generated by arXiv after starting the submission and selecting the intended category.

A possible structure for contacting a relevant eligible author

This is not a universal template, but a short message might be easier to respond to than a broad request for mentorship or full peer review.

Subject: arXiv endorsement request for <category/domain>

Dear <name>,

I am preparing a first arXiv submission in <category>. I found your work on <specific paper/topic> closely related to our treatment of <specific mechanism or question>.

The paper proposes <one-sentence contribution>. Our current empirical claim is limited to <supported claim>; we do not claim <important excluded claim>.

The manuscript, abstract, evaluation summary, and code/reproducibility materials are available here:

- Draft: <link>
- Code or project page: <link>
- Short evaluation summary: <link or brief description>

If you consider the manuscript appropriate for this arXiv subject area, would you be willing to inspect it and consider providing the requested endorsement?

The official arXiv endorsement request is: <request link/code>

I understand that endorsement is not peer review, and I do not expect a detailed review. I would also completely understand if you are unable to consider the request.

Thank you for your time.

A few principles seem important:

• explain why that particular person was contacted;
• mention one genuinely relevant paper or research connection;
• state the claim boundary;
• do not imply that endorsement means agreement with every conclusion;
• make declining easy;
• avoid repeated follow-ups;
• do not send the same generic message to a large list.

How I would package the current technical story

The updated architecture seems to contain several tracks that should remain visibly distinct.

Track A: structured external memory

The currently implemented baseline appears to include:

• candidate-memory extraction;
• typed facts or cards;
• relation-slot extraction;
• conflict handling;
• protected memory;
• recall tracking;
• decay and archive policies;
• reactivation;
• weighted prefix construction.

This is the strongest current engineering claim.

A conservative description such as structured memory-metabolism layer seems reasonably aligned with what is actually implemented, provided each lifecycle operation is defined operationally.

Track B: salience/routing modulation

The original static M-bias experiments showed:

• stronger directional effects at high modulation strength;
• degradation or repetition when the strength became too high;
• limited advantage over prefix injection at lower strengths;
• and no genuine lifecycle behavior such as decay, conflict, or reactivation.

That makes M-bias useful as an experimental routing branch, but not sufficient by itself as the complete memory mechanism.

A later generated modulation:

M_t=G(S_t,Q_t,K_t)

would be a separate experimental stage rather than a description of the current baseline.

Track C: representation adaptation

Changing the model’s representation map would be a stronger claim than:

• storing structured content;
• reconstructing a weighted prefix;
• or changing attention priorities.

It may require evidence such as:

• persistent changes in hidden representations;
• behavioral intervention;
• removal/reset controls;
• unrelated-task preservation;
• and evidence that the change cannot be explained by added context alone.

The current statement that this has not yet been shown is an important and useful boundary.

Track D: formal symmetry and the C₅ model

The Wu Xing / C₅ formalization sounds like a separate formal-symmetry direction.

If it appears in the same manuscript as Afterglow, the connection probably needs to be very explicit:

• Does the C₅ structure constrain the memory update?
• Does it define memory types or transitions?
• Does it generate a measurable equivariance?
• Is it only an interpretive example?
• Does removing it change the Afterglow mechanism or evaluation?

If the two components can stand independently, separating them into clearly labeled tracks—or possibly separate manuscripts—may make each contribution easier to evaluate.

This is not necessarily an argument for splitting them. It is mainly a scope question: a reader should be able to tell whether one result logically or experimentally depends on the other.

A compact claim table

Statement	Suggested status
A structured external memory lifecycle has been implemented	Implementation claim
The pipeline passes unit and regression tests	Software-verification claim
Weighted prefix injection changes later behavior	Behavioral observation
The system stores content outside the immediate prompt	Requires context-removal evidence
M provides selective routing benefits beyond structured memory	Requires controlled comparison
The memory is reversible and update-sensitive	Requires reset, decay, and supersession tests
Representation geometry changes persistently	Not yet demonstrated
The system establishes identity or consciousness	Outside the present empirical claim
A C₅ structure provides useful formal constraints	Requires a defined connection and evaluation

This kind of table might make the abstract, introduction, limitations, and endorsement request more internally consistent.

What may still be useful to discuss publicly in this thread

If specific questions arise publicly here, useful topics for open discussion might include:

• whether a particular claim is supported by the reported experiment;
• whether a baseline separates memory from added context;
• whether a control tests the intended failure mode;
• whether a mathematical term is being used operationally or metaphorically;
• whether two components should be evaluated together or separately;
• whether a result is reproducible from the public materials;
• whether related work is close at the mechanism level rather than only thematically related.

I may occasionally be able to comment on a concrete question like that as another forum reader.

But to avoid giving the wrong expectation, this would only be informal public discussion. It would not be:

• endorsement;
• academic peer review;
• private manuscript review;
• ongoing individual support;
• submission management;
• or outreach to potential endorsers.

One additional practical note: if generative-AI tools were used substantially while preparing the manuscript, every citation, quotation, mathematical statement, experimental claim, and description of prior work should be checked against the original source. arXiv’s policy statement on generative-AI language tools places responsibility for the complete contents on the human authors and says significant use should be reported according to the standards of the subject area.

So, unfortunately, I cannot help with the endorsement or provide an academic connection. The most reliable route I can see is:

1. settle the intended category and endorsement domain;
2. prepare a concise, inspectable manuscript package;
3. identify a small number of genuinely relevant eligible authors through arXiv’s official process;
4. make a narrow and respectful endorsement request;
5. keep implementation claims, controlled evidence, future work, and broader interpretation visibly separate.

And from the technical side, I think the revised ordering remains sensible:

1. establish the structured external state;
2. test write/read/reset and context removal;
3. distinguish content memory from routing effects;
4. compare generated attention modulation against the structured-memory baseline;
5. test a controlled transformation;
6. only then revisit stronger claims about representation geometry.