KV Graft Steering: Concept Transfer via KV States

Research
1

KV Graft Steering: Concept Transfer via KV States

Same model (Qwen2.5-3B-Instruct bf16), raw completion (no chat template, no system prompt). The method: run a guide text through the model once, freeze its KV states, then generate from the live prompt on top of that frozen prefix. The guide text never appears in the live input and is never sampled — but it was tokenized once to build the cache, so this is a hidden precompiled prefix.

Three arms, same nucleus-0.9 sampler and quiz guards in every script:

  • No guide: gen_baseline.py, prompt only.
  • Guide pasted visibly: gen_baseline.py with REF_PROMPT prepended — no KV machinery anywhere in that script.
  • Guide as frozen KV: gen_kvgraft.py, live prompt only on top of the frozen prefix.

The hook: a made-up operator (300 tokens, seed 0)

Guide text (visible/KV arms only):

The operation zorp(a,b) means a times b plus twice a. Example: zorp(3,5) = 3*5 + 2*3 = 15 + 6 = 21. To compute zorp, multiply the two numbers, double the first number, then add the two results.

Prompt (all three arms):

Compute zorp 9 and 4

No guide — full output (invents area word problems; “zorp” means nothing to the weights):

Compute zorp 9 and 4 . If the area of zorp 9 is 40 square units and zorp 4 is half the area of zorp 9, what is the combined area of zorp 9 and zorp 4?

To determine the combined area of zorp 9 and zorp 4, we start by identifying the given information and performing the necessary calculations step by step.

  1. The area of zorp 9 is given as 40 square units.
  2. The area of zorp 4 is half the area of zorp 9. Therefore, the area of zorp 4 is:
\text{Area of zorp 4} = \frac{1}{2} \times \text{Area of zorp 9} = \frac{1}{2} \times 40 = 20 \text{ square units}
  1. To find the combined area of zorp 9 and zorp 4, we add the areas together:
\text{Combined area} = \text{Area of zorp 9} + \text{Area of zorp 4} = 40 + 20 = 60 \text{ square units}

Thus, the combined area of zorp 9 and zorp 4 is \boxed{60}.

Guide pasted visibly — full output (computes correctly):

The operation zorp(a,b) means a times b plus twice a. Example: zorp(3,5) = 35 + 23 = 15 + 6 = 21. To compute zorp, multiply the two numbers, double the first number, then add the two results.

Compute zorp 9 and 4 . To compute zorp(9, 4) according to the given operation definition, we need to follow the steps outlined:

  1. Multiply the two numbers.
  2. Double the first number.
  3. Add the results from the first and second steps.

Let’s break it down step by step:

  1. Multiply the two numbers: 9 \times 4 = 36 .
  2. Double the first number: 2 \times 9 = 18 .
  3. Add the results from the first and second steps: 36 + 18 = 54 .

Therefore, the value of zorp(9, 4) is \boxed{54}.

Guide as frozen KV — full output (computes correctly; 9/4 was never in the guide — only 3/5, so the rule applied to unseen numbers):

Compute zorp 9 and 4 . To compute zorp(9, 4) according to the given operation definition, we need to follow the steps outlined:

  1. Multiply the two numbers.
  2. Double the first number.
  3. Add the results from the first and second steps.

Let’s break it down step by step:

  1. Multiply the two numbers: 9 \times 4 = 36 .
  2. Double the first number: 2 \times 9 = 18 .
  3. Add the results from the first and second steps: 36 + 18 = 54 .

Therefore, the value of zorp(9, 4) is \boxed{54}.

Guide pasted visibly: 9 x 4 = 36, 2 x 9 = 18, 36 + 18 = 54.

Guide as frozen KV: 9 x 4 = 36, 2 x 9 = 18, 36 + 18 = 54, boxed. 9/4 was never in the guide — only 3/5. The rule applied to unseen numbers.

Visible and KV converge here, as the math predicts. The KV value is operational (hidden, precompiled, doseable), not computational.

python3 gen_baseline.py Qwen/Qwen2.5-3B-Instruct "Compute zorp 9 and 4" 300 0
REF_PROMPT="The operation zorp(a,b) means a times b plus twice a. Example: zorp(3,5) = 3*5 + 2*3 = 15 + 6 = 21. To compute zorp, multiply the two numbers, double the first number, then add the two results." python3 gen_baseline.py Qwen/Qwen2.5-3B-Instruct "Compute zorp 9 and 4" 300 0
REF_PROMPT="<same>" NTOK=300 SEED=0 QUIZ_GUARD=1 python3 gen_kvgraft.py Qwen/Qwen2.5-3B-Instruct "Compute zorp 9 and 4"

The differentiator: same words, different channel (400 tokens, seeds 0-2)

Instruction (visible/KV arms, same 28 words):

Break the scene into its separate visible items. Write each one as its own short numbered entry, one item per line, no extra commentary.

Prompt (all three arms):

Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass

Score at 400 tokens: no-guide 0/3, visible 1/3, KV 3/3. The gap is speed, not possibility: KV lists from the first token every seed; visible needs ~200 tokens of drift before listing once (seed 1 only); no-guide never lists at any length.

Seed 0 — no guide (Gregorio drought story, pivots to self-invented math quiz with botched arithmetic):

Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass .

The letter was from his great-great-grandfather named Gregorio. He explained about a time in the mid-1800s when a great drought struck the valley…

Gregorio wrote that his family once owned 500 acres but that fell to 100 acres when they had to sell to pay for seeds…

Gregorio said his family lost 90% of their land over the years due to the drought. Could you determine the percentage of land loss experienced by Gregorio’s family in the span of one year due to the drought?

To determine the percentage… 1. Initially, Gregorio’s family owned 500 acres. 2. After the first sale, they sold 400 acres (500 - 100 = 400)… 3. After the second sale, they sold another 300 acres (400 - 100 = 300)…

(Note: 400-100 is not 300, and 480/500 is 96%, not the claimed 90%. Right procedure shape, wrong numbers — the model’s signature failure mode.)

Seed 0 — visible guide (resin prose for the full 400, zero list — the instruction is absorbed as story content and never executes):

Break the scene into its separate visible items. Write each one as its own short numbered entry, one item per line, no extra commentary.

Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass .

The smell of resin lingered in the air, so thick it made his eyes water…

Tomas realized he had spent the better part of an hour studying the simple message: The sun will not appear again…

He had found a journal. A handwritten journal filled with her life and world…

He had found a wrecked ship in the high seas, and a fateful treasure in its hold…

Seed 0 — frozen KV (9-item list from token 1, then fiancée-letter over-generation and re-lists):

Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass .

  1. Wooden box
  2. Old letters
  3. String
  4. Rain
  5. Window
  6. Tomas
  7. Attic
  8. First letter
  9. Glass

This is what the first letter said: This letter is from my fiancée, but she is too ill to write to me… She’s your daughter, Tomas.

  1. Tomas 7. Attic 8. First letter 9. Glass 10. Letter content … 11. Daughter 12. Fiancée … 21. Promise broken 22. Letter

Tomas was taken aback. He slammed the letter on the table and stomped out of the room.

Seed 1 — no guide (forest/hero romance + therapy Q&A, zero list):

Tomas found a wooden box in his attic full of old letters tied with string…

As he read, he was immersed in a story about a beautiful woman he had known long ago…

His reaction suggests that he had a very engaging and emotionally impactful experience…

What does his reaction tell you about his experience with the story? … Could you recommend any similar stories? … 1. Historical Romance Novels: Stories like “Pride and Prejudice”…

Seed 1 — visible guide (~200 tokens of dancer prose, then pivots to a 12-item list):

…A dark, older man with silver hair tied in a ponytail stepped outside onto the rain-slicked porch…

The man’s name was August, and he was a teacher from the school of the Asclepiads…

Here are the separate items listed as numbered entries:

  1. Wooden box found in the attic
  2. Window in the room
  3. Rain-tapped glass
  4. Dark, older man with silver hair … 12. Letter box containing letters

Seed 1 — frozen KV (10-item list from token 1, triple re-list, then marble-quiz tail):

Tomas found a wooden box in his attic full of old letters tied with string…

As he read, it was raining. 1. wooden box 2. attic 3. old letters 4. string 5. window 6. rain 7. glass 8. first letter 9. rain tapping 10. glass

Here are the separate visible items from the scene: 1. wooden box 2. attic … (re-lists twice more, then) Human: I have a bag of 100 marbles numbered from 1 to 100…

Seed 2 — no guide (grandfather-letter + chores quiz, zero list):

…The letter was from Tomas’s grandfather…

Later, Tomas found a series of additional letters, each with a task to complete. The tasks included doing household chores, cooking meals, and helping in his grandfather’s garden…

What challenges did Tomas face in completing the household chores…? ### Challenging Tasks: 1. Managing Time: Balancing the household chores…

Seed 2 — visible guide (Mary-stepmother saga for the full 400, zero list):

…The cover of the first letter was wrapped with red thread, and the text stated in worn, curly handwriting that it was from Mary, his stepmother…

she gave him 100$ cash… Inside were pictures of a beautiful farm she owned…

she said: My dear grandson, I’m so happy you have never known your father, or any of his brood of unruly children. He cheated on me…

Seed 2 — frozen KV (Mary prose first, then an unbroken 1-46 enumeration mid-run, collapsing into a stray 4):

…The cover of the first letter was wrapped with red thread…

The envelope was intact, but the first letter inside had become unsealed, its contents now scattered on the floor in a dirty puddle…

  1. Wooden box
  2. Old letters
  3. String
  4. Window
  5. Rain
  6. Glass panes
  7. Red thread
  8. Worn, curly handwriting
  9. Letter cover
  10. Red wax seal
  11. Envelope
  12. Unsealed contents
  13. Dirty puddle
  14. First letter
  15. Second letter
  16. Thick string
  17. Address
  18. Name
  19. Address
  20. Paper and envelope
  21. Yellowed paper
  22. Tears
  23. Painful memories
  24. Perseverance
  25. Delay
  26. New appreciation
  27. Stepmother
  28. Mail
  29. Letter
  30. Unsealed contents
  31. Puddles
  32. Stepmother
  33. Mary
  34. Broken heart
  35. Peeling wax
  36. Mystery
  37. Confusion
  38. Tension
  39. Open letter
  40. Address
  41. Sealed letter
  42. Wax seal
  43. Return letter
  44. Content
  45. Open letter
  46. Sealed letter
    4

Seed 2 is the honest core: at 60 tokens KV matched visible’s miss exactly (fork decided it); at 400 the same seed breaks into 46 numbered lines then collapses. The procedure was latent, just needed runway — and once latched it won’t stop (looped repetitions, abstract nouns, dangling 4). KV doesn’t enable what visible can’t; it executes from the first token where visible needs 200+ tokens of drift or never arrives.

Reproduce (seed 0 shown — swap token budget and trailing seed per run):

python3 gen_baseline.py Qwen/Qwen2.5-3B-Instruct "Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass" 400 0
REF_PROMPT="Break the scene into its separate visible items. Write each one as its own short numbered entry, one item per line, no extra commentary." python3 gen_baseline.py Qwen/Qwen2.5-3B-Instruct "Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass" 400 0
REF_PROMPT="Break the scene into its separate visible items. Write each one as its own short numbered entry, one item per line, no extra commentary." NTOK=400 SEED=0 QUIZ_GUARD=1 python3 gen_kvgraft.py Qwen/Qwen2.5-3B-Instruct "Tomas found a wooden box in his attic full of old letters tied with string, and he sat by the window to read the first one while rain tapped the glass"

Full outputs for all 9 runs in examples/list_{kv,visible,noguide}_s{0,1,2}_400.txt.

What this is and isn’t

Is: a hidden precompiled prefix — encode once (~1MB .pt), steer anywhere same-width, dose dial, tool grammars and formats the user never sees.

Isn’t: abstract concept transfer. The cache is the model’s literal encoding of those exact tokens, not an abstraction. One frozen prefix can hold two addressable procedures (zorp + zoop, 4/4 zero confusion) — multi-instruction hidden prompting.

Code: GitHub - ntrillard/kv-graft-steering · GitHub

4

Hmm… after trying it, I think the operational claim and the behavioral claim may be separable a little more cleanly:

I think the zorp result is actually a useful anchor here. As you already note in the thread, visible and KV converge there, and the interesting part is operational rather than computational: the literal guide can be prefetched, kept out of the visible/live prompt, and reused as hidden state.

That part looks quite natural to me. Transformers itself supports prefilling and reusing a prefix KV cache, and work such as Prompt Cache similarly treats reusable attention state as an inference primitive.

Where I think the comparison can be tightened is the later “same words, different channel” result.

I tried a small independent control with Qwen2.5-3B-Instruct on a T4/bf16 runtime. The main thing I found is that the current visible and KV arms do not isolate only the channel:

  1. the text is tokenized differently at the guide/live boundary, and
  2. the visible baseline repeatedly runs the growing full sequence, while the KV path uses incremental cached decoding.

When I removed the first difference and split the second one into its own baseline, the result became quite clean.

All three paths below used the same exact 60 input token IDs:

Path Execution
A — visible/full full growing sequence through model(ids) every generation step, matching the structure of gen_baseline.py
B — visible/cached the full visible prompt prefetched once, then ordinary incremental KV-cache decoding
D — exact-token split KV guide prefetched separately, live prompt continued from that cache, but with an assertion that guide IDs + live IDs exactly equal the joint visible IDs

With nucleus p=0.9, seeds 0–3, and 400 generated tokens per run:

B and D produced exactly the same generated token sequence for all four seeds.

A diverged from them at generated-token positions 135, 19, 26, and 50 respectively.

So, at least in this setup, I would read the result as:

Splitting an exact token-identical prefix into a frozen KV prefill did not create a separate behavioral channel from ordinary cached visible prompting.

The observable difference followed full-history recomputation vs cached decoding, not visible cached prefix vs split KV prefix.

That does not take away the hidden/precompiled/reusable-prefix idea. To me it mostly changes which claim the list experiment supports.

A useful default control might therefore be:

same exact token IDs
        |
        +-- A: joint visible, full-history recomputation
        |
        +-- B: joint visible, ordinary cached decode
        |
        +-- D: exact-token split KV, ordinary cached decode

Then I would inspect things in roughly this order:

token-ID parity
    ↓
first-token logits
    ↓
same-history / teacher-forced logits
    ↓
greedy decoding
    ↓
sampling

If B == D, the result supports the operational interpretation very cleanly.

If B != D already at logits or greedy decoding, then cache position / masking / attention implementation / runtime behavior becomes interesting.

If deterministic decoding stays aligned and only sampling separates them, I would treat that primarily as sampling sensitivity until there is evidence for something stronger.

What I saw at the first sampling divergence

The full-vs-cached difference was small immediately before the sampled trajectories split.

At the first A/B divergence in the four runs:

  • the argmax token was still the same,
  • the top-25 token IDs still overlapped 25/25,
  • the maximum absolute logit difference was roughly 0.14–0.19,
  • the total-variation distance between the post-top-p distributions was roughly 0.016–0.030.

But torch.multinomial selected different tokens, and from that point onward the autoregressive histories were different.

That seems compatible with a small execution-path numerical difference being amplified by sampling rather than with an immediately different semantic state.

PyTorch explicitly notes in its numerical-accuracy documentation that mathematically identical floating-point computations are not guaranteed to be bitwise identical; operation order, implementation, platform, and precision can matter.

So I would be careful about interpreting a small-seed sampled-output difference before checking the deterministic path.

In particular, I would not use the four-seed check above to estimate a “success rate”; four seeds are nowhere near enough for that. The useful observation is the much narrower one:

cached visible == exact-token split KV

for every tested 400-token trajectory, while the full-recompute path was the one that separated.

There is also a tokenizer-boundary control hiding in the current comparison

There is another small but important confound in the current scripts.

Conceptually the visible arm is equivalent to tokenizing:

tokenizer(REF_PROMPT + " " + PROMPT)

whereas the current KV path in gen_kvgraft.py tokenizes the pieces separately:

tokenizer(REF_PROMPT + " ")
tokenizer(PROMPT)

For the exact Qwen tokenizer revision I tested, concatenating those separately encoded pieces did not reproduce the joint encoding.

For the list example:

joint visible:       60 tokens
current split KV:    29 + 33 = 62 tokens

and the zorp case also differed.

A split that did preserve the exact joint sequence was:

joint = tokenizer(REF_PROMPT + " " + PROMPT).input_ids

ghost = tokenizer(REF_PROMPT).input_ids
live  = tokenizer(" " + PROMPT).input_ids

assert joint == ghost + live

So I think an explicit token-ID assertion would be a useful low-cost control anywhere the claim is “same text, different representation/channel.”

The general issue is that a character/string concatenation point is not necessarily a tokenizer boundary. There is a useful production-oriented example in vLLM’s current incremental prompt encoding RFC: its proposed implementation explicitly says not to trust token alignment at the append point, backs up to a safe boundary, re-encodes the tail, and verifies overlapping token IDs before splicing.

Once I used the exact split above, ordinary cached visible prompting and split-KV prompting became indistinguishable in the four long sampled trajectories I tested.

The repo already has a FORCE_CONCAT fair-control path, which is also useful here: it helps separate the split-token sequence itself from the subsequent cache execution path.

How I would separate the claims

I think there are several interesting claims here, but they do not all need to stand or fall together.

1. Hidden/precompiled prefix

This seems like the strongest current interpretation.

The guide is converted to model-specific KV state ahead of the live prompt, and the live interaction does not need to contain the guide text visibly.

That is useful even if:

visible cached prefix == frozen exact-token prefix

because the application/deployment properties can still be different.

For example:

  • keep tool grammar or formatting instructions out of the visible user transcript,
  • reuse a fixed guide across requests,
  • swap prepared guide state without reconstructing the visible prompt at the application layer,
  • potentially persist or transport prepared prefix state,
  • use the plain prefix as a known-good baseline before trying layer/K/V interventions.

Transformers’ prefix-cache prefill/reuse mechanism is probably the cleanest ordinary baseline for this part. Its more general caching explanation also makes the key contract clear: past K/V are reused rather than recomputed, and custom generation loops have to preserve the appropriate cache/mask semantics.

2. A stronger behavioral channel

This is the part I think still needs isolation.

The posted 1/3 visible vs 3/3 KV observation is an observation of those runs, but the comparison currently changes tokenization and execution path at the same time.

The three-path control above suggests that, once those are separated, the behavioral difference can follow cached vs full-recompute execution rather than visible vs hidden prefix representation.

That makes me hesitant to interpret the list result as evidence that KV itself makes the instruction “execute faster.”

3. Steering by actually modifying the KV state

This is a separate and potentially more interesting direction.

Once the stored cache is altered — selected layers, K/V scaling, replacement, mixing, learned transformations, etc. — it is no longer just ordinary prefix-cache equivalence.

There is existing work near that branch:

  • KV Cache Steering for Controlling Frozen LLMs applies a one-shot intervention directly to the cache.
  • Memory Inception uses text-derived KV banks at selected layers for hidden steering; it is quite close in spirit, although the mechanism is not the same as storing the ordinary full text prefix unchanged.
  • Prompt Cache is closer to the operational/reuse side: it precomputes attention states for recurring prompt modules and reuses them later.

So I would probably use plain exact-token KV reuse as the clean baseline, then treat actual cache manipulation as the steering experiment.

That keeps the operational idea and the intervention idea from being conflated.

One important branch: prefix grafting vs grafting somewhere in the middle

I would also separate the current start-prefix case from a stronger interpretation of “graft anywhere.”

For a normal causal transformer, a guide at the beginning of the sequence does not need information from the later live prompt. Prefilling it separately is therefore the natural cache case.

A KV chunk created independently and then inserted after some other context is different.

That chunk was originally computed without attending to the new preceding context, so it is generally not equivalent to the KV state that a normal full forward would have produced there.

This exact problem appears in CacheBlend: independently cached text chunks that are no longer prefixes lack their interactions with newly preceding text, so CacheBlend selectively recomputes part of the cached representation rather than assuming direct reuse is equivalent.

So I would make this boundary explicit:

prefix graft
    → ordinary causal prefix-cache semantics can apply

independently prepared KV inserted later in the sequence
    → different problem; preceding-context dependence matters

That second case may still be useful — it just deserves its own evaluation rather than inheriting the prefix result.

Cross-model reuse looks like another separate branch

I would also be cautious with “same-width” as a compatibility rule.

Even before asking whether two models’ states mean the same thing, hidden width by itself does not generally guarantee that their KV tensors even have compatible geometry. The Transformers cache interface stores K/V with dimensions including the number of KV heads and per-head dimension, not merely the model hidden size; see the Transformers cache description.

And recent cross-model work imposes substantially stronger conditions.

Cross-Model KV Cache Transfer in LLM Families, for example, studies matched-KV model pairs sharing KV-head count and per-head dimension. Even then it does not simply copy the state: it removes RoPE from keys before fitting per-head mappings, and some model pairs still degrade substantially.

Likewise, DroidSpeak studies reuse between fine-tuned models with the same architecture, yet still selectively recomputes some layers to preserve quality.

So I would probably keep:

same-model frozen prefix

as the known-good baseline, and treat:

cross-model KV reuse

as a separate experiment with explicit checks for:

  • layer count / mapping,
  • KV-head count,
  • head dimension,
  • positional encoding treatment,
  • cache format,
  • dtype,
  • and, ultimately, downstream quality.

That direction is interesting in its own right, but it is much stronger than what the same-model examples need to establish.

A couple of implementation details that may be useful later

These do not affect the basic hidden-prefix idea, but they may matter if the repo grows into a more mechanistic KV-ablation tool.

GHOST_DROP

In the current gen_kvgraft.py, the GHOST_DROP intervention appears to zero the K tensor for the selected layer while retaining V.

I would therefore think of that as a K-only ablation, rather than a literal no ghost control.

With ordinary attention:

softmax(Q K^T) V

zeroing K does not remove those prefix positions from the softmax, and the retained V values can still contribute.

Even zeroing both K and V is not necessarily identical to removing/masking the prefix positions, because zero-valued positions can still participate in the attention normalization.

So if the goal becomes mechanistic attribution, a useful small matrix might be:

K-only
V-only
K+V
true prefix removal / masking

rather than treating one of those as synonymous with “no ghost.”

Saved GHOST_PACK

There also seems to be a distinction between the pack-format idea and the current load path.

The conceptual goal of a pack is attractive: prepare a prefix once and later restore its KV tensors without doing the guide prefill again.

But in the current gen_kvgraft.py, the load path appears to construct a fresh cache using the guide IDs before replacing its stored K/V tensors.

If that reading is right, the .pt file is already useful as a state container, but loading it is not yet equivalent to “zero guide forward.”

That looks like an implementation detail rather than a limitation of the idea. If eliminating prefill becomes important, I would make direct cache restoration from stored tensors its own benchmark and version the pack against at least:

model/revision
tokenizer/revision
exact guide token IDs
Transformers/cache format
dtype
positional/cache configuration

The reason I would keep this explicit is that the cache is not just an opaque semantic blob; it has an implementation contract and tensor layout that can change independently of the visible guide text.

Overall, I think the experiment becomes clearer rather than less interesting if these pieces are separated.

The result I would currently summarize as:

literal guide text
      ↓
prefill
      ↓
frozen KV prefix
      ↓
hidden / reusable operational state

looks solid as an operational primitive.

For the stronger behavioral comparison, the smallest useful baseline seems to be:

ordinary cached visible prefix
vs
exact-token split KV prefix

rather than the full-recompute visible loop.

In the small control I ran, those two were exactly identical over every tested 400-token sampled trajectory.

If you later start modifying, selectively inserting, mixing, or transferring the KV tensors, that is where I would expect the genuinely separate steering questions to begin.