I think that makes sense:
I would move to a dependency-rich task-level test next, with one qualification: I would not treat that as “replace the cheap Spiral gate with the semantic selector.”
Your latest results look more useful to me if they are read as exposing two different operating points:
cheap Spiral gate
→ very low-overhead generic context reduction
query-/target-aware semantic selection
→ more expensive dependency protection when the task supplies
a meaningful signal about what must survive
That distinction seems worth preserving rather than forcing both selectors into a single winner/loser comparison.
On the generic local corpus, the cheap gate is still doing something useful: it stays very close to random/full-context loss while retaining roughly the same large step-time/VRAM reduction. On the deliberately constructed dependency probe, however, the cheap gate and random selection can cut the required pair, while the query-aware semantic selector can protect it.
So my default next step would be a small task-level bridge, not a large benchmark sweep:
dependency-rich dataset with known required evidence
↓
selector-only coverage check
↓
small task-level inference check
↓
only then: controlled QLoRA comparison
For the eventual training comparison, I would keep the core conditions small:
full context
matched random 60%
cheap Spiral 60%
query-aware / task-aware semantic 60%
and, when the task exposes an actual query, optionally add a very cheap lexical baseline such as BM25 at the same block budget.
For every condition I would record three things together:
1. did the selected context retain the required evidence?
2. did that translate into task success?
3. what did it cost end-to-end?
That would answer a more useful question than either aggregate held-out loss or evidence coverage alone:
does preserving the dependency actually buy training/task quality at a cost that is worth paying?
A 2-hop MuSiQue subset looks particularly suitable for a first pass because its questions were constructed specifically so that one reasoning step depends on another, and the official repository exposes the underlying multi-hop structure. I would use something like RULER as a mechanism probe alongside that, rather than making a synthetic benchmark the main task-level result.
One useful extra evaluation-only condition is:
gold evidence only
That is almost free once the dataset provides support labels. If gold-only works well but a selector that retained all gold evidence still performs poorly, the next bottleneck is probably no longer selection recall; it is somewhere in the noisy-context recognition / prioritization / synthesis path.
That separation has become fairly explicit in recent multi-hop work as well. Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck separates locating the evidence from integrating it, and its released experimental code includes fixed-distractor multi-hop setups and gold-focused controls that seem useful as evaluation-design references.
I would therefore see your next boundary roughly as:
"can the selector preserve both required blocks?"
↓
"can the model use those blocks among distractors?"
↓
"does training on that selected context improve the task?"
↓
"does the improvement justify selector overhead?"
rather than jumping directly from 144/144 coverage to a claim about fine-tuning quality.
Why I would keep the cheap gate and semantic selector as separate design branches
The current results seem to support keeping both ideas alive.
For the generic 8K QLoRA experiment, your reported means were approximately:
| Condition |
Step time |
Full-context held-out loss |
| Full |
2676.57 ms |
0.443962 |
| Random ~60% |
1596.40 ms |
0.445005 |
| Cheap Spiral gate ~60% |
1594.98 ms |
0.444530 |
| Tail-anchored semantic ~60% |
1624.53 ms |
0.445099 |
The cheap gate therefore still looks like a very inexpensive way to obtain most of the physical-sequence-shortening benefit on this corpus.
Its selector overhead was about 1.27 ms/step, versus 31.28 ms for the semantic training selector. But even 31 ms is only a small part of the ~1.62 s selected-context step in this particular run, so I would not read that overhead as a reason to discard semantic selection either. If semantic selection buys measurable task quality on dependency-rich data, that could still be a very reasonable trade.
The controlled dependency probe answers a different question:
cheap gate: required pair retained 0 / 144
random: 31 / 144
query-aware semantic: 144 / 144
To me this is less:
semantic selector > cheap selector
and more:
generic compression objective
!=
dependency-preservation objective
That distinction could eventually lead to a hybrid policy rather than one universal selector:
ordinary / redundant samples
→ cheap gate
samples with an explicit task/query or strong long-range dependency
→ dependency-aware selector
There is some related motivation for thinking about the data as dependency-rich or dependency-poor rather than assuming every long sequence needs the same treatment. ProLong, for example, is not a block selector like yours, but explicitly distinguishes raw sequence length from the strength, distance, and specificity of long-range dependencies in the training data.
I would treat that as a later design direction rather than something required for the next experiment.
A low-cost task-level protocol
I would stage this so that an expensive QLoRA run is the last step, not the first.
Stage A — selector-only audit
No model training is needed.
For each task example, record the gold-support blocks and run:
random 60%
cheap Spiral 60%
semantic 60%
optional BM25 60%
Then report something like:
P(all required evidence retained)
P(hop 1 retained)
P(hop 2 retained)
number of retained fragments / joins
selected-token count
selector time
If the semantic selector does not separate from cheap/random here, there is little reason to spend GPU time training it on that dataset.
If it strongly separates, proceed.
Stage B — frozen-model task probe
Before fine-tuning, feed the selected contexts to a frozen model and measure:
question only
gold evidence only
random-selected context
cheap-Spiral-selected context
semantic-selected context
full context, if practical
This is not the final fine-tuning result. It just tells you whether the selected context is usable.
The informative comparisons are:
gold-only strong
semantic weak despite retaining gold
→ evidence recall is not the remaining bottleneck
semantic ≈ gold-only
random / cheap much lower
→ selector-specific dependency preservation is promising
all selected conditions weak
→ task/model/prompt may be the bottleneck before training
Stage C — small controlled QLoRA run
Only after A/B show a meaningful separation would I run the expensive comparison.
Keep fixed what you already made explicit:
model
initial adapter state
split
optimizer
batch size
sequence length
block size
compact-position policy
join-loss masking
seed
and vary only the context-selection policy.
The useful output table would combine quality and compute rather than publishing them separately:
| Policy |
Evidence retained |
Task score |
Full-context held-out loss |
Selector ms |
End-to-end step ms |
Peak VRAM |
| Full |
— |
|
|
— |
|
|
| Random 60% |
|
|
|
|
|
|
| Cheap Spiral 60% |
|
|
|
|
|
|
| Semantic 60% |
|
|
|
|
|
|
If the existing 0 → 12 → 24 → 48 checkpoints are already available, I would keep them as a useful secondary diagnostic, but I would not make a large checkpoint sweep a prerequisite for this task-level experiment.
Why evidence retention and answer quality should be separate columns
One trap here is that a selector can be objectively better at retaining required information without producing an immediate improvement in aggregate model quality.
Those are different stages:
selection:
are the necessary blocks present?
recognition:
does the model locate / prioritize them among distractors?
synthesis:
can it combine them correctly?
learning:
does the resulting training signal improve the adapter?
The Weakest Link work is useful here because it explicitly asks whether multi-hop failures come from evidence recognition or evidence integration.
That distinction also fits your current result surprisingly well:
semantic selector:
controlled evidence coverage looks excellent
generic held-out LM loss:
no clear advantage over random
Those two observations do not have to conflict.
The generic continuation corpus may simply not reward the type of dependency preservation that the controlled probe tests.
This is also why I would keep your full-context held-out loss but stop asking it to carry the entire quality claim.
LongPPL / “What is Wrong with Perplexity for Long-context Language Modeling?” gives a useful general warning: average LM loss can hide a small subset of tokens whose prediction actually depends on distant context.
You do not need to adopt LongPPL itself for the next experiment. A task with known dependency-bearing evidence/targets already gives you a much simpler version of the same diagnostic idea.
Why MuSiQue first, RULER second, and a broader benchmark later
I would give these tools different jobs rather than choosing one benchmark to do everything.
MuSiQue — first task-level bridge
MuSiQue was specifically designed around connected multi-hop reasoning: one sub-question is selected so that a later reasoning step critically depends on its answer.
That makes it unusually convenient for your case because you can ask both:
did the selector preserve the required support?
and:
did the resulting context support the final answer?
The MuSiQue repository also makes the dataset structure reusable rather than requiring you to invent a private synthetic task.
I would probably start with 2-hop cases before 3/4-hop examples so the interpretation stays simple.
RULER — controlled mechanism stress
RULER remains useful for things such as:
- multiple-needle retrieval,
- variable tracking,
- configurable hop counts,
- aggregation.
Its own README explicitly says it is a sanity-check test bed and is not comprehensive enough to replace realistic tasks.
That seems exactly right here.
Use it to answer:
what kind of dependency does this selector sever?
rather than:
does this prove downstream quality?
HELMET / realistic RAG — later breadth check
HELMET is useful mainly for the next stage.
Its large study found that synthetic needle-style performance does not reliably predict downstream long-context performance, and it recommends its RAG tasks as a relatively convenient development-time signal before broad evaluation.
So a progression like this still looks economical:
controlled dependency coverage
↓
small MuSiQue / multi-hop task
↓
small realistic RAG or long-document QA task
↓
only then broader long-context evaluation
That gives each test a clear purpose.
The selector's information contract is worth making explicit
I would separate at least three selector classes.
1. Query-only / task-input-aware
The selector can use information available at inference or ordinary task execution time:
question / instruction
+
candidate context
This is the cleanest contract if the future application is QA/RAG-style selection.
2. Target-aware / training-only
The selector can use a protected target tail, labels, or another signal that exists during fine-tuning but would not necessarily exist during inference.
I would not call this inherently invalid or “leakage.”
If SpiralCoreAttention is specifically a training-time context-selection method, using training-only information may be a perfectly legitimate design.
It just supports a different claim:
query-only:
online/task-conditioned selection
target-aware:
offline/training-time selection
The useful thing is simply to label the distinction.
3. Gold/oracle
The selector is explicitly given the known evidence/support labels.
That is not a deployable baseline; it is a ceiling/diagnostic.
For example:
semantic retains all gold evidence but task score is far below gold-only
is much more informative than simply saying semantic selection failed.
It tells you that improving selector recall further is unlikely to address the main bottleneck.
Because your training semantic selector uses a protected target tail as an offline anchor, while the controlled coverage probe is described as query-aware, I would keep those two experimental contracts visibly separate rather than merging them under one generic “semantic selector” label.
A cheap lexical baseline may help isolate what the semantic scorer is buying
Once there is an explicit query/task signal, I think BM25 block ranking at the same budget is a useful cheap control.
The comparison becomes:
random
→ no task information
BM25
→ cheap lexical task information
semantic
→ richer semantic task information
Then:
semantic ≈ BM25 >> random
→ query-awareness matters, but a semantic scorer may not be necessary
semantic >> BM25 >> random
→ evidence for value beyond lexical matching
BM25 ≈ semantic ≈ random
→ task/query signal is not helping selection much
coverage improves but task score does not
→ selection recall is no longer the main boundary
I would only add BM25 where there is a genuine query/task anchor. It is not an obvious baseline for generic next-token continuation where there is no meaningful query.
This is similar in spirit to the attribution question behind instance-aware methods such as TokenSeek: the mechanism is different, but it is useful to separate benefits from reducing the token budget from benefits obtained by selecting tokens using instance-specific information.
About the 31 / 144 random dependency result
I would not compare 31/144 directly with:
0.6 × 0.6 = 36%
unless every block is independently eligible for the same 60% sampling probability.
If part of the selected-token budget is reserved in advance—for example for a protected target region—then the random selector may actually be doing something more like:
protect some blocks first
then:
choose k additional blocks
from N remaining eligible blocks
For two required blocks that are both in that remaining pool, their joint inclusion probability is governed by sampling without replacement from N, not simply by squaring the global retained-token ratio.
So the only extra reporting I would want here is very small:
total block count
protected block count
random-eligible block count
number randomly selected from that pool
With those four values, the expected random pair-retention rate becomes unambiguous.
I would treat this as documentation of the comparison contract, not as a problem with the 31/144 result.
One small independent sanity check that changed how I would interpret the result
I also tried a deliberately small public-data side probe—not a reproduction of your training setup, and not evidence about the Spiral selector itself.
I built 48 fixed ~8K two-hop cases from public MuSiQue-derived material, with two known required evidence blocks and distractors, then compared matched ~60% block selection policies.
On that constructed fixture:
required pair retained:
random, one selection seed: 21 / 48
BM25 query-only: 39 / 48
semantic query-only: 48 / 48
So the semantic selector did exactly what I would hope at the coverage layer.
I then used a frozen 4B instruction model as a crude task probe on those selected contexts.
With a longer output cap, the diagnostic primary-answer F1s were roughly:
gold evidence only: 0.539
semantic 60%: 0.162
BM25 60%: 0.164
random 60%: 0.141
I would not treat those numbers as a benchmark:
- the fixture was constructed;
- most distractor material was still synthetic/controlled rather than a natural long document;
- this was frozen inference, not QLoRA;
- the model was not your 7B training model;
- the metric used only the primary answer rather than the full official MuSiQue alias set.
But qualitatively it was useful.
It showed a very concrete failure mode:
semantic selector:
nearly perfect evidence retention
model:
still far below the gold-only condition
In other words, improving selection recall can expose a second bottleneck rather than automatically solving the task.
That is why I would report:
retained?
used successfully?
task correct?
as separate quantities in the next experiment.
A decision tree I would use after the next run
dependency-rich task
|
v
Does semantic selection retain required evidence
more reliably than random / cheap Spiral?
|
+-- no
| |
| → semantic scoring is not buying the intended mechanism
| on this task
|
| → keep the cheap gate unless another task motivates it
|
+-- yes
|
v
Does task quality improve too?
|
+-- yes
| |
| → selector-specific dependency preservation
| now has task-level evidence
|
| → compare quality gain against the extra
| selector overhead
|
+-- no
|
v
Is gold-only much better?
|
+-- yes
| |
| → selection recall is probably no longer
| the main bottleneck
|
| → investigate distractor sensitivity,
| evidence prioritization / ordering,
| and multi-hop synthesis
|
+-- no
|
→ the model/task/training horizon itself may
be limiting the observable separation
A second branch is useful if BM25 is included:
semantic ≈ BM25 > random
→ task-aware selection matters;
expensive semantic scoring may not be necessary
semantic > BM25 > random
→ stronger evidence that semantic matching itself contributes
semantic coverage > BM25
but task quality ≈ BM25
→ the extra retrieved evidence is not yet translating into utility
That seems like a fairly high-information experiment without turning the project into a benchmark campaign.
The main thing I would preserve from the current results is that the original cheap gate does not look invalidated by the dependency failure.
Instead, the new probe seems to have identified a boundary of what that gate is optimizing.
For a generic, redundant corpus, an extremely cheap selection rule may be enough to get most of the compute/VRAM benefit.
For a task where one distant dependency is disproportionately important, a different selector may be worth paying for.
If the next task-level experiment confirms that split, I think that is actually a cleaner result than trying to make one selector dominate every regime:
cheap shortening when shortening is enough
dependency-aware selection when the dependency matters
And if semantic selection still fails to improve task quality despite reliably preserving the evidence, that is also a useful result: it moves the next design question downstream from which blocks are selected to how selected evidence is recognized and used.