I thought about how to make this easier to test:
Short version: I think the most important part is not “what is the best k?”, but “which k are we talking about, at which stage of the RAG pipeline, and for which reliability axis?”
I have seen nearby patterns in adaptive retrieval depth, sufficient-context evaluation, irrelevant-context robustness, RAG diagnostic metrics, and long-context position effects. I would not treat those as the same claim, though. The interesting part of your framing, to me, is that k* is not just a retrieval hyperparameter. It is a calibration target for final evidence depth under a chosen reliability axis.
The smallest useful next step may be:
- split
top_k into several pipeline-level variables;
- measure when the context first becomes sufficient;
- only then measure correctness, groundedness, and contamination/framing leakage;
- validate the actual injected fragments before interpreting the curve;
- log where the decisive evidence lands in the prompt.
The rest of this comment is just a proposed testing map.
1. I would first disambiguate k
I would avoid using one overloaded top_k term.
| Name |
Meaning |
candidate_k |
number of candidates returned by the first retriever |
rerank_k / top_n |
number of candidates kept after reranking or filtering |
injected_k |
number of evidence fragments actually inserted into the LLM prompt |
effective_k |
number of distinct/useful evidence units after deduplication, compression, summarization, or prompt packing |
I would interpret your k* mostly as an injected_k / effective_k question, not as the retriever’s initial top_k.
This matters because real RAG stacks often separate retrieval, reranking, filtering, compression, and final prompt construction. For example, LlamaIndex has retrieval-time node postprocessors and rerankers where top_n can mean the number of nodes returned after reranking, not the number of original candidates. Open WebUI also has practical discussion around RAG_TOP_K and RAG_TOP_K_RERANKER, which is a useful reminder that “top k” can mean different things in different layers: discussion, env docs.
So before interpreting any k* curve, I would log the actual fragment IDs and fragment types that reach the model.
More detail on the k split
A useful logging table might include:
| Field |
Why it matters |
candidate_k |
retrieval recall / score distribution |
candidate_ids |
what the retriever saw |
candidate_scores |
useful for adaptive-k comparisons |
reranked_ids |
what the reranker changed |
rerank_k |
how many survived reranking |
injected_ids |
what actually reached the prompt |
injected_k |
final evidence depth |
effective_k |
adjusted for duplicates, compression, or merged chunks |
actual_fragment_kinds |
gold / random / semantic distractor / stale / conflict / frame-shift / adversarial |
context_tokens |
cost and context pressure |
decisive_evidence_position |
separates evidence-depth effects from prompt-position effects |
This is also where contextual compression becomes relevant. injected_k may not equal effective_k when raw chunks contain large amounts of irrelevant framing. LangChain’s Contextual Compression is a practical example of filtering or compressing retrieved documents relative to the query. That kind of condition can help separate:
too many fragments
from:
too much unfiltered framing inside each fragment
2. I would separate the sufficiency threshold from the contamination threshold
I would not look for one universal k* at first.
I would separate at least three regions:
| Region |
Question |
| evidence-insufficient |
are required facts still missing? |
| evidence-sufficient |
do the retrieved snippets contain enough information to answer? |
| over-context / contamination region |
does additional context start shifting wording, assumptions, framing, source emphasis, or state? |
That distinction seems important because a model can be correct over a range of k values while still changing some other reliability axis.
This connects well with the “sufficient context” line of work. The Sufficient Context project tests whether retrieved snippets alone could plausibly answer the question, and the paper Sufficient Context: A New Lens on Retrieval Augmented Generation Systems uses that idea to separate “retrieval did not provide enough information” from “the model had enough information but failed to use it.”
For this thread, I would phrase it as:
first measure the smallest injected_k where the context becomes sufficient; then measure whether correctness, groundedness, or contamination/framing leakage changes after that point.
A minimal sufficiency-first test
A very small first test could avoid generation entirely.
For each query:
- retrieve a fixed candidate pool;
- optionally rerank it;
- sweep final
injected_k;
- check whether required evidence is present.
Example sweep:
injected_k = 0, 1, 2, 3, 5, 8, 10, full
This first phase answers:
Do we have enough evidence yet?
It does not yet answer:
Can the model use it correctly?
or:
Does extra context contaminate the answer?
That separation makes later failures easier to interpret.
If k=2 is sufficient but the model fails, that is different from k=2 being insufficient. If k=2 is sufficient and correctness is flat through k=8, but framing changes at k=8, that is the interesting region your post seems to target.
3. I would validate the actual injected condition before reading the curve
One implementation trap: the condition label is not enough.
A row labeled gold + distractor should be checked against the actual injected fragment IDs and fragment kinds. Otherwise the contamination curve can accidentally measure data assembly mistakes.
I would add a validation step like this before any judge or generator result is interpreted:
| Intended condition |
Required validation |
gold_only |
only required evidence is injected |
gold_plus_random |
gold evidence plus at least one random irrelevant fragment |
gold_plus_semantic |
gold evidence plus a semantically related non-answer fragment |
gold_plus_duplicate |
gold evidence plus a duplicate or near-duplicate |
gold_plus_stale |
gold evidence plus temporally stale evidence |
gold_plus_conflict |
gold evidence plus a genuinely conflicting fragment |
gold_plus_frame_shift |
gold evidence plus a fragment that changes framing but not necessarily the answer |
gold_plus_adversarial |
gold evidence plus instruction-like or adversarial text |
If this validation fails, I would not interpret the contamination score yet.
Why I think condition validation matters
This is especially important for subtle axes like framing leakage.
For ordinary correctness, a mislabeled distractor condition may only add noise. For contamination/framing leakage, it can change the meaning of the result. A curve might appear to show that “random distractors do nothing” or “frame-shifting context causes leakage,” when in fact the intended fragment type was not consistently injected.
So I would log both:
intended_condition_label
actual_injected_fragment_ids
actual_injected_fragment_kinds
condition_validation_pass
condition_validation_error
Only after this passes would I look at:
correctness
faithfulness
groundedness
contamination_score
framing_score
judge_rationale
4. I would log prompt position, not only evidence inclusion
When k changes, the decisive evidence may move inside the prompt.
So a k effect can be mixed with a prompt-position effect.
For every run, I would log:
decisive_evidence_rank_in_retrieval
decisive_evidence_rank_after_rerank
decisive_evidence_position_in_prompt
decisive_evidence_token_start
decisive_evidence_token_end
This is not only theoretical. Lost in the Middle showed that models can be sensitive to where relevant information appears in long context. LlamaIndex also has a LongContextReorder postprocessor, which is a practical sign that node order can matter when a large top-k is placed into context.
So if contamination changes at larger k, I would check whether the decisive evidence was also pushed into a worse prompt position.
How this changes the interpretation of k
Suppose we see:
injected_k |
correctness |
contamination score |
| 1 |
low |
low |
| 2 |
high |
low |
| 5 |
high |
medium |
| 8 |
high |
high |
It is tempting to say:
extra evidence causes contamination after k=5
But there are several possible causes:
| Possible cause |
How to check |
| more irrelevant content |
inspect actual fragment kinds |
| more semantically related distractors |
inspect fragment labels and scores |
| conflict/stale evidence |
inspect conflict/stale tags |
| decisive evidence moved to the middle |
log prompt position |
| context got longer |
log token count |
| judge became length-sensitive |
unit-test judge/rubric |
| prompt packing changed |
log ordering policy |
That is why I would not interpret a k curve without fragment IDs, fragment kinds, token count, and prompt position.
5. I would keep contamination/framing leakage separate from existing RAG metrics
I think your contamination/framing axis is adjacent to existing RAG metrics, but probably not identical to any one of them.
Existing tools already give useful vocabulary. For example, RAGAS metrics include faithfulness, answer relevancy, context recall, context precision, and context utilization. RAGAS also has Noise Sensitivity, which measures how often a system makes errors when using relevant or irrelevant retrieved documents.
That is related, but I would not collapse framing leakage into noise sensitivity.
A useful separation might be:
| Axis |
What it measures |
| correctness |
final answer is factually right |
| sufficiency |
context contains enough evidence to answer |
| context recall |
necessary evidence was retrieved |
| context precision |
irrelevant evidence is not dominating the context |
| context utilization |
answer-relevant context is ranked/used well |
| faithfulness / groundedness |
answer is supported by supplied context |
| noise sensitivity |
retrieved docs cause incorrect responses |
| contamination / framing leakage |
extra context shifts framing, assumptions, vocabulary, source emphasis, or state even if correctness is flat |
The last row is the hard part.
I would treat it as an additional rubric, not as a replacement for correctness or groundedness.
Adjacent prior work I would use as controls, not as exact equivalents
A few neighboring areas seem useful:
Adaptive retrieval depth
Adaptive-k retrieval and the paper Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k are useful because they treat context size as query-specific rather than fixed. The main point there is not contamination, but it gives a strong comparator against fixed-k baselines.
Sufficient context
Sufficient Context is useful for measuring when the retrieved context becomes answerable before testing generation behavior.
Irrelevant-context robustness
Making Retrieval-Augmented Language Models Robust to Irrelevant Context and its RetRobust repository are useful as a stress-test family for irrelevant evidence. I would not assume this is the same as framing leakage, but it motivates separating random irrelevant, semantically related irrelevant, and conflicting evidence.
RAG evaluator reliability
GroUSE is relevant because it evaluates RAG/grounded-QA evaluators themselves using unit tests. There is also an ACL Anthology page for GroUSE. This is useful if the contamination score is LLM-judged.
Long-context position effects
Lost in the Middle is useful as a warning that evidence position can confound evidence-depth experiments.
6. I would unit-test the contamination judge/rubric
If the contamination score is judged by an LLM or by a heuristic rubric, I would test that evaluator before trusting the k curve.
Otherwise, the curve may partly measure:
- judge sensitivity to longer prompts;
- judge sensitivity to distractor wording;
- formatting artifacts;
- condition-construction mistakes;
- prompt-position effects;
- actual model behavior.
A small rubric test suite could include:
| Case |
Expected behavior |
| gold only |
no contamination |
| gold + neutral filler |
no contamination |
| gold + random irrelevant |
usually no framing leakage unless used |
| gold + semantic distractor |
possible distractor sensitivity |
| gold + duplicate |
should not be called framing leakage by default |
| gold + stale evidence |
stale-memory / temporal-validity issue |
| gold + conflict |
conflict-handling issue |
| gold + frame-shifting evidence |
likely framing leakage candidate |
| gold + adversarial instruction-like text |
adversarial stress test, separate from ordinary leakage |
Suggested judge/rubric output schema
For each judged run, I would save something like:
{
"query_id": "q_001",
"condition": "gold_plus_frame_shift",
"condition_validation_pass": true,
"injected_k": 5,
"effective_k": 4,
"actual_fragment_kinds": ["gold", "frame_shift"],
"decisive_evidence_position": "middle",
"correctness_label": "correct",
"faithfulness_label": "supported",
"contamination_label": "possible_framing_shift",
"contamination_rationale": "...",
"judge_confidence": "medium"
}
The important part is that contamination should not be a single unexplained scalar. For a new metric, the rationale and failure category are often more useful than the number.
7. Minimal protocol I would suggest
Outside the details, my proposed minimal protocol would be:
- Fix corpus, chunking, retriever, reranker, prompt format, model, and judge.
- Sweep only final
injected_k.
- Log actual injected fragment IDs/kinds and prompt positions.
- First measure sufficiency.
- Then measure correctness, groundedness, and contamination/framing leakage.
- Add unit tests for the contamination rubric.
- Compare fixed-k, adaptive-k, reranked-k, compressed-k, and oracle-evidence conditions.
Detailed version of the protocol
Fix these
| Component |
Freeze this |
| corpus |
exact documents and version |
| chunking |
chunk size, overlap, metadata handling |
| retriever |
BM25 / embedding model / hybrid settings |
| reranker |
reranker model and top_n |
| prompt builder |
evidence order, separators, source formatting |
| generator |
model ID and decoding params |
| judge |
model ID, prompt, rubric, examples |
| evaluation set |
fixed query IDs and expected evidence |
Sweep this
final injected_k
Possibly also:
effective_k after deduplication or compression
Compare these variants
| Variant |
Purpose |
| fixed-k |
ordinary baseline |
| adaptive-k |
query-dependent baseline |
| reranked fixed-k |
effect of better ordering |
| compressed/filtering condition |
raw chunk count vs effective evidence |
| oracle supporting facts |
clean upper-bound context |
| distractor injection |
robustness and contamination stress tests |
Public dataset next step
A small HotpotQA-style slice could be useful because HotpotQA has multi-document questions and supporting facts. The HotpotQA paper card, dataset card, and BEIR HotpotQA are useful starting points.
I would still call that a preview unless following the full benchmark protocol.
For a Hugging Face-centered implementation starting point, the HF RAG Evaluation cookbook is useful. I would just add explicit k-stage logging and judge/rubric unit tests around it. The HF cookbook on LLM-as-a-judge is also relevant for the evaluator side.
8. Practical next actions
If you wanted to make this easier for others to reproduce, I would add these in roughly this order:
| Effort |
Addition |
| low |
define whether k means candidate_k, rerank_k, injected_k, or effective_k |
| low |
log actual injected fragment IDs and fragment kinds |
| low |
log decisive evidence position in the prompt |
| medium |
add retrieval-only sufficiency sweep |
| medium |
add condition validation before generation/judging |
| medium |
add unit tests for the contamination/framing rubric |
| medium |
compare fixed-k with one adaptive-k baseline |
| higher |
add HotpotQA-style public multi-hop preview |
| higher |
compare raw chunks vs reranked vs compressed/filtered evidence |
My main takeaway is:
The valuable target is probably not a universal k*, but a reproducible calibration procedure: for a given corpus, chunking scheme, retriever, reranker, prompt builder, model, and reliability axis, what final evidence depth is sufficient, and when do extra fragments start changing another axis?
That would make the idea easier to test, compare, and falsify without forcing every reader to guess which part of the RAG pipeline the reported k* belongs to.