Quick update — both annotation runs are complete and most of the gaps I listed in my previous post are now closed. This is a final results report, not another work-in-progress.
A few things changed since my last post that I want to be explicit about, because they affect interpretation of the numbers below:
1. **Production ranking changed from engine-count to RRF + quality filter.** In my post 3 I described engine-count as the production default. During the benchmark cycle, the ablations showed RRF with trust weighting and quality filtering consistently outperformed both engine-count and BM25. I switched the production path to RRF + quality filter (RRF k=10, trust-weighted per engine, then quality filter applied). The benchmark numbers below reflect this — “rrf-quality” is the current production configuration, not the one I described in post 3.
2. **RRF k changed from 60 to 10.** My post 3 mentioned k=60 following Cormack et al. SIGIR 2009. After empirical tuning on the frozen corpus, k=10 produced better top-10 rankings for this domain. The Cormack paper used k=60 on TREC newswire; onion SERPs have shorter result lists and different rank-depth distributions, which favors a smaller k. The benchmark numbers below all use k=10.
3. **Engine count: 13, not 14.** My post 1 said 14 engines. Torgol was removed during the benchmark cycle (transient Tor circuit issues, not a parser bug — it returned results intermittently). The frozen corpus and all benchmark numbers below use 13 engines. The engine overlap matrix is 13×13.
4. **Quality filter explanation.** Several posts refer to “rrf-quality” without explaining what the quality filter does. It’s a 6-strategy demotion pipeline that never removes results — it only reorders: (1) domain collapsing (max N per domain), (2) SimHash near-duplicate detection (Hamming distance ≤3), (2b) semantic near-duplicate via embeddings (cosine >0.85), (3) information-quality scoring (snippet/entity/title signals), (4) spam/mirror detection (same title, different domain), (5) commercial-intent penalty (buy/shop/price keywords demoted for research queries), (6) URL-path mirror network detection. The filter is OFF by default and must be explicitly enabled. “rrf-quality” in the benchmark = RRF ranking followed by quality filter reordering.
5. **Test count: 223, not 180.** My post 1 and 3 said 180 tests. The current suite has 223 tests (220 pass, 3 stdio purity tests are environment-dependent and time out without a running MCP server process).
## Calibration sequence — final status
All 8 steps are now complete. The two that were in progress last time:
| Step | Previous status | Final status |
|------|----------------|--------------|
| 5 — Full re-annotation | “~50% complete” | **Done** — 24/24 queries, 1197 pairs, both 3-judge and 5-judge |
| 6 — Atomic ablations | “provisional, heuristic qrels” | **Re-validated** against both qrels_v2 and qrels_v3 |
## Two complete qrels sets
I now have two independent qrels for the same 1197-pair judgment pool:
**3-judge (qrels_v2):** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B. Majority >=2 of 3, DeepSeek tiebreaker. This is the run that had the parser-fallback contamination in q01-q07. I kept it as-is rather than re-running, because the comparison itself is informative.
**5-judge (qrels_v3):** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B + Llama-3.3-70B + Kimi K2.6. Majority >=3 of 5, DeepSeek tiebreaker. Fixed parser. This is the canonical qrels going forward.
One composition change since last post: I replaced GLM-5.2 with Llama-3.3-70B as the 4th judge. I ran a 25-document head-to-head test (5 queries, same prompts, alternating):
| Model | Exact match | Parse failures | Avg latency |
|-------|------------|----------------|-------------|
| Llama-3.3-70B | 24/25 (96%) | 0 | 1138ms |
| GPT-4.1-Nano | 18/25 (72%) | 0 | 796ms |
| GLM-5.2 | 15/25 (60%) | 8 | 1207ms |
| Qwen3.8-Flash | 1/10 (10%) | 5 | 5453ms |
| Mistral-Large-3 | 0/25 | 25 | unavailable |
GLM-5.2 had 8 parse failures in 25 calls and only 60% exact match. Llama-3.3-70B had 0 parse failures and 96% match. The full 5-judge run confirmed this: Llama achieved 100% coverage over 872 documents with QWK 0.79 against DeepSeek. The replacement was the right call.
I also switched Kimi-K3 to Kimi K2.6 mid-run (73% cost reduction, parser already handles code-fences). This is a model-version change, not a model-family change.
## 3-judge vs. 5-judge comparison
This was the empirical question I raised last time — whether the ensemble composition actually matters for this benchmark.
| Metric | Value |
|--------|-------|
| Per-doc agreement | 89.9% (1076/1197) |
| Disagreement | 10.1% (121) |
Grade shift matrix (3j rows × 5j cols):
```
| 5j:0 5j:1 5j:2
-------|-----------------
3j:0 | 709 40 8
3j:1 | 18 249 10
3j:2 | 7 38 118
```
The 5-judge ensemble is more conservative at Grade 2 (-27 docs) and more generous at Grade 1 (+50 docs). The biggest shift is 3j=0 → 5j=1 (40 docs) — these are documents the 3-judge called irrelevant that the 5-judge calls peripheral. The reverse shift (3j=1 → 5j=0, 18 docs) is smaller.
I attribute the Grade-2 conservatism partly to Kimi, who votes Grade 1 for 45.6% of all documents and only assigns Grade 2 at 1.8%. The other four judges assign Grade 2 at 8-16%. Kimi is the weakest ensemble member by pairwise agreement:
| Pair | QWK | Landis-Koch |
|------|-----|-------------|
| deepseek ↔ llama | 0.7901 | substantial |
| deepseek ↔ gpt-oss | 0.7753 | substantial |
| gpt-oss ↔ nemotron | 0.7620 | substantial |
| gpt-oss ↔ llama | 0.7425 | substantial |
| deepseek ↔ nemotron | 0.7361 | substantial |
| nemotron ↔ llama | 0.7322 | substantial |
| llama ↔ kimi | 0.3383 | fair |
| nemotron ↔ kimi | 0.2919 | fair |
| gpt-oss ↔ kimi | 0.2810 | fair |
| deepseek ↔ kimi | 0.2574 | fair |
All non-Kimi pairs are “substantial” (0.61-0.79). All Kimi pairs are “fair” (0.23-0.34). Kimi is a candidate for replacement in the next iteration. I have not replaced it yet because the ensemble decisions are robust to a single weak judge — in all 1197 pairs, the other 4 judges always had >=3 agreement, so Kimi’s vote never changed a majority outcome.
## Ranking stability — the key question
The most important result: **the system ranking is identical on both qrels versions.**
| Strategy | v2 nDCG@10 | v3 nDCG@10 | Delta |
|----------|-----------|-----------|-------|
| engine-count | 0.2607 | 0.2762 | +0.016 |
| bm25 | 0.2864 | 0.2915 | +0.005 |
| rrf | 0.3219 | 0.3592 | +0.037 |
| **rrf-quality** | **0.3930** | **0.4566** | **+0.064** |
Order on both: rrf-quality > rrf > bm25 > engine-count.
The 5-judge qrels lift rrf-quality by +16.3% (0.3930 → 0.4566) because Llama and DeepSeek identify more Grade-2 documents than GLM did (16.1% vs 3.1% Grade-2 rate). But the relative order of strategies doesn’t change. This is the strongest evidence I have that the benchmark is robust — two independent judge ensembles with different compositions and a parser-fix in between produce the same ranking conclusion.
## Atomic ablations — re-validated against both qrels
The ablation numbers from last post used heuristic qrels. I’ve now re-run them against both LLM-judge qrels. The relative ordering of components is consistent:
- Trust weighting remains the largest single contributor (+0.12 nDCG over plain RRF)
- Quality filter adds +0.057 nDCG independently
- Pure engine agreement remains the worst strategy
- BM25 underperforms RRF on this corpus
The absolute numbers shifted because the qrels changed, but the component ranking didn’t. Trust + Quality (the production configuration) wins on both qrels versions.
## External validation against NIST human qrels
This is new work I didn’t mention last time. I found a public dataset that allowed me to validate my judges against human ground truth: asukul’s RAG-Eval-LLM-Judge repository (Adisak Sukul, Iowa State University), which releases 537 stratified-balanced TREC RAG 2024 pairs with NIST human relevance labels.
I ran a 15% stratified sample (80 pairs) through my 5-judge ensemble:
| Judge | kappa vs NIST | QWK vs NIST | Coverage | Errors |
|-------|--------------|-------------|----------|--------|
| DeepSeek V4 Flash | 0.4095 | 0.5833 | 100% | 0 |
| GPT-OSS-120B | 0.3302 | 0.4419 | 100% | 0 |
| Llama-3.3-70B | 0.3239 | 0.5484 | 100% | 0 |
| Kimi K2.6 | 0.3091 | 0.4710 | 100% | 0 |
| Nemotron-3-Super | 0.2760 | 0.4250 | 100% | 11 |
| 5-judge ensemble | 0.3458 | 0.5090 | 100% | 11 |
For context, asukul reports a 9-judge ensemble kappa of 0.4941 against the same NIST qrels, and their DeepSeek V4 Pro achieved kappa=0.4705 but with only 39% coverage (212/537 valid responses).
My DeepSeek V4 Flash (kappa=0.4095, 100% coverage) has overlapping confidence intervals with their DeepSeek V4 Pro (kappa=0.4705, 39% coverage). The coverage difference is the more practically significant finding — V4 Flash returns valid responses for every pair, V4 Pro fails on 61%.
My ensemble kappa (0.3458) is below asukul’s (0.4941). I attribute this to three factors, in approximate order of magnitude:
1. **Domain mismatch.** My judge prompt is calibrated for onion-network security content (journalists, investigators, security researchers). asukul’s prompt is 4 lines of generic RAG instructions. On academic TREC queries, my domain-specific calibration hurts — the “commercial pages = grade 0” rule and “intent fit = security research” criterion shift thresholds in ways that don’t match NIST assessor behavior on academic topics.
2. **Scale collapse.** asukul uses a 4-grade scale (0,1,2,3). I use a 3-grade scale (0,1,2) and collapse 2+3 → 2. This loses information exactly where my judges are weakest — they correctly identify Grade 3 documents 85% of the time but Grade 2 documents only 45% of the time.
3. **Ensemble size.** 5 judges vs. 9 judges. More judges reduce variance, though the effect is smaller than domain and scale.
I did not run the full 537 pairs because the domain mismatch is structural — more data would confirm the gap, not close it. The validation was worth doing to confirm that my judges produce reasonable kappa against human ground truth (0.41 for DeepSeek is “moderate” agreement, not random), but I would not use TREC RAG as a primary validation target for an onion-search benchmark.
## Nemotron error analysis
I mentioned last time that Nemotron had infrastructure failures. In the full 5-judge run: 85/1197 errors (7.1%), all HTTP 429 (NVIDIA NIM rate limiting). 10 of 11 errors in the TREC validation run were on high-relevance pairs (human grade 2 or 3).
The error latency pattern is consistent with rate limiting: Nemotron’s OK responses average 981ms, error responses average 1125ms — the request goes out, waits, then gets rejected. On my onion corpus (longer prompts, slower throughput), Nemotron had 0 errors in 509 documents during the initial run. The TREC corpus has shorter passages and faster request rates, which triggers the rate limit.
Ensemble impact: zero. In all 85 error cases, the other 4 judges had >=3 agreement, so Nemotron’s fallback vote (relevance=1) never changed a majority outcome. The 5-judge majority design is robust to single-judge failure, but this is luck as much as design — if two judges failed simultaneously on the same document, the remaining 3 would decide, and the fallback votes could matter.
## Items from the “not done yet” list
All closed except the Nemotron lift experiment:
| Item | Previous | Now |
|------|----------|-----|
| Nemotron query refinement lift | “not measured” | **Partially measured, not with your design.** I ran a frozen-ranking test on 2026-08-25 using heuristic qrels (not the calibrated LLM-judge qrels): nDCG -0.0015, MAP -0.0008, MRR -0.0139 — no lift, 23/24 queries unchanged. But this was only the frozen-ranking half, with provisional qrels. Your full design — frozen-ranking test + live-acquisition test, with calibrated qrels — has not been run. It’s the first item in the post-benchmark phase. |
| stdio purity test | “not implemented” | Done — 3 tests, stdout = JSON-RPC only |
| Tor stream isolation | “configured, not observed” | Still configured-only; ControlPort observation deferred |
| Per-engine timeout citation | “don’t have source” | Done — Tor spec `cbtinitialtimeout=60s` + Tor Stack Exchange q/21966 |
| judged@10 / judged@20 | “not yet implemented” | Done — rrf=1.0, rrf-quality=0.892 (quality filter pushes some unjudged docs into top-10) |
| Pairwise engine overlap matrix | “not done” | Done — 13×13 Jaccard, only excavator↔oss=0.55 correlated, ahmia 97.9% unique contribution |
| bpref / incomplete-judgment | “not needed yet” | Still not needed — judged@10 >=0.89 for all strategies |
## What I’m taking from this
The benchmark answered the question I needed it to answer: **which ranking strategy should production use, and is that answer robust to judge-ensemble composition?** The answer is rrf-quality, and yes, it’s robust — the same strategy wins on both a 3-judge qrels with a known parser bug and a 5-judge qrels with the fix and a different ensemble composition.
The external validation against NIST human qrels confirmed that my judges produce moderate agreement with human assessors (DeepSeek kappa=0.41), which is not validation of correctness but is validation of non-randomness. The gap to asukul’s 9-judge ensemble (0.49) is attributable to domain and scale differences, not judge quality.
The qrels remain LLM-generated and provisional. I have not done human review of any of the 1197 pairs, and I won’t claim correctness I haven’t measured. What I can claim is that the ranking conclusion is stable across two independent judge ensembles, and that the judges produce moderate agreement with external human ground truth on an out-of-domain benchmark.
The next phase is production features — applying the benchmark findings to the actual search system. The benchmarking cycle is complete, with one exception: the Nemotron query refinement lift experiment you requested in post 4 (frozen-ranking test + live-acquisition test with calibrated qrels) has only been partially run — a frozen-ranking test with heuristic qrels showed no lift, but the full design with LLM-judge qrels and the live-acquisition arm is still open. It’s the first item on the post-benchmark roadmap.
Thanks again for the calibration sequence. It shaped the pipeline from start to finish, and the ranking-stability result is directly attributable to your emphasis on diverse pooling and double-annotation before full re-annotation.