SplatRAG 3+ Visualizer: SciFact 0.8804 / 0.7624 with 5,000 slang rows in the same memory

I did not try to reproduce SplatRAG v3 itself. The current implementation is not public, and your post uses Qwen3-Embedding-8B / 4096-D plus the Rust memory stack.

Instead I used a deliberately smaller public proxy:

• SciFact: 5,183 documents, the full 300-query test set
• Qwen3-Embedding-0.6B
• full 1024-D embeddings and normalized 64-D prefixes
• 5,000 out-of-domain public slang rows
• exact cosine search
• hnswlib as an ANN proxy
• PCA 64-D → 3-D for the geometry check

The 0.6B model is from the same Qwen3-Embedding family and is also Matryoshka-capable, but the numbers below should not be transferred to the 8B system. I mainly wanted to see whether the proposed controls actually produced useful distinctions.

The basic retrieval result was:

Across the 300 claims, full-D was better on 106, tied on 177, and 64-D was better on 17.

That does not tell us what the 8B 64-D/4096-D gap will be. It does tell me that dimension ablation can carry enough signal to be worth measuring explicitly, rather than treating 64-D and full-D as interchangeable views of the same retriever.

For reference, Qwen3-Embedding-8B explicitly supports Matryoshka-style output dimensions from 32 through 4096.

This is the part of the post I would investigate first, because you already identified it as unfinished: the 3-D union-find version produced the 4,035-paper well, and basin packing has now moved back to the 64-D space.

That separation — high-dimensional structure for the basin, low-dimensional projection for the picture — seems useful.

My proxy produced a fairly dramatic example of why.

For 64-D → PCA-3D:

• the first three PCA components explained only about 29.7% of variance;
• neighborhood trustworthiness at k=10 was about 0.84;
• at the same cosine threshold of 0.80, the largest connected component contained:
  • about 16% of points in 64-D;
  • essentially 100% in the normalized PCA-3D proxy.

That is not a reproduction of your basin code — in particular, I do not know whether your old 3-D union-find used exactly the same normalization/metric — so I would not conclude “PCA caused the 4,035-paper well.”

But it does show something narrower and probably useful:

A threshold that has a reasonable graph meaning in the embedding space can land in a completely different connectivity regime after projection.

The 64-D side was also surprisingly threshold-sensitive in the proxy:

cosine threshold    largest component
~0.668              ~94%
~0.700              ~48%
~0.750              ~39%
~0.800              ~16%
~0.850               <1%

So before changing the basin algorithm, I would probably just generate this curve for the real store:

threshold
    -> number of connected components
    -> largest_component_size / N
    -> maybe median / p90 component size

for both:

64-D basin space
3-D display space

If the giant well appears as a sharp transition, that tells you much more than a single threshold value.

There is another cheap way to look at the same structure. If the basin is effectively built by thresholding pairwise/k-NN edges and unioning connected components, then it is closely related to single-linkage connectivity. A single chain of locally valid edges can connect two regions whose endpoints are not particularly similar.

So instead of immediately replacing union-find with HDBSCAN/Leiden/etc., I would inspect the edges that cause the largest merges:

small components
      |
      | one bridge edge
      v
large component
      |
      | another bridge
      v
giant component

A single-linkage dendrogram / MST-style view can make those merge distances explicit. The standard SciPy hierarchical linkage implementation is useful as a reference for this interpretation.

If the 4,035-paper basin is mainly a few bridge documents, that suggests one class of fix.

If instead most of the graph is genuinely dense at that threshold, that suggests a different one.

And if only the 3-D projection collapses while 64-D remains well separated, then the current decision to keep 3-D as the picture rather than the index/basin geometry is probably already the important fix.

I would only reach for a different clustering family after that diagnostic.

The 64-D + full-4096-D arrangement reminded me strongly of Matryoshka Representation Learning, particularly its Adaptive Retrieval experiment.

Their basic pattern is:

small-dimensional representation
        |
        v
retrieve a shortlist
        |
        v
larger-dimensional representation
        |
        v
rerank the shortlist

They also describe a Funnel Retrieval variant where representation dimensionality increases while the candidate set shrinks.

The Hugging Face Matryoshka embeddings overview and the Sentence Transformers Matryoshka examples make the same shortlisting/reranking use case explicit.

That makes me think the most informative statistic for the 64-D path is not necessarily its final nDCG@10.

It is:

Does the 64-D stage preserve the candidates that the later stages need?

For example, I would measure something like:

64-D top-30  qrel recall
64-D top-50  qrel recall
64-D top-100 qrel recall

and, separately:

How many documents that would be high-ranked by full 4096-D
survive the 64-D shortlist?

That gives a much cleaner interpretation.

If 64-D has mediocre final top-10 quality but near-perfect relevant-document recall at top-50 or top-100, it may be doing its job perfectly well as a first-stage candidate generator.

Conversely, if relevant documents are already missing from the 64-D candidate set, no full-D reranker can recover them unless the full-D path also searches independently.

That leads to a small ablation that seems more informative than a large grid search:

If the three-arm RRF wins because 64-D and 4096-D retrieve different useful documents, that is interesting.

If BM25 + 4096-D already gives almost the same result, then the main value of 64-D may instead be speed/storage/candidate generation.

Both outcomes are useful; they just describe different roles.

I would also be slightly careful with the phrase “the remaining 4032 dimensions.” With Matryoshka representations, I would not treat dimensions 65-4096 as an independent feature block added on top of a complete 64-D vector. A safer interpretation is simply:

the full representation provides information/ranking signal that is not preserved by the current 64-D path.

One implementation detail worth checking while doing this: truncated Matryoshka embeddings may need to be renormalized after truncation if cosine similarity is being used. The Hugging Face Matryoshka guide calls this out explicitly.

Qwen3-Embedding is also instruction-aware, so I would keep the query prompt/instruction, pooling path, normalization, and indexed text fields fixed across this comparison. Otherwise a “64-D vs 4096-D” ablation can accidentally become several ablations at once.

I would separate these two claims:

A. Co-location / filtered-store test

Can unrelated domains physically coexist in the same memory/index while a SciFact-scoped query still behaves normally?

B. Distractor-competition test

If unrelated or confusing documents are actually eligible to rank, can they displace the correct SciFact evidence?

Your current domain=scifact evaluation seems much closer to A.

That is still a meaningful systems property.

In particular, filtered ANN is not always equivalent to “pretend the other vectors do not exist.” The behavior depends on where the filter is applied.

The current fast-hnsw API, for example, exposes an in-traversal search_filtered path where rejected nodes:

• cannot enter the returned top-k;
• but can still remain navigable as graph nodes.

If SplatRAG’s current path is using that style of filtering, then slang rows can still be part of the ANN topology even though they cannot appear as false positives.

I cannot tell from the public material whether your private call path is using exactly that API/version, so I would keep this conditional.

The possibilities are roughly:

same physical store
|
+-- separate/partitioned SciFact ANN
|      -> slang has little ANN effect
|
+-- shared HNSW + in-traversal filter
|      -> slang cannot rank, but can affect graph navigation
|
+-- global ANN first, post-filter later
       -> slang can consume candidate slots before filtering

Those are three different meanings of “same memory.”

My small proxy also suggested that ordinary slang is a very weak distractor for SciFact.

With 5,000 slang rows added and no domain filter at all, the 64-D exact result changed from roughly:

nDCG@10 0.55587

to:

nDCG@10 0.55528

Only about 11 of 3,000 top-10 slots were occupied by slang, and only one of the 300 queries changed nDCG.

So I would not interpret “5,000 slang rows” as a strong adversarial retrieval test by itself.

But I also would not remove that experiment.

I would simply name the two goals separately:

The current run is a nice heterogeneous co-location / filtered-memory sanity check.

If you later want retrieval-interference robustness, that can be a second experiment with harder distractors:

• scientific documents from another domain;
• lexical near-matches;
• semantically similar but non-relevant abstracts;
• deliberately confusing hard negatives.

That way the current test keeps its value instead of being judged against a different question.

At this corpus size, I would keep one exact-search path around as a diagnostic.

Not because HNSW looks broken — my proxy actually suggested the opposite.

At high ef, the proxy produced almost identical clean and mixed-filtered retrieval:

clean HNSW vs exact top-10 overlap          ~1.000
mixed + filtered HNSW vs exact overlap      ~0.9997

So I did not find evidence that heterogeneous co-location inherently damages HNSW retrieval.

But an exact baseline is still very useful because it splits:

representation / scoring problem

from:

ANN graph / search-parameter problem

A compact diagnostic is:

Then:

A ~= B and C ~= D
    -> mixed co-location is probably not the interesting variable

A ~= B but C != D
    -> inspect ANN/filter path, ef, graph construction

exact != HNSW
    -> inspect ANN fidelity before changing the embedding/fusion logic

This is also why I would measure exact-neighbor overlap/recall separately from task nDCG.

An approximate search can occasionally produce a better nDCG by accident even while being a worse approximation to the exact nearest neighbors. My proxy did this slightly at one lower-ef setting.

So:

task quality

and:

ANN fidelity

are worth keeping as separate axes.

The lexical signal clearly looks important in your current configuration, but I would probably avoid generalizing “SciFact is lexical-first” into a property of the dataset itself.

A reproducible Anserini SciFact BM25 baseline is indeed strong:

BM25 nDCG@10 = 0.6789

so there is nothing surprising about lexical retrieval doing a lot of work here.

But current Anserini BEIR regression/fusion tables also contain dense configurations that exceed BM25 on SciFact, so I would read your result more narrowly:

In this SplatRAG configuration, the lexical arm is carrying an important part of the useful signal.

That also makes the per-query view interesting.

Instead of only asking:

Did 4096-D improve aggregate nDCG?

I would look at:

Which claims did 4096-D rescue?
Which claims did it hurt?
Which were already solved by BM25?
Which relevant documents appear only in one arm?

With only 300 test queries, that kind of win/loss table is small enough to inspect and may tell you why the fusion works.

For example, if the full-D wins mostly come from paraphrastic claims where lexical overlap is weak, that tells a more useful architectural story than the aggregate score alone.

I would leave RRF tuning until after the representation roles are clear.

The current configuration has several coupled choices:

BM25 weight = 1.3
RRF k = 10
candidate depth = 30
64-D arm
4096-D arm

RRF itself is well established, but these are still hyperparameters. Current Elasticsearch RRF documentation exposes the analogous rank_constant and rank_window_size explicitly because both affect the fused ranking.

So if two architectural variants end up close, a small sensitivity check would be enough:

k:          10 / 30 / 60
BM25 weight 1.0 / 1.3
window:     current / somewhat larger

I would not do a giant grid.

One other low-cost separation: if these values are still being tuned, SciFact has a train split, so parameter selection can happen there and the 300-query test set can remain the final check.

That is especially useful now that the old first-50 evaluation has already been corrected: it keeps the new full-300 number as a clean anchor rather than gradually turning the same 300 queries into both development data and final evaluation.

A few things looked possible in theory but did not seem first-order after the proxy:

HNSW insertion order

I saw a small difference between sequential/interleaved/random insertion in one proxy run, and insertion-order sensitivity is a known HNSW topic.

But one seed/run is nowhere near enough to make it a diagnosis here.

I would only revisit this if the exact-vs-HNSW control starts moving unexpectedly as the append-only memory grows.

Hubness

High-dimensional k-NN hubness could contribute bridge-like nodes, but I would only measure it after finding suspicious bridge documents in the real basin graph.

No need to diagnose a phenomenon that may not be present.

Immediately replacing union-find with HDBSCAN/Leiden/etc.

Those may eventually be useful, but replacing the algorithm before plotting the connectivity transition would throw away an opportunity to understand the current representation.

The cheap question is first:

At what similarity level does the giant basin appear, and through which edges?

Making the slang test much bigger just because 5,000 had little effect

If generic slang is far from SciFact in embedding/lexical space, 50,000 rows may still answer almost the same question.

If retrieval interference is the goal, harder negatives are probably more informative than simply more easy negatives.