Semantic answer caching in production RAG — worth it, or stick to embedding/rerank caches?

Intermediate
1

Building RAG over internal company documents and policies. Accuracy matters far
more than latency to us.

Current caching (Redis): query-embedding cache, rerank-score cache, query-variant
cache. A retrieval-result cache is built but disabled. No answer cache of any kind.

That gap looks deliberate, but I’d rather hear it from people running one.

  1. At which layer do you cache semantically and of what kind (Embedding , Retrieval, Reranker sematic cache etc )?
    (a) final answer text
    (b) full response payload incl. citations
    (c) reranked top-k, generation still runs
    (d) retrieved candidate set, rerank + generation still run
    Has anyone used (c)/(d) specifically to keep generation grounded in real
    documents while still skipping retrieval? That seems safer than (a) but I see
    almost nothing written about it.

  2. What hit rate do you actually see in production on internal-document QA?

  3. What similarity threshold, and how do you stop near-miss paraphrases that mean
    genuinely different things (“2024 policy” vs “2025 policy”)?

  4. How do you invalidate when the knowledge base changes?

2

For accuracy-critical internal docs, answer-text caching (a) is the risky one, it’s exactly the “2024 policy vs 2025 policy” near-miss problem you’re describing, cosine similarity doesn’t know the difference between “similar phrasing” and “similar but wrong answer.” Most teams I’ve seen doing accuracy-first RAG stick to (d), caching the retrieved candidate set and still running rerank+generation, since it skips the expensive vector search but keeps generation grounded in fresh documents every time. That gets you most of the latency win from retrieval with basically none of the correctness risk from (a).

Hit rates for internal-doc QA tend to be much lower than consumer chatbot caches, often 15-30%, since employee queries are far less repetitive/paraphrased than public FAQ traffic, so the ROI of (a)/(b) is usually smaller than people expect anyway. On thresholds: cosine similarity alone won’t catch “2024 vs 2025 policy” type near-misses since date/version tokens barely move the embedding, you need an explicit metadata/entity check (extract dates, doc IDs, version numbers) that invalidates a cache hit even above threshold if those don’t match. Invalidation is easiest if you key retrieval-cache entries by source doc ID/hash rather than just query embedding, that way a document update maps directly to which cache entries to drop instead of guessing from query similarity.

3

For accuracy-first RAG, I would key any retrieval cache to a corpus generation, access-policy version, and retrieval configuration, not only document hashes. A document can stay unchanged while permissions, chunking, filters, or reranker behavior changes.

I would also keep a small “must differ” regression set beside the hit-rate metric: pairs such as 2024/2025 policy, employee/contractor, draft/approved, and regional variants. A cache that looks fast on aggregate but collapses those pairs is worse than no cache. Option (d) feels like the safest starting point; (c) only after those near-miss tests are boringly reliable.