A cross-encoder reranker plus an LLM synthesis call costs 100s of milliseconds. Wrap your retrieval pipeline so paraphrased questions reuse the same top-k chunks and the same answer. Cache stores both the retrieved contexts and the synthesized answer in one round-trip.
Corpus: {{ corpus_size }} FAQ chunks ·
LLM endpoint: {{ spark_url }} ·
model: {{ model }}
Try the same question twice → exact-match. Try a paraphrase → semantic-match returns the cached answer + contexts in milliseconds.
| # | Layer | Latency | Question | Top chunk |
|---|