OmniCache: local exact and near-duplicate caching for LLM coding agents

1Introduction

Autonomous coding agents repeat themselves. Within one session they re-read the same files, re-list the same directories and re-ask near-identical questions after a retry or a restart. Each repeat is a full round trip to the provider, billed per input token. Most of those tokens are context that has not changed since the last turn.

OmniCache runs on the developer's machine at 127.0.0.1:8000. The agent is pointed at it with ANTHROPIC_BASE_URL or OPENAI_BASE_URL. It serves a stored response when it can prove the request is a repeat, and forwards to the provider otherwise. This report makes four contributions:

1. A description of the lookup pipeline and its intent-gated thresholds, taken from the source (§2).
2. A reproducible latency characterisation of each cache path at 10,000 samples, replacing the 5-sample CLI benchmark (§4.1).
3. An end-to-end measurement of four synthetic agent workloads, in-process and over loopback TCP (§4.2–4.3).
4. A match-quality experiment and an audit of every public performance claim against what we measured (§4.4, §7).

3Methodology

3.1Environment

Table 2. Test environment. All runs used one machine with no other benchmark load.

3.2Experiments

1. Lookup microbenchmark. The same DualTierCache.lookup calls as omnicache benchmark, with 10,000 timed samples per path after 500 warm-up calls, timed with time.perf_counter_ns. The CLI defaults to 5 iterations, where p95 and p99 both equal the maximum. The harness is listed in Appendix A.
2. End-to-end load test. benchmarks.load_test with four generators from benchmarks/scenarios.py: copilot, agent_swarm, burst and multi_tenant. Each ran with n=200 and concurrency 8, three times, in two configurations: in-process ASGI (--url internal) and over loopback TCP against omnicache start.
3. Match quality. We stored the 8 base prompts from scenarios.py. We then issued 40 paraphrases (5 templates × 8 prompts) and 20 unrelated prompts, and recorded status, similarity and whether the returned answer belonged to the right prompt.

3.3Controls and caveats

The TCP server restored 63 entries from its SQLite snapshot at start-up. Runs 2 and 3 are therefore warmer than run 1. We report the median of three runs and give the in-process run 1 separately as the coldest case.

5Discussion

The cache is not the bottleneck. At 9 µs for L1 and 0.18 ms for L2, lookup cost is at least two orders of magnitude below any network round trip. Optimisation effort is better spent on the HTTP path: that is where an agent's cached request spends roughly 90% or more of its 1.7 ms (single client) to 14 ms (8 concurrent clients).

"Replay in 0.08 ms" describes the lookup, not the request. The figure is in the right range for an in-process lookup; we measured lower. What the agent experiences is an HTTP response. Its latency is 20–175× that number, depending on concurrency. Both numbers are good. Only the second one is user-visible.

Gating works, and costs a little recall. The strict classes rejected one legitimate paraphrase and accepted no false ones. For a tool whose failure mode is silently wrong code, that is the right trade-off. The keyword rules are coarse, though. "Calculate" in a SQL question makes it a math question.

Savings are a property of the workload. Cost saved is roughly the hit rate times the cost of the calls avoided. Neither can be measured on synthetic generators. The 40–70% bill reduction in the README is plausible for agent sessions dominated by file re-reads. It remains a hypothesis until it is measured on recorded sessions.

6Limitations and threats to validity

• No real provider. Misses were served by a local stub, so miss latency and cost were not measured.
• Synthetic workloads. Hit rates come from generators built to repeat. No recorded agent sessions were replayed.
• Easy paraphrase set. Four of five templates differ from the stored prompt only by stop words, so recall is an upper bound. A set of human-written paraphrases is needed.
• Lexical embedder only. Results apply to the default FastHashEmbedder. The ONNX and quantized backends were not tested.
• One machine. A shared-tenancy cloud VM. Tail latencies (p99.9 and above) are sensitive to noisy neighbours.
• Warm snapshots. The TCP server restored 63 entries at start, which inflates TCP hit rates relative to a cold start.
• Open anomaly. The copilot TCP slowdown (§4.2) is unexplained.
• Small committed samples. The repository's latest_report.json uses n=40 per workload, where p99 equals the maximum. It agrees with our TCP medians (6.5–8.9 ms vs 9.7–13.9 ms) but should not be cited for tails.

8Reproducibility

Everything in this report can be regenerated from a clean checkout in about ten minutes.

Copy# install
$ python -m venv .venv && . .venv/bin/activate && pip install -e .

# §4.1 lookup microbenchmark (CLI, 5 iterations; see Appendix A for n=10k)
$ omnicache benchmark

# §4.2 end-to-end, in-process (repeat 3x)
$ python -m benchmarks.load_test -n 200 -c 8 --url internal --quiet --export-json e2e.json

# §4.2 end-to-end, loopback TCP (repeat 3x)
$ omnicache start &
$ python -m benchmarks.load_test -n 200 -c 8 --url http://127.0.0.1:8000 --quiet --export-json tcp.json

Appendix A · Lookup harness

Copyimport time, statistics
from core.vector_cache import DualTierCache

N, WARM = 10_000, 500
cache = DualTierCache()
stored = {"model": "claude-3-5-sonnet", "temperature": 0.0,
          "messages": [{"role": "user", "content": "Write a python fast fourier transform algorithm"}]}
cache.store(stored, {"choices": [{"message": {"role": "assistant", "content": "import numpy as np..."}}]})
variant = {**stored, "messages": [{"role": "user", "content": "Write a python fast fourier transform (FFT) algorithm in numpy"}]}
novel = lambda i: {**stored, "messages": [{"role": "user", "content": f"Unrelated question {i}: explain B-tree node splits"}]}

def sample(make):
    xs, statuses = [], set()
    for i in range(WARM + N):
        p = make(i)
        t0 = time.perf_counter_ns()
        status = cache.lookup(p)[0]          # HIT_EXACT | HIT_SEMANTIC | MISS | BYPASS
        dt = (time.perf_counter_ns() - t0) / 1e6
        if i >= WARM: xs.append(dt); statuses.add(status)
    xs.sort()
    return statuses, {q: xs[int(len(xs) * q)] for q in (.5, .9, .95, .99, .999)}

for name, fn in [("l1", lambda i: stored), ("l2", lambda i: variant), ("miss", novel)]:
    print(name, *sample(fn))

—References

1. core/vector_cache.py: DualTierCache.lookup, classify_intent
2. core/embeddings.py: FastHashEmbedder, AutoEmbedder, FastSemanticEmbedder
3. benchmarks/scenarios.py, benchmarks/load_test.py: workload generators and load driver
4. benchmarks/reporter.py, benchmarks/latest_report.json: committed report and baseline constants
5. server/gateway.py, server/cli.py: HTTP entry point, omnicache benchmark