# Memory Benchmarks — Plan + Baseline

**Status:** Active draft (baseline established 2026-04-26)
**Owner:** Ben Booth
**Related:** `prd-memory.md` §3, `spec-memory.md` §12 (compliance checklist)

---

## 1. Why benchmark

Three reasons specific to Axiom Memory's positioning as *the* choice:

- **Differentiator claims need numbers.** "Cryptographic provenance," "federation-native," "DPM-aligned," "classification-aware" each map to a measurable invariant. Without measurements they're marketing.
- **Migration moves need a tripwire.** ADR-033 stages 2/3/4/5 each change a layer. Benchmarks catch silent regressions when the L2 graph changes how concepts are extracted, or Stage 4 swaps the blob backend.
- **Extension outcomes need ground truth.** "Aggregate views reflect reality," "forget actually erases," "interaction N+1 sees interaction N" — end users notice when these break, regardless of which extension surfaces them.

The plan: ship a baseline this week against current Stage 1; ratchet targets per stage; gate every release on the suite.

## 2. The compliance suite — invariants this is *the choice* for

These are the invariants no competitor architecture can match without an architecture change. Every release MUST pass.

| ID | Test | Pass condition | Maps to |
|---|---|---|---|
| **C1** | Replay determinism | `project(events≤t, task) == project(events≤t, task)` over N=1000 runs | DPM I14 |
| **C2** | Tombstone propagation | A tombstoned fragment never appears in any projection ≤ 1 read cycle later | spec-memory I4, I15 |
| **C3** | Per-scope isolation | `EventStore.list(scope=A)` never returns scope-B fragments under any read pattern | spec-memory I1 |
| **C4** | Federation isolation | A `SCOPE_INTERNAL` fragment never appears in any `FederationGateway.project_for_peer` output | spec-federation-policy §6 |
| **C5** | Classification trump | A `CUI` fragment with `visibility=PUBLIC` collapses to `SCOPE_INTERNAL` outflow at the gateway | spec-federation-policy §4 |
| **C6** | Provenance integrity | Every projection cites composing fragment_ids; every concept cites `extracted_from`; every edge cites `evidence` | spec-memory I10, I16 |
| **C7** | Extractor classification gate | A CUI fragment never receives an extractor whose `capability.max_classification < cui` | spec-memory I11 |
| **C8** | Forget actually erases | After `EventStore.tombstone(id)`, ALL projections in ALL scopes return zero matches for content of `id` | spec-memory I4, I15 |
| **C9** | Hop-bounded federation | No federation operation iterates the full peer set; default `max_hops=1`, max `2` enforced | spec-federation-policy §6, project_federation_scale_target |
| **C10** | Backward-compat decode | Fragments missing `visibility` / `classification` / `event_time` decode with documented defaults; no `KeyError` | spec-memory §3.1 |

These are deterministic tests, not probabilistic. A single failure is a release blocker.

## 3. Performance suite — measurable per stage

Numbers to track. Targets ratchet per ADR-033 stage; baseline is current Stage 1 (2026-04-26).

| Metric | Baseline (Stage 1, SQLite, 11k frags) | Stage 3 target | Stage 4 target | Stage 5 target |
|---|---|---|---|---|
| `CompositionService.write` p50 | 2.6 ms | ≤ 5 ms (with signing) | ≤ 5 ms | ≤ 5 ms |
| `RecentActivityProjection` p95 (window=5) | 153 ms @ 11k frags | ≤ 50 ms with cache | ≤ 50 ms | ≤ 50 ms |
| `RecentActivityProjection` p95 (window=5) | 13 ms @ 1k frags | ≤ 20 ms (preserved) | ≤ 20 ms | ≤ 20 ms |
| `EventStore.list(scope)` p95 | linear scan (no index) | ≤ 30 ms @ 100k frags | ≤ 30 ms | ≤ 30 ms |
| `BlobStore.put(1MB)` p95 | n/a (not yet extracted) | n/a | ≤ 10 ms | ≤ 10 ms |
| `FederationGateway.project_for_peer` p95 | n/a (not implemented) | n/a | n/a | ≤ 200 ms |

Reproducible numbers below in §6. Write throughput is fsync-dominated (each `cs.write` commits a SQLite transaction); write batching is a Stage 3 concern.

## 4. Public benchmarks — calibrate vs the field

### 4.1 LongMemEval — long-term agent memory

Tests an agent's ability to recall + use information across thousands of conversation turns. Five capabilities: information extraction, multi-session reasoning, knowledge update, temporal reasoning, abstention.

**Plan:**
- Build `axi memory bench longmemeval` that drives the LongMemEval test set against an Axiom-backed agent (chat extension over the L1 + RecentActivityProjection wiring).
- Compare against published Mem0 v0.1, Letta, MemGPT numbers on the same set.
- Target: parity at v0.1, surpass after concept graph (Stage 2) lands.

**Baseline status:** not yet measured. Run after the chat extension consumes Layer 3 (likely Phase 3, post Stage 2).

### 4.2 LoCoMo — long conversational memory

Multi-session conversational consistency. 30 sessions per profile, ~1000 turns total.

**Plan:**
- Same harness as LongMemEval, different drivers.
- Public scores published; ours go in `working/memory-benchmarks-results-YYYY-MM-DD.md` per release.

### 4.3 MemBench — episodic memory eval

Multi-hop, multi-session reasoning. Smaller than LoCoMo; useful for fast iteration during Stage 2/3.

### 4.4 Where these benchmarks miss us

LongMemEval / LoCoMo / MemBench all measure **information-recall**. They don't measure:

- Federation isolation (their world is single-tenant)
- Classification trump (no regulated regimes in the test sets)
- Replay determinism (no event log substrate to replay)
- Provenance integrity (no provenance contract)

So public benchmarks calibrate the *table-stakes* layer. Our compliance suite (§2) is what defends the differentiation.

## 5. Custom Axiom benchmarks — the differentiation surface

Beyond the compliance suite (which is binary), these are quantitative + comparative.

### 5.1 Aggregate-view accuracy vs ground truth

**Setup (extension-agnostic):** Run a synthetic scope (10 memory subjects, 200 interactions over a week). Collect:
- The actual interaction log (ground truth) from L1
- A scope-operator's aggregate view (the consuming extension's brief / summary / status / report — whichever it ships)

**Measure:**
- **Coverage:** % of distinct topics in the log that appear in the aggregate view
- **Fabrications:** topics in the aggregate view that don't appear in the log (target 0)
- **Edge-case detection:** subjects whose `unanswered_count` (or extension-equivalent metric) exceeds a threshold are surfaced in the view

**Baseline status:** No automated ground-truth comparison yet. The first canary extension's aggregate-view command exists; full automation is Stage 3 work.

### 5.2 Forget propagation latency

**Setup:** Write fragment F at t=0. Issue forget(F) at t=1. Probe N projections at t=2.

**Measure:**
- **Latency:** time from forget call to "F invisible in all projections" (target: ≤ 1 read cycle, < 100 ms)
- **Completeness:** % of projections F is invisible from (target 100%)
- **Audit preservation:** raw event log still contains both F and the tombstone (target 100%)

**Baseline status:** The canary extension's retraction primitive passes the binary test; latency not yet measured.

### 5.3 Episodic continuity

**Setup:** A simulated memory subject runs 5 interactions across 5 sessions. Each session, measure how many of the previous N interactions appear in the LLM context.

**Measure:**
- **Window faithfulness:** when `RecentActivityProjection(window=5)` runs, does it return the actual most-recent 5? (Sanity)
- **Semantic relevance:** of the N=5 returned, how many are semantically relevant to the current question? (Future Stage 2 with concept graph: this should improve.)

**Baseline status:** RecentActivityProjection ships and is tested for ordering; semantic relevance becomes measurable post-Stage 2.

### 5.4 Federation isolation under load

**Setup:** Two cohorts A and B with overlapping members. Issue concurrent ask + brief + federate-projection requests. Probe whether B-private fragments ever surface in A's view.

**Measure:**
- **Isolation:** zero leaks across 10k concurrent requests (target 100%)
- **Throughput:** concurrent ops/sec (baseline TBD; targets per profile)

**Baseline status:** unit tests cover the binary case; load-test harness is Stage 5 work.

### 5.5 Classification handling under multi-regime composition

**Setup:** Generate fragments combinatorially: every level × every compartment × every export-control regime × every visibility. Probe outflow + extractor selection + retention.

**Measure:**
- All 13 classification scenarios from `spec-classification-boundary.md` pass.
- Most-restrictive composition rule honored (already covered by `TestClassificationOutflowCeiling`).
- No external-provider extractor ever invoked on CUI (already covered by I11; needs runtime audit).

**Baseline status:** unit-tested for the rules; runtime audit harness is Stage 5.

### 5.6 Provenance traceability

**Setup:** Write 100 fragments, run an extractor over them, run a projection, ask "show me everything that produced this projection result."

**Measure:**
- **Coverage:** every projection output cites composing fragment_ids (target 100%)
- **Concept-back:** every concept cites `extracted_from` (target 100%)
- **Edge-back:** every concept edge cites `evidence` (target 100%)

**Baseline status:** RecentActivityProjection result carries fragments by reference (passes I16 trivially). Concept graph not yet shipped.

## 6. Baseline numbers — Stage 1, current

Established 2026-04-26 on a development workstation (M-series Mac, Python 3.14, single-process). All numbers are wall-clock means over 1000 runs unless noted.

### 6.1 Write throughput

```
InMemoryBackend (fastest possible — no fsync):
  Write 1000 fragments:                     2041 ms = 2.04 ms/fragment
  RecentActivityProjection over 1k:         9.9 ms

SQLiteBackend (production default for Edge / Workstation profiles):
  Write 1000 fragments:                     2973 ms = 2.97 ms/fragment
  RecentActivityProjection over 1k:         13.1 ms

  Write +10000 more fragments (cumulative 11k):
                                            25588 ms = 2.56 ms/fragment
  RecentActivityProjection over 11k:        153.2 ms
```

### 6.2 Determinism

```
Replay determinism (2 runs, same projection):  PASS (byte-equivalent)
```

### 6.3 What the numbers say

- **Write is fsync-dominated.** Each `cs.write` commits a SQLite transaction with audit-log fsync. Batching writes is the obvious Stage 3 lever — `~10x` headroom available.
- **Projection is O(N) over the artifact registry.** `RecentActivityProjection` iterates every artifact and filters in Python. At 11k fragments it's 153 ms; at 100k it'll be ~1.5 s. Stage 3 cache OR a `SELECT … WHERE kind='fragment' AND data_json LIKE '%scope_id%'` SQLite query lifts this by 10x+.
- **Determinism is structural.** No measurement noise; either it's pure or it's not. Today: pure. Stage 2's async extractor scheduling is the next determinism risk; the `ConceptGraph.snapshot_at(t)` invariant is how we defend it.

### 6.4 Reproducing the baseline

```bash
source ../.venv/bin/activate
python3 docs/working/memory-benchmarks-baseline.py
```

Pinned baseline run captured in `docs/working/memory-benchmarks-baseline-2026-04-26.json`. Subsequent runs go in dated siblings; CI compares against the most recent baseline.

Compliance suite runs as a focused pytest marker:

```bash
pytest tests/memory/ -m memory_compliance
```

10 invariants pass against current Stage 1; 4 are explicitly skipped pending Stage 2 / Stage 5 (extractor classification gate, concept-graph provenance, federation gateway scope leakage, hop-bound enforcement).

The path scope (`tests/memory/`) is intentional: unrelated extension-collection errors elsewhere in the tree don't poison this gate.

## 7. The harness — `axi memory bench`

The proposed CLI surface (Stage 3 work):

```
axi memory bench                            # run the compliance suite (§2)
axi memory bench --perf                     # run the performance suite (§3)
axi memory bench --public longmemeval       # run a public benchmark
axi memory bench --custom brief-accuracy    # run a custom benchmark
axi memory bench --json                     # machine-readable output for CI
axi memory bench --baseline                 # write current numbers to disk for next-run comparison
axi memory bench --compare <baseline.json>  # report regressions vs prior baseline
```

Default invocation is the compliance suite. CI gates on no-regression vs the most recent merged baseline.

## 8. Iteration cadence

Per-stage measurements; per-release release-note section.

| Cadence | Action |
|---|---|
| Per commit (CI) | Compliance suite (§2). Binary pass/fail. Block merge on failure. |
| Per release | Performance suite (§3). Numbers go in release notes. Regressions ≥ 20% require justification. |
| Per ADR-033 stage | Custom benchmarks (§5) refresh. Trend graph in `working/memory-benchmarks-trend.md`. |
| Per quarter | Public benchmarks (§4). Compare against latest published competitor scores. |

## 9. What to build first — the actually-this-week list

1. **Compliance test extraction.** Lift §2 invariants into `tests/memory/compliance/test_*.py`. Most are already covered by existing tests; they just need to be tagged and grouped under a `compliance` marker so `pytest -m compliance` runs them as a single gate. **Effort: ~3 hours.**
2. **Performance baseline harness.** Standalone Python script `docs/working/memory-benchmarks-baseline.py` that produces §6.1 numbers reproducibly. Commit alongside this doc. **Effort: ~1 hour.**
3. **CI gate.** Add `pytest -m compliance` to the GitHub Actions workflow. **Effort: ~30 min.**
4. **`axi memory bench` skeleton.** CLI command that calls into the compliance + perf suites. Stub now; stages flesh it out. **Effort: ~2 hours.**

That's a full day. Yields: a baseline pinned in version control, a CI gate that catches regressions, a CLI affordance to extend.

Stage-by-stage from there:

- **After Stage 2 (concept graph):** add C7 runtime audit; add §5.3 semantic-relevance measurement; benchmark `ConceptGraph.neighbors()` p95.
- **After Stage 3 (more projections):** add §5.1 brief accuracy; cache benchmarks for the projection layer.
- **After Stage 4 (blob store):** add `BlobStore.put` / `BlobStore.gc` perf; classification-aware blob outflow.
- **After Stage 5 (federation gateway):** add §5.4 federation isolation under load; LongMemEval / LoCoMo runs.

## 10. What's measured vs what's claimed

To pre-empt the question "but how do we know Axiom is *the* choice?":

| Claim | Test |
|---|---|
| "Cryptographic provenance" | C6 + audit-log signature verification (existing) |
| "Federation-native" | C4 + C9; performance baseline at peer-set size |
| "Classification-aware" | C5 + C7 |
| "DPM-aligned" | C1 (replay determinism) |
| "MIRIX cognitive taxonomy" | spec-memory §3.2 type-validators |
| "Sovereign LLM extraction" | C7 (extractor classification gate) |
| "Apache 2.0 + open governance" | LICENSE + governance doc |
| "Pluggable backends; no vendor lock" | Multiple backend impls passing the same compliance suite |
| "Shadow-memory discipline" | `axi ext lint` + `axi ext shadow-audit` per spec-memory §9 |

Every claim has a test. That's how the benchmark plan defends the positioning.

---

_Copyright (c) 2026 The University of Texas at Austin and B-Tree Labs. Apache-2.0 licensed._
