April 1, 2026

What Happens When AI Assistants Forget: Benchmarking Structured Knowledge vs Session Memory

We ran four experiments across three fictional domains to measure how well structured knowledge objects preserve information compared to compressed summaries. The results show clear strengths and honest weaknesses.

The Problem

AI coding assistants maintain session memory through context summaries and markdown files. When conversations grow long, these summaries get compressed: specific numbers are dropped, rationale is flattened, constraints are generalized. The assistant retains a rough sense of what happened but loses the details.

This matters in practice. If an assistant forgets that a pipeline's batch size was reduced from 512 to 64 after an OOM crash, it may suggest 512 again. If it forgets that all database writes must go through the validation layer, it won't push back when someone proposes a direct insert.

We tested whether storing knowledge as structured objects (subject-predicate-value triples) preserves information that compressed summaries lose. All experiments use entirely fictional domains with zero prior knowledge in the model's training data.

Methodology

Every domain in these experiments is invented. The facts, system names, team members, and technical decisions are all fictional. This ensures the model cannot answer from training data. Baselines confirm this: the no-memory condition scores 10-21%, reflecting pure guessing.

Scoring uses keyword matching. Each question has a set of expected keywords and a minimum threshold. A question passes if the response contains enough expected terms. This is deliberately conservative: partial credit is not awarded.

Three Memory Conditions

Experiment 1: Main Benchmark (Nexus Domain)

Domain: Nexus, a fictional bioinformatics pipeline for protein folding validation. 50 invented facts across 5 simulated sessions, 25 questions in 3 categories. All facts are fictional (enzyme names, threshold values, team decisions). 3 seeds, zero prior knowledge.

Aggregate Results (3-seed mean)

Category Breakdown (seed 42)

The .md condition performs almost identically to no memory on fact retention and decision recall. Both score near zero. Only constraint enforcement shows a gap: 71% for .md vs 43% for none, because even vague rules ("always validate inputs") are enough for the model to push back on some violations. KO memory retains everything.

Experiment 2: Sharing Value Curve (Helios Domain)

Domain: Helios, a fictional quantum calibration platform for satellite constellations. 40 invented KOs across 4 categories (architecture, calibration science, team processes, operational constraints). 30 questions. Since every fact is fictional, the 0% sharing baseline is a true zero.

Constraint enforcement benefits the most from sharing. At 0% sharing, constraint questions score 30%. At 100% sharing, they reach 100%. This is the steepest improvement of any category, suggesting that operational rules are particularly hard to guess and particularly well-served by structured storage.

The single failure at 100% sharing (97%, not 100%) confirms the benchmark is not trivially easy.

Experiment 3: Cross-Session Continuity (Aether Domain)

Domain: Aether, a fictional IoT fleet management platform. 30 KOs across 3 sessions where decisions evolve and override each other (e.g., session 1 sets a polling interval to 30s, session 3 changes it to 5s after a latency incident). 20 questions in 3 categories.

On cross-session facts, KO scores 100% vs .md's 25%. These are questions like "What threshold was set in session 1 and then referenced in session 3?" The .md summary compresses away the session-specific details. KO preserves them. The .md condition drops to 55% overall (from 65% in the previous run) due to tighter scoring on synthesis questions.

Experiment 4: Scaling Curve (Nexus + Distractors)

Setup: 50 real Nexus KOs plus increasing numbers of distractor KOs drawn from 5 unrelated fictional domains. All KOs are dumped into context with no retrieval filtering. 25 questions about the Nexus facts only.

Honest Limitations

• Keyword scoring is coarse. A response that paraphrases the correct answer in different words may fail. An LLM-as-judge approach would give partial credit but introduces its own biases. We chose the conservative option.
• The .md summaries are hand-written. Real compaction output varies by model and session length. Our summaries are representative but may be more or less lossy than any specific compaction run.
• Only 3 seeds on the main benchmark. Standard deviation is low (0-5%), but more seeds would strengthen confidence.
• KO previously lost on override detection (Experiment 3). Storing all versions of a fact created ambiguity. Subject-level deduplication fixed this: only the latest version of each subject+predicate pair is retained in context. KO now scores 100% on overrides.
• Scaling hits a cliff at ~250 KOs (Experiment 4). Without selective retrieval, structured memory degrades to baseline levels. This experiment uses raw context dumping, not a retrieval pipeline. Real deployments need filtering.
• All domains are fictional and relatively small. Real-world projects have messier facts, more ambiguity, and larger knowledge bases. These results represent a controlled upper bound.

What We Learned

Structured objects outperform compressed summaries for fact retention and decision recall. The gap is large: 100% vs 0-23% on specific facts. When you need exact values, thresholds, or rationale, compressed text loses them and structured storage does not.

Subject-level deduplication eliminates the override problem. An earlier version of this benchmark showed .md outperforming KO on override detection (86% vs 43%), because KO stored both old and new values. The fix is straightforward: deduplicate by subject+predicate so only the latest version enters context. After dedup, KO scores 100% on all continuity categories.

Sharing even a small fraction of knowledge has outsized impact. Going from 0% to 25% sharing triples accuracy (13% to 50%). The value curve is steepest at the low end.

Raw context dumping does not scale. At ~250 KOs, performance collapses. Any system that stores more than ~100 knowledge objects needs selective retrieval, not wholesale injection.

Benchmark run on Claude Sonnet (claude-sonnet-4-20250514), April 1, 2026. 4 experiments across 3 fictional domains (Nexus, Helios, Aether). All facts invented, zero prior knowledge confirmed by baselines. Full results and reproduction scripts at github.com/coretxinc/coretx-connect.