Benchmark

Each strategy ingests the same chat-session corpus, recalls answers to the same questions, and is scored by the same evaluator. Click a column header to sort. Per-category metrics show the count of questions in parentheses; "—" means no question of that category was evaluated.

0 strategies

Headline metrics

StrategyAccuracy ▼iLLM-judge accuracy averaged over all evaluated questions. Opus grades each answer 0..100 against the reference; the headline number reports the mean / 100.Recall@5iMean session-level recall@k: did the strategy retrieve the gold-truth supporting session id inside its top-k results? Top-k held at 5 across all strategies.LatencyiAverage wall-clock recall latency per question, end-to-end (retrieval + generation). Lower is better.CostiTotal USD spent across the run (ingest LLM calls + recall LLM calls). Vendor-internal costs the SDK pays out-of-band are not included — see the README footnote.QuestionsiNumber of questions evaluated for this strategy on this corpus. The cost cap can halt a run mid-corpus, in which case this drops below the corpus total.

Memory-specific axes

These three metrics only count over questions in the relevant category. The number in parentheses is the question count. "—" means no question of that category was evaluated, so the metric is not defined.

StrategyUpdate PrecisionTemporal CorrectnessAbstention F1

Question categories

Information Extraction

Single-fact recall from one session. Example: 'What programming language did the user say they prefer?'

Multi-Session Reasoning

Combine facts from 2+ sessions to answer. Example: 'Given the user's job and city, what is their commute like?'

Temporal

Time-aware queries that require knowing when something was said. Example: 'When did the user first mention mountain biking?'

Knowledge Update

User changed their mind across sessions. Test: does the system reflect the latest version, not an earlier one?

Abstention

Question has no answer in the chat history. Test: does the system correctly say 'I do not have that information'?