# Benchmark Results — Yadgar

Published benchmark numbers for Yadgar's memory retrieval pipeline.

> **v5.26.0 — Full 500q Sonnet 4.6 run published (2026-06-01)**
> Phase 2 QA accuracy: **69.4% (347/500)**, Claude Sonnet 4.6 reader + judge, 470 min wall-clock.
> Phase 1 retrieval (reader-independent, 500q natural distribution): MRR 0.928, Recall@10 0.906, NDCG@10 0.863.
> This SUPERSEDES the Haiku pilot (96 stratified questions, 61.46%). Closes Adopt-1.

---

## v5.26.0 — LongMemEval Benchmark (Full 500q, Sonnet 4.6)

### Setup

| Field | Value |
|---|---|
| Yadgar version | v5.26.0 |
| Dataset | LongMemEval `s` variant (`longmemeval_s_cleaned`) |
| Dataset SHA-256 | `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442` |
| Dataset license | MIT — Wu et al., ICLR 2025 |
| Dataset source | https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned |
| Questions | 500 (full dataset, natural type distribution) |
| Embedding model | `all-MiniLM-L6-v2` |
| Reader + Judge LLM | `claude-sonnet-4-6` |
| Retrieval settings | NLI reranking ON, CE reranking ON, WRRF vector+FTS fusion (graph signals off: `WRRF_PPR_WEIGHT=0.0`) |
| Commit | `36bca02f2eee29f63e758793c4f3cc1daf13fe1a` (feature branch at run time) |
| Run date | 2026-05-31 → 2026-06-01 |
| Wall-clock | 470 min (~7.84 hours) via `claude -p` subprocess (Max 20x quota path; no cash spend) |

---

### Phase 1 — Retrieval Metrics (embedded in full run)

Run: `benchmarks/results/longmemeval_v5.26.0_s_full.json` (retrieval aggregates from 500q full run)

| Question Type | Count | MRR | Recall@5 | Recall@10 | NDCG@10 |
|---|---:|---:|---:|---:|---:|
| single-session-user | 70 | 0.961 | 0.984 | 0.984 | 0.967 |
| single-session-assistant | 56 | 0.982 | 0.982 | 0.982 | 0.982 |
| single-session-preference | 30 | 0.780 | 0.867 | 0.900 | 0.807 |
| multi-session | 133 | 0.930 | 0.815 | 0.893 | 0.832 |
| temporal-reasoning | 133 | 0.892 | 0.781 | 0.833 | 0.785 |
| knowledge-update | 78 | 0.979 | 0.924 | 0.931 | 0.893 |
| **overall** | **500** | **0.928** | **0.869** | **0.906** | **0.863** |

> Note: the 96q stratified pilot (16/type) had MRR=0.935, Recall@10=0.964, NDCG@10=0.913 on that balanced sample.
> The 500q natural distribution is weighted toward harder types (133 multi-session + 133 temporal-reasoning = 53% of sample),
> which explains the lower aggregated numbers vs the pilot — not a regression.

---

### Phase 2 — QA Accuracy

Run: `benchmarks/results/longmemeval_v5.26.0_s_full.json`
Model: `claude-sonnet-4-6` (reader + judge)

| Question Type | Count | QA Accuracy |
|---|---:|---:|
| single-session-user | 70 | 92.9% (65/70) |
| single-session-assistant | 56 | 96.4% (54/56) |
| single-session-preference | 30 | 33.3% (10/30) |
| multi-session | 133 | 55.6% (74/133) |
| temporal-reasoning | 133 | 63.9% (85/133) |
| knowledge-update | 78 | 75.6% (59/78) |
| abstention | 30 | 80.0% (24/30) |
| **overall** | **500** | **69.4% (347/500)** |

Wall-clock: 470 min (500 questions, sequential). Elapsed seconds: 28237.

**Failure-mode analysis:** Retrieval is strong (MRR ≥ 0.78 for every type), so QA gaps are
answer-synthesis or judge-strictness, not retrieval. Two structurally weaker buckets:

- `single-session-preference` (33.3%): preference questions have open-ended/subjective gold answers;
  strict-match judging penalises plausible alternatives even for a Sonnet-class reader. This is a
  judge rubric issue, not a memory retrieval issue.
- `multi-session` (55.6%): requires aggregating facts across sessions; even Sonnet loses precision
  under context dilution from 100+ retrieved sessions stacked in context.

Sonnet vs Haiku (stratified pilot) comparison (same bucket order):

| Type | Haiku 96q pilot | Sonnet 500q full | Delta |
|---|---:|---:|---:|
| single-session-user | 87.5% (14/16) | 92.9% (65/70) | +5.4pp |
| single-session-assistant | 93.8% (15/16) | 96.4% (54/56) | +2.6pp |
| single-session-preference | 12.5% (2/16) | 33.3% (10/30) | +20.8pp |
| multi-session | 31.2% (5/16) | 55.6% (74/133) | +24.4pp |
| temporal-reasoning | 75.0% (12/16) | 63.9% (85/133) | −11.1pp |
| knowledge-update | 68.8% (11/16) | 75.6% (59/78) | +6.8pp |
| **overall** | **61.46% (59/96)** | **69.4% (347/500)** | **+7.9pp** |

> `temporal-reasoning` shows −11.1pp vs the stratified pilot, almost certainly a sample-composition
> artifact: the pilot's 16 stratified temporal questions were a biased draw (pilot MRR 0.938) vs the
> 133 natural-distribution questions (full-run MRR 0.892). The hardest temporal questions are
> underrepresented in any stratified 16-question sample. Not a Sonnet regression.

---

### Comparison Table

| System | LongMemEval Variant | Questions | QA Accuracy | Notes |
|---|---|---:|---:|---|
| mem0 | s | 500 (full) | 94.4% | mem0 state-of-memory blog, 2026 (GPT-4o) |
| Zep | s | 500 (full) | 63.8% | Zep announcement (GPT-4o) |
| **Yadgar v5.26.0** | **s** | **500 (full)** | **69.4%** | **Sonnet 4.6 reader+judge; Adopt-1 closed** |

> Comparison is now apples-to-apples on sample size: all three systems use full 500-question evaluation.
> Reader/judge model differs: Yadgar uses Sonnet 4.6, competitors used GPT-4o.
> A weaker reader (Haiku) at 61.46% (96q pilot) confirms the gap was partly reader-model, not purely retrieval.

---

### Reproducibility

```json
{
  "yadgar_commit": "36bca02f2eee29f63e758793c4f3cc1daf13fe1a",
  "dataset": "longmemeval_s_cleaned",
  "dataset_sha256": "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442",
  "embedding_model": "all-MiniLM-L6-v2",
  "surreal_version": "3.0.5 for linux on x86_64",
  "reader_llm": "claude-sonnet-4-6",
  "judge_llm": "claude-sonnet-4-6",
  "python_version": "3.14.3 (main, Feb  3 2026, 15:32:20) [GCC 15.2.0]",
  "run_date_utc": "2026-05-31T17:02:30.497222+00:00",
  "elapsed_seconds": 28237,
  "questions": 500
}
```

### Reproduction commands

```bash
# Full 500q QA run (470 min, zero cash spend via Max quota claude -p)
uv run python benchmarks/run_longmemeval.py \
  --model claude-sonnet-4-6 \
  --output benchmarks/results/longmemeval_v5.26.0_s_full.json \
  --save-hypotheses benchmarks/results/longmemeval_v5.26.0_s_full_hypotheses.jsonl \
  --resume
```

---

## v5.26.0 — Haiku Pilot (96 stratified questions) — SUPERSEDED

Historical record. Superseded by the full 500q Sonnet run above.

Phase 1 retrieval (96q stratified, reader-independent): MRR=0.935, Recall@10=0.964, NDCG@10=0.913.
Phase 2 QA (Haiku reader+judge, 16/type × 6 types): **61.46% (59/96)**. Wall-clock 100.1 min.

Run files: `benchmarks/results/longmemeval_v5.26.0_s_retrieval.json` (Phase 1),
commit `1e63607182f0bcfa0db557aad419c29298392e86`.

---

## Dataset Attribution

**LongMemEval** (MIT License):

> Wu, Junhao, Shangyu Xing, Bin Wang, Shengyu Zhang, Wei Fan, Pengfei Liu, and Chunhua Shen.
> "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory."
> ICLR 2025. arXiv:2410.10813 (2024).
> Dataset: https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned

**LoCoMo** (CC BY-NC 4.0): Deferred — not used in v5.25.x or v5.26.0.
See `docs/BENCHMARK_LICENSE.md` for details.

---

## Caveats

- Full 500q run uses natural type distribution (not stratified). Retrieval aggregates differ from pilot due to harder type weighting.
- mem0 (94.4%) and Zep (63.8%) used GPT-4o; Yadgar used Sonnet 4.6. Different reader/judge models may favor different answer-generation styles.
- NLI reranking ON, graph signals OFF (`WRRF_PPR_WEIGHT=0.0`). D2 A/B (NLI on/off) deferred: no NLI-off run exists yet. See `docs/PLAN_V5_25_X_D2_NLI_AB.md`.
- PC algorithm / causal discovery untested: graph signals disabled in benchmark. D3 A/B deferred pending a run with `WRRF_PPR_WEIGHT > 0`. See `docs/PLAN_V5_25_X_D3_PC_AB.md`.
- `single-session-preference` low accuracy (33.3%) reflects open-ended gold answers penalised by strict-match judging. Not a retrieval failure (MRR 0.780).
