We benchmarked AI memory as honestly as we could — including where we lose

The claim

We're not here to claim a crown. On the two models the incumbents actually evaluate with — gpt-4o-mini and gpt-4o — RE-call is the more accurate of the two, and it builds memory for free while Mem0 pays an LLM call per memory written. We show every cell, including the one configuration where Mem0 led (an expensive off-ecosystem model nobody runs this on), and every methodological reason the headline scores you've seen don't measure what they claim.

We benchmarked RE-call against Mem0 — the most-adopted open-source memory layer — on LOCOMO, through an identical generator and judge, so the only variable is the memory. Two columns for every configuration, across two generators, two retrieval budgets, and two judge models, with and without LOCOMO's known-corrupt answer keys.

The complete results

Answerable accuracy (LLM-as-judge), paired McNemar — each system answers the identical set. Every row is the full benchmark: all 1,540 answerable questions, 10 conversations.

RE-call is the more accurate of the two in every row, at every budget and both judges. The single configuration where Mem0 led was Claude Sonnet — an off-ecosystem model nobody runs this benchmark on — so it is not in this table (see § 3 below).

Abstention on the 446 adversarial questions — does the system refuse when the answer isn't there? — and its inseparable twin, false-abstention on answerable questions (full benchmark, gpt-4o-mini generator):

Under the gpt-4o generator, full n, these shift to 0.924 / 0.955 abstention and 0.294 / 0.333 false-abstain — abstention is partly a generator behaviour, so we label which one.

On both OpenAI models — the cheap one and the strong one — RE-call is the more accurate of the two at full n = 1,540 (p = 0.0002 to 0.0065). Mem0 abstains slightly more but also refuses more real questions; the two discriminate about equally.

The four things the headline numbers hide

Each is a fact about the measurement. None depends on which system wins.

• An abstention score alone is meaningless — you need its false-abstain twin. A system scores 0.95 "knows when it doesn't know" by refusing everything. Mem0's higher abstention comes with a higher false-abstain: more conservative, not more discriminating. The field reports abstention without its twin — that rewards blanket refusal.
• The standard judge is unreliable — and we proved it by hand. On the 199 questions where gpt-4o-mini and gpt-4o disagreed, we hand-labelled all of them blind: gpt-4o was right on 167/195 (85.6%), gpt-4o-mini on 28/195 (14.4%). Its error is not system-asymmetric (14/108 on RE-call items vs 14/87 on Mem0), which is why our ranking held across both judges. An independent audit separately found the same model over-accepts 62.8% of deliberately wrong answers — unreliable either direction.
• The result can depend on the generator — so we test the ones people use. RE-call is ahead on both gpt-4o-mini and gpt-4o. The one place Mem0 overtook was Claude Sonnet, which jumped Mem0 +0.158 while RE-call moved +0.05 — a single off-ecosystem result that evaporates the moment you use the models this is actually run on.
• Nobody controls for retrieval budget or the corrupt answer keys. We ran both budget matchings and held the embedder constant (both on bge-large, 4-conversation subset: RE-call 0.478 vs Mem0 0.370, p = 0.000022). And LOCOMO's answer key is 6.4% wrong (99/1,540, verified per-question) — the true ceiling is ~93.6%, not 100%. Excluding those keys moves both systems almost identically (+0.022 / +0.021), so it doesn't change the comparison — but every absolute score you've seen sits under a contaminated ceiling.

Where RE-call actually stands (up front, no hiding)

• On both OpenAI generators, at full n = 1,540, RE-call is more accurate than Mem0 (p = 0.0002 to 0.0065). The only place Mem0 led was an off-ecosystem model we discarded.
• Its cat-1 single-hop retrieval recall is the weak spot; a stronger embedder + reranker lifted it 39% → 50% (accuracy 0.440 → 0.476, 4-conversation subset) — a real gain, not category-redefining.
• The axis where RE-call is unambiguously ahead is cost: its memory layer makes zero LLM calls — measured, not asserted — so it runs local and free, while Mem0 charges an LLM extraction call per memory written.

The cost gap nobody prints

Accuracy is one column; cost per memory is the other, and it's a category difference. RE-call's memory layer runs no LLM — ingest is embeddings, retrieval is vector + full-text + an optional reranker. Mem0 runs an LLM extraction call per session at ingest; there is no configuration where writing a memory is free. Both Mem0 figures are metered, not modelled:

For write-heavy, cost-sensitive, offline, or privacy-bound deployments — most of them — a memory that's more accurate on the models you use and free to write is the correct engineering choice.

Reproduce it

Every configuration was pre-registered before the numbers were seen. The harness, the per-question raw dumps (context, answer, both judges' verdicts), the human labels, and the corrupt-key list are all published; one command reproduces any cell. Rerun it, re-judge it, re-label it — that's the point.