Memory Arena
Which memory architecture works best for your agent? Sixteen systems (Mem0, Graphiti, Cognee, LangMem, Memori, plus pure-Python baselines and advanced retrievers like HyDE, RAPTOR, Reflection, and Karpathy's LLM Wiki) run on the same LongMemEval corpus, scored by the same evaluator. Empirical evidence, not vendor-tuned numbers.
Active corpora: LongMemEval-S
How it works
Same corpus
All 16 strategies ingest the LongMemEval-S sessions in the same order. Each is namespaced by run id so concurrent runs do not contaminate each other.
7-axis evaluator
Structural checks, source attribution, LLM judge (Opus), plus three memory-specific axes: temporal correctness, update precision, and abstention F1.
Honest cost
Ingest-side LLM calls (entity extraction, summarization) are tracked separately from recall cost. Most vendor numbers hide ingest cost. Memory Arena does not.
The 16 strategies
Three baselines, seven advanced retrieval patterns (BM25, Hybrid RRF, HyDE, Persona Profile, Reflection, RAPTOR, Karpathy's LLM Wiki), and six vendor SDKs (Mem0, Mem0+Graph, Graphiti, Cognee, LangMem, Memori).
Full Context
Stuff every ingested turn into the prompt up to the model context limit. Ceiling baseline for what an LLM could answer with perfect recall.
Recency Window
Last N turns across all sessions. Cheapest baseline. Approximates a chatbot with no memory beyond its current scrollback.
Naive Vector
Embed every turn with text-embedding-3-large, retrieve top-k by cosine similarity. The 'put it in a vector DB' default.
BM25
Pure-Python lexical baseline using rank-bm25. Old-school keyword search, what Google did before vectors.
Hybrid RRF
Reciprocal Rank Fusion (k=60) over a vector ranking and a BM25 ranking. Captures both meaning matches and exact-keyword matches.
HyDE
Hypothetical Document Embeddings: ask the LLM to write a plausible answer first, embed THAT, retrieve real turns by similarity, then answer.
Persona Profile
Build a one-page JSON profile of the user via Haiku once before recall, then stuff that profile into every answer's system context.
Reflection
Generative-agents pattern (Park et al. 2023): every N sessions write a synthetic LLM-authored summary, index it alongside raw turns.
RAPTOR
Hierarchical k-means clustering with LLM cluster summaries (Sarthi et al. 2024). 4 levels deep, branch=4. Like a table of contents for the chat.
Karpathy LLM Wiki
Karpathy's LLM-maintained wiki: every session triggers an LLM call that creates or appends entity pages with [[wikilinks]] and [session=...] citations. Periodic lint pass merges duplicates and resolves contradictions.
HippoRAG 2
HippoRAG 2 (Gutierrez et al., ICML 2025): hippocampus-inspired memory. Haiku open-IE extracts (subject, predicate, object) triples per session into a networkx graph. Synonym edges link near-duplicate entities. At recall, query embeddings seed personalized PageRank over the graph; passages are scored by PPR mass on their entities.
A-MEM
A-MEM (NeurIPS 2025, Xu et al.): the LLM emits structured memory notes (content + keywords + context + tags) per session, embeds them, and a periodic link-evolution pass adds directed edges between related notes. Retrieval fans out from each top-k hit to its linked neighbors.
QISS (Quantum-Inspired)
Quantum-Inspired Semantic Similarity: reranks naive_vector's candidates by quantum-state fidelity Tr(rho_q rho_d) = cosine squared over the same embeddings. Pure NumPy, no new deps. An optional multi-query superposition mode adds interference cross-terms that classical rank fusion cannot express.
SQR (SWAP Test)
Simulated Quantum Reranker: runs a real SWAP-test circuit on the Qiskit Aer simulator to estimate each query-document overlap. Embeddings are PCA-reduced to 2^n_qubits dims, amplitude-encoded, and compared in exact statevector mode. The accuracy cost of that dimensionality reduction is reported honestly.
Mem0
Mem0 SDK (v2), leveled to the harness: Chroma vector store, OpenAI embeddings, Anthropic Sonnet for fact extraction during ingest (same model as the pure-Python baselines).
Graphiti
Zep's Graphiti: temporal knowledge graph with valid_at/invalid_at edges. Each session ingests as one episode. Searches return time-aware facts.
Graphiti (FalkorDB)
Zep's Graphiti running on FalkorDB (a Redis-based, GraphBLAS graph engine) instead of Neo4j. Identical ingest and recall to graphiti; the graph database is the only variable, so the latency and cost deltas isolate the engine and put FalkorDB's headline latency claims to an independent test.
Cognee
Open-source knowledge-graph memory. Cognify extracts entities/relationships into a graph; search answers via the graph as context.
LangMem
LangChain's memory store: extracts facts as conversations happen, stores them in LangGraph's InMemoryStore, recalls by similarity.
Memori
SQL-native agent memory. Stores extracted facts in Postgres for SQL-style recall. Memori 3.x with autocommit BYODB conn.
The 5 question categories
Information Extraction
Single-fact recall from one session. Example: 'What programming language did the user say they prefer?'
Multi-Session Reasoning
Combine facts from 2+ sessions to answer. Example: 'Given the user's job and city, what is their commute like?'
Temporal
Time-aware queries that require knowing when something was said. Example: 'When did the user first mention mountain biking?'
Knowledge Update
User changed their mind across sessions. Test: does the system reflect the latest version, not an earlier one?
Abstention
Question has no answer in the chat history. Test: does the system correctly say 'I do not have that information'?