Memory Tape v0

Interactive architecture

How the pieces connect

Pick an operation to trace its path. Pick a version to see what switches on as we add small pieces of training and, later, the offline librarian. Click any box for the chosen component and why.

on the selected pathpart of this versionnot in this version yet

Chosen components · click a card to find it in the diagram

Chosen components

Protocol · two operations

append and query

One HTTP endpoint per operation, plus the same two as MCP tools so agents can call them directly. Requests and answers are JSON. Like Jev, a query carries named questions and gets answers keyed by those names; each question may carry its own time.

append

Send a whole chat or any part of it. Each event's id is unique per thread, so resending a chat adds only the new messages.
t_event is when it was said; the memory stamps t_ingest when it arrives. Nothing is ever edited or deleted.
kind covers everything else: message, document, retract (hides the events it names), note (offline summaries and links, citing their sources).
The reply comes after the tape write (milliseconds). Embedding follows in a background batch, so new text is searchable by keyword at once and by meaning about a second later.

{
  "op": "append",
  "memory": "team-alpha",
  "source": "agent:planner",
  "thread": "launch-q4",
  "events": [
    { "id": "msg-8812", "role": "user",
      "t_event": "2026-10-15T09:12:00Z",
      "text": "Launch moves to 14 November. Bob takes over the checklist." },
    { "id": "fix-0031", "kind": "retract", "refs": ["msg-7710"],
      "text": "Ad budget figure was entered by mistake." }
  ]
}

{ "ok": true, "added": 2, "skipped": 0,
  "seq": [1849220, 1849221], "chunks": 1, "ms": 4 }

query

Named questions in plain text. Each takes an optional time: as_of answers with what the memory knew then; since / until limit when things happened.
mode: relevant (best matches, each dated), latest (among matches that say the same thing, the newest first, older ones attached), timeline (matches in time order; with since_seq it streams raw events, which is how another memory syncs).
Answers come back keyed by question name: exact text, times, source and a score. With the reranker on, the score is the probability that the item answers the question.
A budget caps time and tokens out; the trimmer keeps the sentences that best match.

{
  "op": "query",
  "memory": "team-alpha",
  "filters": { "thread": "launch-q4" },
  "questions": {
    "launch_date": { "text": "When is the Q4 launch?", "mode": "latest" },
    "owner_then":  { "text": "Who owns the launch checklist?",
                     "as_of": "2026-10-01T00:00:00Z" },
    "risks":       { "text": "known launch risks", "since": "2026-10-01", "k": 5 }
  },
  "budget": { "max_ms": 100, "max_tokens_out": 800 }
}

{
  "launch_date": { "items": [ {
      "text": "Launch moves to 14 November. Bob takes over the checklist.",
      "t_event": "2026-10-15T09:12:00Z", "source": "agent:planner",
      "seq": 1849220, "score": 0.83,
      "older": [ { "seq": 1712004, "t_event": "2026-09-12T10:00:00Z",
                   "text": "Q4 launch planned for 7 November." } ] } ] },
  "owner_then": { "items": [ { "text": "Alice owns the launch checklist.",
      "t_event": "2026-09-20T15:30:00Z", "score": 0.79 } ] },
  "risks": { "items": [] },
  "usage": { "ms": 38, "candidates": 200, "tokens_out": 96 }
}

Time, explained · example thread

What a query sees at a given time

Five events in one thread. Move the "as of" date to ask the memory what it knew then, and switch the mode. Nothing on the tape changes; only the view does.

As of

Mode

In v0, "says the same thing" means the matches' vectors are very close (and both mention the same subject). The offline librarian later replaces this guess with explicit "replaced by" notes.

Compression and speed

How small, how fast

Compression comes in rungs. Each one is optional, none is quantization, and the tape always keeps the exact text, so every rung can be redone later.

full vector1,536 B

768 numbers × 2 bytes per chunk, as the model outputs them.

v0 · cut to 256512 B

Keep the first 256 numbers (the model is trained for this). About one byte per token of text at 160-token chunks.

v0 option · cut to 128256 B

Smaller again, at a larger retrieval cost; worth it only if memory size binds.

v1 · learned 64128 B

A small linear map trained on our data to keep the ranking, applied to stored vectors without re-embedding.

sent per answer≈ 2–4 KB

The trimmer returns only matching sentences: the compression that matters to the reading LLM.

Memory size	1M tokens	10M tokens	100M tokens
Chunks (about 160 tokens each)estimate	6.3k	63k	625k
Tape text (4.2 bytes per token)measured	4.2 MB	42 MB	420 MB
Vectors at 256 numbers, 16-bitestimate	3.2 MB	32 MB	320 MB
Keyword index (roughly the text size)estimate	≈ 4 MB	≈ 40 MB	≈ 400 MB
Exact vector search, one question, 8 CPU threadsmeasured at 100M, scaled below	< 1 ms	≈ 6 ms	61 ms
Exact vector search on one GPUmeasured at 100M	< 0.1 ms	< 0.1 ms	0.45 ms
Embedding one questionmeasured	59 ms on 8 CPU threads · 30 ms on one GPU
Whole query, CPU, no rerankersum of the above	≈ 60 ms	≈ 65 ms	≈ 120 ms: over budget
Bulk ingest, one RTX 3090measured at 1M, scaled	14 s	≈ 2.4 min	≈ 24 min
Bulk ingest, 8 CPU threadsmeasured rate, scaled	≈ 11 min	≈ 1.9 h	≈ 18 h
Our page writer, for comparison, one RTX 3090measured	0.44 s	≈ 4 s	≈ 44 s

Three numbers to remember, measured on Polonez on 17 October with the same 1M-token document for both models. First, EmbeddingGemma 2 ingests 33× slower than our page writer on a GPU (14.3 s against 0.44 s) and 14× slower on 8 CPU threads. Second, at 100M tokens a CPU-only query takes about 120 ms (59 ms to embed the question, 61 ms of exact search), so it misses the 100 ms budget; below about 60M tokens it fits, and above that the fix is a graph index at full precision (still no quantization) or a GPU. Third, at 256 numbers per 160-token chunk the vectors are about the size of the text; our own latent notes are about ten times the text, so off-the-shelf chunk vectors are the more compact index.

Plan · each version is usable on its own

Build order

No training

Tape, chunker, EmbeddingGemma cut to 256, exact vector search, keyword search, rank fusion, time filters.
HTTP service with append and query; the same two as MCP tools.
Exams: our world documents from 4k to 1M tokens with time questions, and LongMemEval (about 1.5M tokens of chat).

Done when: a query answers in under 100 ms on CPU at 10M tokens, and far lookups in our world documents pass.

Small training

A query adapter: one small linear map on question vectors, trained on question–evidence pairs. Stored vectors never change, so nothing is re-indexed.
Optional: a learned 256 → 64 map for 4× smaller storage, trained to keep the ranking.
Optional reranker on GPU, giving each answer a probability.

Done when: each piece beats v0 on the same exams; otherwise it is dropped.

Offline librarian

A scheduled job reads the tape since its last run and appends notes: summaries, links between facts, "replaced by" markers.
Starts as a prompted LLM; later our trained models (the E33 reader, the E36 librarian) take over.
It uses the same append operation, so the protocol never grows.

Done when: "latest" and two-step questions improve with no change to the query path.

What this gives up compared with the full design: multi-step lookups happen in the calling LLM, which simply asks again; "latest" is a similarity guess until the librarian writes its markers; and the embedding model ingests 33× slower than our own page writer on a GPU (measured: 14.3 s against 0.44 s for the same 1M-token document). None of these blocks a first working service, and each is a later slot for our research models.