That a separate freshness-metadata table was already rejected, and that a repeated lookup inside one request doubles latency, is not derivable from the code: it stayed in a chat that's closed.

The agent proposes the rejected option — you explain it again, or you don't notice and step on the same trap twice.

/engmem add response caching for a new endpoint

The sub-agent gets only the task description: no store, no search, no mention that engmem exists. If nothing can be spawned, the agent writes those 2–3 lines itself, but strictly before the first search.

Add a cache in front of the new endpoint, with
its own freshness-metadata table — easier to
control invalidation that way.

pre-reg source: sub-agent
baseline_tokens: 1840

The plan already assumes the table that was rejected last time.

• This is pre-registration: a plan written after the documents are read is no longer evidence — memory rewrites itself to fit what it now knows.
• The human never types it, because this step's payoff goes to a review a month later, and steps like that are the first ones abandoned.
• baseline_tokens is the sub-agent's spend on the naive plan, not a measured cost of the no-memory path.

~/Developer/engmem/sessions/20260820-cache-new-endpoint.md

---
id: 20260820-cache-new-endpoint
title: Caching for a new endpoint
date: 2026-08-20
task_date: 2026-08-20
status: draft
superseded_by:
backfilled: false
tags: []
entities: []
related: []
covers_files: []
verified_at_commit:
capture_minutes:
---

## Pre-reg

Add a cache in front of the new endpoint, with
its own freshness-metadata table — easier to
control invalidation that way.

• status: draft keeps the document out of search: unfinished work must not pass itself off as knowledge.
• backfilled: false means the document was written alongside the work. Older tasks can be entered into the store after the fact — those carry backfilled: true, are searched like any other, and are left out of the measurement, since the ritual was never run on them.
• The spine is written in full not for the parser (it derives what's missing on its own, see T2) but because these are exactly the fields search ranks on.

engmem search "ResponseCacheController ETAG revalidation"

The agent assembles the keywords itself — from your phrasing and the class names it sees nearby in the code.

### 1000001-response-cache (score: 20.8)
path: /Users/…/sessions/1000001-response-cache.md
matched: entities=responsecachecontroller,response,cache,controller,etag; §4-cold-start-primer=responsecachecontroller,response,cache,controller,etag
  §4-cold-start-primer   0.2 KB   ResponseCacheController serves cached responses for read endpoints using the ETAG (entity tag) and TTL (time to live) values already…
  §2-decision-log   0.3 KB   Reused the ETAG and TTL headers the upstream service already sends instead of adding a separate cache-metadata table; rejected the…
  §3-landmines   0.1 KB   ResponseCache lookups must be memoized per-request — a repeated call inside the same request re-runs the upstream fetch and doubles latency.
ResponseCacheController serves cached responses for read endpoints using the ETAG (entity tag) and TTL (time to live) values already present on the upstream response.

### ttl-etag-revalidation-v2 (score: 16.5)
path: /Users/…/sessions/ttl-etag-revalidation-v2.md
matched: entities=cache,controller,etag,revalidation; §4-cold-start-primer=response,cache,controller,etag,revalidation
  §4-cold-start-primer   0.2 KB   CacheRevalidationController v2 resolves a response's TTL (time to live) and ETAG (entity tag) together in one pass, reusing the freshness…
  §2-decision-log   0.2 KB   Kept revalidation inside CacheRevalidationController rather than extracting a separate RevalidationResolver; rejected the extraction…
  §1-pre-reg   0.1 KB   Rework revalidation so TTL and ETAG are carried through to v2 of the cache API.
CacheRevalidationController v2 resolves a response's TTL (time to live) and ETAG (entity tag) together in one pass, reusing the freshness metadata from the response-cache work.
related: 1000001-response-cache (/Users/…/sessions/1000001-response-cache.md)
docs: 7 (1 failed to load) | drafts: 1 | last doc: 19d ago

• The gap between 20.8 and 16.5 separates a direct hit from a tangential one, and matched shows by what: five words, not one.
• The § lines are section addresses: the agent reads §2-decision-log, three hundred bytes instead of twenty kilobytes.
• The footer is the state of the memory: 7 documents, 1 draft from step 02, the newest written 19 days ago.

The 4 KB cap is a condition of measurement: the channel's ceiling is fixed, so its cost can be written to telemetry and compared against a baseline. The same three documents are shown however many matched — thirty matches yield the same three blocks plus a single "N more" line. And (1 failed to load) goes to stdout rather than stderr: the agent reads only stdout, so seven documents instead of eight would look to it like a complete answer.

Up to three paths from the results are opened section by section, not whole, plus their related at depth exactly 1; the agent never follows the chain further.

At minimum the primer and the Decision Log of each document; these three usually settle the matter:

## 8. Decision Log
Reused the ETAG and TTL headers the upstream service already
sends instead of adding a separate cache-metadata table;
rejected the separate table because the response already
carries the freshness data and an extra lookup would add
latency for no real benefit.

## 11. Lessons Learned
ResponseCache lookups must be memoized per-request — a
repeated call inside the same request re-runs the upstream
fetch and doubles latency.

## 13. Future LLM Context (cold-start primer)
ResponseCacheController serves cached responses for read
endpoints using the ETAG and TTL values…

The --role flag returns one section instead of a document (lessons, decisions, primer; there are exactly nineteen roles, as many as there are mandatory headings, and engmem roles prints the list and the store's coverage). Together with top 3 and link depth 1, these are limits with one purpose: so the context fills with the task rather than with memory.

Your documents' contents are physically transmitted to the model on every task start: the tool never touches the network, but the agent does, and for a work environment that's a security sign-off question, not something settled by construction.

Pulled from memory:
· 1000001-response-cache — a separate metadata table was
  already rejected: ETAG and TTL arrive on the response.
· ttl-etag-revalidation-v2 — TTL must resolve before ETAG,
  or the tag is silently dropped.

Your plan assumed a new table — I suggest using the headers
that already arrive instead.

If nothing was found: prior context: none found, and the work carries on without inventions.

The diverging plans are a secondary signal: a plan written blind diverges from the final one for many reasons, and reading the repository itself is one of them. The primary evidence is the quoted rows of the Reuse Log at step 07.

Code, tests, discussion; the tool does nothing, and the draft waits, staying hidden from other searches.

You type not one field: you confirm the header and answer whether this work overturns a decision in some prior document. A "yes" marks that document superseded — the memory prunes itself.

tags: [platform, caching]
entities: [EndpointConfig, ResponseCacheController, ETAG]
related: [1000001-response-cache, ttl-etag-revalidation-v2]
covers_files: [EndpointConfig.java, ResponseCacheController.java]
author: <from git config user.name>
repos: [platform-core]
branch: feat/response-cache
pr:                        # no PR opened yet — the field stays blank

Confirm? (y / edits)

Those last four take the header from thirteen fields to seventeen — pure provenance: search never reads them, so a blank pr: is harmless and an invented number is not.

The Reuse Log is one row per document that actually influenced the work; without a direct quote the row is invalid, and if nothing helped it writes exactly Prior docs used: none.

| 1000001-response-cache
| "rejected the separate table because the response
|  already carries the freshness data"
| dropped my own table, reused the upstream headers
| reuse

## Pre-reg                     ← carried over verbatim
## 1. Executive Summary
## 2. Business Context
## 3. End-to-End Business Flow
     … 4–15: requirements, criteria, architecture, code,
       decisions, production, testing, lessons, entity
       graph, primer, keywords, cheat sheet
## 16. Reuse Log
## 17. Status at a Glance      ← appended as 17, not inserted
## Search Trace                ← how this session's retrieval went

• No section is optional: one with nothing to report gets one honest line saying so.
• The numbering 1–16 is inherited from a knowledge-base format the author kept separately — so both feed one searchable corpus.
• Number 17 was appended rather than inserted where it belongs by meaning: other documents already cite "section 13 — the primer".

A full save asks you to check the YAML, check the quotes, and recall the store for the overturn question; /engmem.save.quick removes all three demands and leaves a single ok.

The price is real: the document comes out shorter — five sections, none numbered — the primer is never written, so there will be no preview in future results, and the overturn question goes unanswered. That makes it the emergency path.

status: draft  →  active
capture_minutes:  →  9        (draft creation to save)
verified_at_commit: → 4f1c2a8   (HEAD at save time)
entities: []  →  [EndpointConfig, ResponseCacheController, ETAG]

docs: 7 (1 failed to load) | drafts: 0 | last doc: 0d ago

verified_at_commit answers the future question "which code was this true against"; capture_minutes accumulates the ritual's price.

Four optional fields answer whether memory paid for itself in context:

baseline_tokens: 1840   # the sub-agent's spend on the naive plan
context_bytes:   3120   # document text pulled into context
answer_steps:       2   # tool calls AFTER the search
answer_tokens:   9600   # only when the runtime really reports it

Cheap is a couple of kilobytes recovered and almost no further trips to the code; expensive is tens of kilobytes pulled in and the same number of tool calls you'd have made anyway. A field is omitted when the number isn't there: a fabricated figure corrupts the one analysis this whole thing exists for.

This does not measure the cost of the same task without memory: the naive plan is a plan, not an execution, and only running one task twice gives a real side-by-side number.

A month from now, the query EndpointConfig ETAG finds it through the entities field, and the table isn't proposed a third time.

telemetry.jsonl accumulates search records and the documents accumulate quoted reuse logs; after 3–4 weeks those are reviewed.

["search", "ResponseCacheController ETAG revalidation"]

resolve_store() takes the first of three sources, in strict order: --store, ENGMEM_HOME, ~/Developer/engmem. The tilde is expanded by the code itself (Path.expanduser()): quoted, it reaches argv literally.

sessions_dir.is_dir() false — fail(...) and exit 2, without ever reaching load_store:

error: store not found: /Users/…/sessions does not exist
(run `engmem install` to create the store, or check --store/ENGMEM_HOME)

fail() prints that line to both streams because the agent reads only stdout: a failure visible on stderr alone is indistinguishable to it from a silent empty run, and a missing store used to mean "nothing found".

An empty field from SPINE_FIELDS (id, title, date, task_date, status, tags, entities) is derived: id from the filename stem, title from the first <h1>, date from a "- Date: …" line in the first 40 lines of the body or the file's mtime, status from the active default. Nothing is raised outward — each parse_document(path) is wrapped in try/except (yaml.YAMLError, ValueError, OSError):

LoadResult(docs=7, errors=1, warnings=1)
errors[0]   = "broken.md: while parsing a flow sequence…"
warnings[0] = "20260820-cache-new-endpoint.md: entities is empty"

Then _cmd_search calls stray_documents(store) and warns on stderr about markdown outside sessions/*.md: neither the store root nor subfolders are scanned.

What excludes a file from the result set: broken YAML, a header that isn't a mapping, an unclosed ---, a date that won't parse, a file that can't be read, a field of the wrong type (but not the reverse: _coerce_list_field unfolds a bare string into a one-element list with a warning — a slip, not corruption), and a newline inside id, which in a result header can forge a second ### block.

An incomplete document can still be ranked honestly; a corrupted one can't even be displayed without putting the output itself at risk.

_build_query_tokens() splits the query into CamelCase fragments and adds an initials acronym; short tokens (≤3 characters) and numeric ones compare only against unsplit field tokens, or ET would find ETAG and 482 would find 1000001. Then, for every token, _spine_score() picks the field with the highest weight where it matched, and _body_scores_from_entries() adds the BM25 of the best matching section:

score = spine + body
spine = Σ max(weight of the field where the token matched) × (matched / total)
        weights: id 5 · entities 3 · title 2 · tags 1
body  = BM25 of the best section (k1 1.2, b 0.6)

spine = (3+3+3+3+3) × 5/7 = 15 × 0.714 = 10.71
body  = 10.13                    →  score 20.8

The coverage multiplier punishes partial hits — it's what drops the second document's spine to 6.86: four tokens out of seven (16.5 in total — the body supplied the rest).

• draft — dropped silently during the filtering pass, though it still counts toward the scoreboard.
• superseded — redirected: the active document superseded_by points at lands in superseded_notes, not in hits.
• active — the only status that reaches hits.
• A homonym (MQ as both MessageQueue and MetricsQuery) yields one representative per meaning, flagged ambiguous, rather than a silent pick.

For the first TOP_N = 3 hits, a block is built: header, path, a matched line, up to three section addresses, up to two lines of the primer (capped at 240 characters), up to three links. Then the superseded redirects — but only those whose score beats the weakest of the shown three.

1. body = the assembled blocks
2. withheld_line = "(N more document(s) matched below the top 3 — …)"
   — computed and given its own byte budget BEFORE trimming
3. _trim_to_bytes(body, limit − withheld_line's budget)
   — cuts body on a line boundary, appends "[output trimmed to fit 4 KB]"
4. result = trimmed body + withheld_line

_trim_to_bytes() cuts from the end. The withheld-count line used to be appended to the same list of blocks: appended last, it was cut first — in exactly the case it exists to cover.

Nothing matched — render_no_match(), exactly one line: prior context: none found. The scoreboard prints on both branches, because the state of the memory is part of the answer, including when the answer is empty:

docs: 7 (1 failed to load) | drafts: 1 | last doc: 19d ago

128 bytes are subtracted from the budget up front for it: render_search_results is capped at 4096 − 128. The cap was doubled when section locators joined the output.

Nothing: log_search() catches OSError itself and returns either None or a reason string, and note: telemetry not recorded (…) prints after the results, which are not rolled back.

Two checks before anything else: --local together with claude-desktop or copilot-cli is rejected (their files always live under $HOME), and a directory-scoped mode with no .git in the current folder is rejected with exit 2. Then _ensure_store() creates sessions/ and runs git init, but only if the store isn't already a repository.

The directories are on the diagram above; what matters here is each mode's trap.

• claude — engmem.start.md installs as engmem.md, so the command is /engmem, not /engmem.start; then the <!-- engmem-trigger-rule --> sentinel and the rule line below it are appended.
• copilot-ide — the same templates, but with a .prompt.md extension: the Copilot plugin ignores a plain .md.
• copilot-cli — dots are illegal in skill names, so /engmem.save is rewritten to /engmem-save — longer pattern first, or /engmem.save.quick would decay into /engmem-save.quick.
• claude-desktop — the config is not overwritten if it doesn't parse or if mcpServers isn't an object; one key is added and the rest of the file stays as it was.

A silent no-op is worse than an error: a command that misses its directory litters someone else's repository unnoticed. A re-run is safe all the same — the rule isn't duplicated, because detection goes by the broad `engmem search marker, which is fit only for the "skip" decision, never for "what to delete".

Both of T5's checks are repeated, and _agent_targets() computes the targets: template files plus an instructions file — or None for copilot-cli and claude-desktop, which have no shared file carrying the rule.

• copilot-cli — the skill folder is pruned only if it's empty after SKILL.md is deleted.
• claude-desktop — exactly the mcpServers.engmem key is removed; every other server is untouched.
• claude / copilot-ide — a file that can't be removed lands in the report rather than aborting the loop.

_remove_trigger_rule() keys off the sentinel and the rule line right after it, not the broad marker install used to decide whether to skip writing: deleting on the broad marker once destroyed a user's own line that simply happened to mention engmem search in its text.

A false conclusion that engmem was never installed — against which _other_agents_with_files_present(), on a zero count, checks the three other modes and suggests a different --agent. The store is never touched:

engmem uninstalled: 3 template file(s) removed, 2 trigger rule line(s) removed, agent=claude
store left untouched: /Users/…/Developer/engmem  (your documents — remove it yourself if you want)

_cmd_mcp resolves the store with the same resolve_store() path 1 uses, lazily imports mcp_server, and hands off to serve(store).

It reads stdin line by line: a blank line is skipped; on JSONDecodeError it writes _error_response(None, PARSE_ERROR, …) immediately and keeps reading, without ever reaching _dispatch.

_dispatch() tells a request from a notification by whether id is present: a notification gets no reply at all, even on failure — that is JSON-RPC itself. The method is looked up in the _METHODS table: the six handlers on the diagram plus ping. Exceptions are caught at two levels: _ProtocolError becomes a response carrying that error's code, a bare Exception becomes INTERNAL_ERROR with a diagnostic on stderr.

_write() is the only writer to stdout, and a line like error: … in the middle of a stream of frames would tear the next one and end the client's session. The same technique is correct on path 1 and catastrophic here — because the two paths have different readers of stdout.

The chain on the diagram is exactly T2–T4, and the same functions produce byte-identical text. Two things differ: load_store's errors and warnings go to the real process stderr, never into the response text, and strays are found by the very same stray_documents() — one function imported from spine.py, not a mirrored copy.

• An empty arguments.query is a _ProtocolError(INVALID_PARAMS), not a result with isError: true: inputSchema declares query required, so the call violates the contract instead of failing at runtime.
• A missing sessions/, by contrast, is an ordinary — just unsuccessful — call result: (error text, is_error=True) inside the normal response content, because here there is no other channel.

A search over MCP writes the same telemetry line a terminal search does, tagged channel="mcp". Without it the Desktop path would be invisible when the experiment is reviewed — and that is where most of the work now happens.