# Usage Guide: terse

## What This Does

terse makes an AI agent's tool output smaller without throwing anything away. When
a tool returns a big block of JSON (a list of search results, a directory listing,
an API response), a lot of that text is repetition — the same field names on every
row, the same values over and over, extra spacing. terse rewrites that block into a
denser form the model can still read directly, so it takes fewer tokens while keeping
every piece of information.

It only compresses where it helps. Some tools already return tidy output; terse
leaves those alone rather than doing pointless work. You tell it which tools to
compress (and how hard) with a small policy file.

The one rule terse never breaks: the compressed output can always be turned back
into the exact original. If it ever couldn't, that is treated as a failure, not a
trade-off.

## How to Use It

Everything runs through the `terse` command. Set up once with `uv sync`.

### Check whether a payload is worth compressing

Pipe a tool's JSON output into `terse gate`. It tells you three things: whether the
compression is perfectly reversible, what *shape* the data is, and how many tokens
were saved.

```
cat some-output.json | uv run terse gate -
```

You'll see something like `round-trip lossless: PASS` and `cl100k tokens: 1810 -> 881
(51.3% saved)`. A PASS means nothing was lost. A small or zero saving just means that
payload didn't have much repetition to remove — that's normal and honest.

### Compress a payload through your policy

```
cat some-output.json | uv run terse compress --tool gh.api.repos --policy policy.example.json -
```

The compressed text comes out; a short summary goes to the side (which tiers ran and
the percent saved). `--tool` is the name terse matches against your policy to decide
how to treat that tool. If you leave off `--policy`, terse uses a safe lossless
default for everything.

### Run a tool server "behind" terse (automatic compression)

Instead of compressing outputs by hand, you can put terse in front of a whole tool
server so it compresses that server's results automatically, as they come back:

```
uv run terse proxy --policy policy.example.json -- <the command that starts the server>
```

Everything still works exactly as before from the outside — terse just quietly shrinks
the results the policy says to shrink, and passes everything else through untouched. If
anything ever goes wrong with a compression, terse sends the original result instead, so
a tool call is never lost. Add `--debug` to see what it compressed.

**One proxy, one downstream, either transport.** By default each `terse proxy` wraps one
downstream MCP server — either **stdio** (a command that talks newline-delimited JSON-RPC
over stdin/stdout) or **HTTP/SSE** (MCP Streamable HTTP, configured by a `url`). Point the
same `proxy` command at a URL instead of a command to proxy a remote server:

```
uv run terse proxy --policy policy.example.json -- https://example.com/mcp

# add auth/other headers (repeatable); secret-shaped values are redacted wherever
# terse prints a command line (e.g. install-mcp --print), never in the request itself:
uv run terse proxy --policy policy.example.json \
  --header 'Authorization=Bearer sk-...' -- https://example.com/mcp
```

`install-mcp` (below) wraps a `url`-configured `mcpServers` entry the same way
automatically. For several servers, either give each its own wrapper (one `terse proxy --
<server cmd>` entry per server — `install-mcp` does this by default) or front all of them
from one proxy process with `--config` (see "Fronting multiple servers from one proxy"
below).

**Tell terse which server it's fronting.** If your policy has rules scoped to a server —
`runecho.*`, `codegraph.*` — add `--server-name` so they actually match:

```
uv run terse proxy --policy policy.json --server-name runecho -- runecho-mcp
```

The catch this solves: MCP tool names don't inherently name their server. kb calls its
tools `kb.read.search`, so a `kb.*` rule matches unaided — but runecho calls its tool
plain `structure`, so a `runecho.*` rule matched *nothing* and quietly fell through to
your defaults. The rule looked authored and did nothing. `--server-name` supplies the
missing half so a server-scoped rule means the same thing for every server; it also
labels `terse stats` with the real server rather than the launch command's basename.
`install-mcp` (below) fills it in for you from your MCP config, and `--config` uses each
peer's `name` — so you only pass this by hand when running `proxy` directly.

### Fronting multiple servers from one proxy (`--config`)

`--config` fans one `terse proxy` process out to N downstream peers — any mix of stdio and
HTTP — behind a single policy/primer/process instead of one wrapper per server:

```json
// peers.json
{ "downstreams": [
    { "name": "gh", "policy": "gh-policy.json", "command": ["uvx", "github-mcp"] },
    { "name": "kb", "policy": "kb-policy.json", "url": "https://kb.example/mcp",
      "headers": { "Authorization": "Bearer sk-..." } }
] }
```

```
uv run terse proxy --config peers.json
```

Each peer's tools are advertised **under their own names**. A name is qualified with the
peer's `name` (`gh__search`, `kb__search`) only when two or more peers export the same
one, and terse strips that prefix before forwarding. A tool whose name no other peer
claims keeps that name.

Note this changes the TOOL segment only. A Claude Code permission entry is
`mcp__<server>__<tool>`, and consolidating N servers behind one proxy rewrites the
*server* segment either way — `mcp__kb__kb.read.search` becomes
`mcp__terse__kb.read.search`. Collision-only naming removes one of the two mismatching
segments, not both, so plan on revisiting the allowlist when you adopt `--config`; it is
a smaller edit than before, not no edit.

**Check your fleet before assuming it is a drop-in.** Collisions are not rare in
practice: a real 9-server fleet measured 6 collisions across 49 distinct tool names —
three language-server peers exporting an identical set (`definition`, `diagnostics`,
`hover`, `references`, `rename_symbol`, `edit_file`). Every one of those 18 tool instances
is qualified, so allowlist entries naming them must be updated. Servers that namespace
their own tools (`kb.read.search`) never collide; servers with generic verb names usually
do when you run more than one of a kind.

Two caveats, both needing a peer to miss a `tools/list` broadcast entirely — the same
measured fleet answers in 0.6-4.8s against a 30s broadcast timeout, so this means a hung
server, not a slow one. A tool is named from the listing it appeared in, so if peers that
*do* collide are reduced to one answering peer, that copy is advertised bare that listing
and qualified the next. And the routing table is exactly the most recent listing — a peer
that missed it is not advertised, and calling one of its tools from a stale client-side
list returns -32601 until the next listing. See terse#178.

The synthetic `terse.retrieve` tool (drop-to-retrieve) is advertised once, shared across
every peer, regardless of which peer dropped the field; it is a reserved name, so a peer
exporting `terse.retrieve` is qualified rather than shadowing it.
This is an ergonomics convenience — MCP clients can already talk to several servers
directly — so keep expectations proportionate: broadcast requests (`initialize`,
`tools/list`) wait on every peer up to a bounded timeout before merging what arrived, and
methods outside `initialize`/`tools/list`/`tools/call` fall back to peer 0 for now (a
debug-logged, documented v1 limitation, not a silent gap).

### Fold your whole fleet into one proxy (`install-mcp --multiproxy`)

```
# preview — config diff AND the permission entries you must update:
uv run terse install-mcp kb runecho codegraph --policy policy.json --multiproxy --print

uv run terse install-mcp kb runecho codegraph --policy policy.json --multiproxy
uv run terse uninstall-mcp --all          # restores all three originals
```

Why bother: one policy, one process, and one permission surface instead of N.

The original reason was stronger and no longer applies, so it is worth being precise
about. Before #211 a standalone `terse proxy` injected its format primer into every
`initialize` reply and the client re-read it **every turn**, so cost scaled with
(servers x turns) while savings scaled with (compressible calls) — measured at six
servers, +23.1% raw input one-proxy-each. Consolidating was how a fleet stopped losing.
The lazy primer (#211) removed that tax at the source: a standalone entry now attaches
its primer once, to the first result carrying a terse wire form, and an idle server pays
nothing at all. The same 1/3/6-server sweep re-run against it is flat — **-3.8% / -3.5%
/ -3.5%** weighted — where it used to degrade from -9.2% to +17.4%.

So consolidate for the operational reasons above, not to rescue a scaling loss. The
router keeps one shared `union_primer`, which is a boolean OR over five fixed sections
rather than a concatenation of N — bounded at 555 cl100k tokens whether it fronts 1 peer
or 20 — but it still rides `initialize`, so unlike a standalone entry it is paid from the
first turn whether or not any peer is ever called. `terse stats` reports the two cadences
separately for exactly this reason.

This rewrites permission entries — `mcp__kb__kb.read.search` becomes
`mcp__terse__kb.read.search` — so run `--print` first and update your allowlist before
restarting the client.

**It also widens them.** A whole-server grant (`mcp__kb`) used to reach exactly one
server; after folding, the equivalent `mcp__terse` reaches *every* peer behind the
router. If that matters, use the per-tool form (`mcp__terse__kb.read.search`) instead
of the server-wide one. `--print` names both forms per server.

Re-running is **additive**: `install-mcp c --multiproxy` after `install-mcp a b
--multiproxy` leaves a fleet of three, not one. The peers file records the fleet and is
namespaced per scope, so a local-scope router and a user-scope router (both living
beside the same `~/.claude.json`) never overwrite each other.

Every runtime flag a single-server wrap accepts — `--capture-dir`, `--diff`/`--no-diff`,
`--no-stats`, `--no-join-blocks`, `--never-lossy` — works with `--multiproxy` too and is
baked onto the router entry, which applies it to all peers. A peer's `env` and `cwd` are
carried into the peers file and re-applied when the router launches it, so a server that
needed a pinned `PATH` or an API key in `env` keeps getting one.

`terse mcp-status` reports a folded fleet as one `router` row (with `wraps=` listing its
peers) plus a `folded` row per peer naming the router it sits behind — not as drift. Four
more states exist for when something has gone wrong, each with a line saying what to do:
`folded-unstashed` (stash entry gone — `uninstall-mcp --all` rebuilds it from the peers
file, launch fields only), `folded-and-live` (in the peers file *and* live, so the
downstream runs twice), `router-ambiguous` (two entries front one peers file — delete the
duplicate), and an `unreadable peers file` note naming the path.

`uninstall-mcp --all` is the escape hatch from every one of those, including a deleted or
corrupt peers file. It removes the router once nothing is left for it to front.

> **Project scope: the peers file carries secrets.** `--scope project` writes
> `.terse-peers-project-<hash>.json` next to the checked-in `.mcp.json`, and it contains
> each peer's `env` block — API keys included. It is created `0600`, but that does not
> stop a `git add -A`. **Add `.terse-peers-*.json` to `.gitignore`** before folding a
> fleet in project scope.

### Wire terse into Claude Code automatically (`install-mcp`)

Rather than editing `~/.claude.json` by hand, let terse wrap your MCP servers for you:

```
# preview the change (writes nothing):
uv run terse install-mcp runecho --policy policy.example.json --print

# apply it (backs up ~/.claude.json first), then restart Claude Code:
uv run terse install-mcp runecho codegraph --policy policy.example.json

# undo — restore the original command(s) exactly:
uv run terse uninstall-mcp runecho        # one server
uv run terse uninstall-mcp --all          # every terse-managed server
```

`install-mcp` rewrites each named `mcpServers` entry so its command becomes
`<terse> proxy --policy <policy> -- <the original command>` — see *Which terse a
wrapped entry launches* below for what `<terse>` resolves to — preserving
the entry's `env`/`cwd`/etc. The original is saved verbatim in a sidecar stash
(`.terse-mcp-stash.json` next to the config), so `uninstall-mcp` restores it
byte-for-byte. It's **idempotent** (re-running re-wraps from the stashed original
instead of nesting proxies). Cross-call diffing is OPT-IN since #170, so a plain
wrap does NOT diff; pass `install-mcp --diff` to bake an opt-in into a server's
entry (or `--no-diff` to override a policy-file `"diff": true`), and
`--diff-keyframe-interval K` to tune re-anchoring. Flags always reflect the latest
install invocation. It honors `$CLAUDE_CONFIG` if your config isn't at
`~/.claude.json`. Start with one high-win, read-only server (e.g. `runecho`) and
confirm it works before wrapping more.

If the capture dir you wire already holds payloads, the install ends by pointing at the
command that turns them back into policy:

```
→ corpus at /home/you/corpus already holds 1,674 payload(s) — `terse policy autotune`
  re-tunes /home/you/.config/terse/policy.json against them.
```

On a *first* install it stays quiet: the corpus is empty by definition, and per-tool shape
decisions need payloads, not a `tools/list`.

Hand-edits to a wrapped entry (say, an `env.PATH` pin) survive re-installs: the
re-wrap rebuilds `command`/`args` from the stashed original but keeps every other
live key, and reports what it carried (`kept hand-edited key(s) …`). One edge
remains: `uninstall-mcp` restores the **pre-terse original**, which does not carry
such edits — put them on the original entry too if they must survive an uninstall.
This matters most when the edit is what makes the *downstream* server work at all
(a Node-version `env.PATH` pin for a server that crashes on the system's default
runtime, say): kept on the wrapped copy alone, it vanishes the moment you uninstall.

#### Which terse a wrapped entry launches (`$TERSE_MCP_CMD`)

`install-mcp` picks a launcher at install time, in this order. Everything it picks
by itself is an **absolute path**, because an MCP client does not necessarily start
your shell's environment and a bare `terse` would resolve unpredictably or not at all;
only `$TERSE_MCP_CMD` can override that, and only deliberately.

1. `$TERSE_MCP_CMD`, if set (see below).
2. The `terse` console script on your `PATH`, as long as it is **not inside the
   virtualenv running `install-mcp`**, sits in an absolute `PATH` entry, and answers
   `--version` as terse — normally `~/.local/bin/terse`, the script both
   `uv tool install terse-mcp` and `pipx install terse-mcp` provide. A candidate that
   fails any of those (a `pyenv`/`asdf` shim that resolves at launch time, an unrelated
   tool of the same name) is skipped rather than baked in, and tier 3 is used instead.
   `install-mcp` always prints the `launcher:` it settled on, so check that line if you
   expected tier 2 and did not get it.
3. Failing that, `<absolute interpreter> -m terse` — the interpreter running
   `install-mcp`. This is the right answer only when terse is installed nowhere but
   that venv, i.e. a source checkout.

Tier 2 exists because a wrapped entry has to keep working for weeks, and tier 3's
interpreter path is only as stable as the venv behind it. Run `install-mcp` with `uv
run` from a throwaway git worktree and tier 3 would capture *that worktree's*
`.venv/bin/python3`; the config keeps working until the worktree is removed, and then
every wrapped server at once fails **silently** — the servers just show up with no
tools, days after the install-mcp run that caused it. An isolated-tool install
(`uv tool`, `pipx`) has a milder version of the same exposure, since its interpreter
lives in a versioned venv that an upgrade or a distribution rename can move.

Note the consequence of tier 2: running `install-mcp` from a terse *checkout* wires
your MCP servers to your **installed** terse, not the checkout. That is what you want
almost always. To wire a development build in on purpose, name it explicitly:

```
TERSE_MCP_CMD="$PWD/.venv/bin/terse" uv run terse install-mcp runecho \
  --policy policy.example.json --print     # confirm the command, then drop --print
```

`install-mcp` also warns when the launcher it picked reports a **different version**
than the terse writing the entries — a checkout emits flags in its own grammar, and a
launcher older than those flags exits 2 the moment your client spawns it, which the
client shows only as a server with no tools. A dirty working tree is not counted as a
difference (hatch-vcs stamps it with a build date), so the warning stays meaningful.

`$TERSE_MCP_CMD` is whitespace-split and always wins. A leading `~` is expanded for
you (a wrapped entry is spawned without a shell, so a literal tilde would never
resolve), and a path that does not exist is **rejected at install time** rather than
written into the config — the same treatment `--policy` already gets. A bare name like
`terse` is passed through untouched, since it resolves against the launcher's `PATH`,
which the installer cannot know.

Whichever tier supplied it, `terse mcp-status` flags an entry whose launcher stopped
resolving, so it is worth a look after any terse upgrade:

```
  runecho              wrapped  policy=/home/you/.config/terse/policy.json
                       wraps=runecho-mcp  diff=default  stats=on
                       launcher=/home/you/.local/share/uv/tools/terse-mcp/bin/python (MISSING) — this entry cannot start; re-run install-mcp
```

Then confirm each server completes an `initialize` + `tools/list` handshake once the
client restarts. A broken wrapper does not announce itself to the client — it only
ever looks like a server that has gone quiet.

Claude Code has three MCP scopes, and `--scope` targets any of them (default
`user`, i.e. today's behavior):

```
# project scope — a .mcp.json checked into the repo and shared with every clone
uv run terse install-mcp runecho --policy policy.example.json --scope project
uv run terse uninstall-mcp runecho --scope project             # --file overrides the path (default ./.mcp.json)

# local scope — personal to one repo on one machine, nested in ~/.claude.json's
# projects."<repo-path>" block; --repo-path defaults to `git rev-parse
# --git-common-dir` (the bare-repo root for a claudew/codexw worktree, so every
# worktree of the same repo shares one entry instead of one per worktree)
uv run terse install-mcp runecho --policy policy.example.json --scope local
uv run terse uninstall-mcp runecho --scope local
```

The sidecar stash is namespaced per scope, so the same server can be
independently managed in more than one scope at once (e.g. wrapped with a
stricter policy at `user` scope and a looser one at `local` scope for one repo)
without the two colliding.

With three scopes, "what's wrapped where" is no longer one file to eyeball —
`mcp-status` checks all three and prints one report, read-only (it never writes
anything):

```
uv run terse mcp-status
# [user] /home/you/.claude.json
#   runecho              wrapped  policy=/home/you/.config/terse/policy.json
#                        wraps=runecho-mcp  diff=default  stats=on
#   codegraph            wrapped  policy=/home/you/.config/terse/gone.json (MISSING)
#                        wraps=codegraph serve --mcp  diff=off  stats=on
#   some-other-server    unwrapped
uv run terse mcp-status --json   # the same rows as JSON, for scripts / CI checks
```

Each server is one of `wrapped` (terse-managed, present), `unwrapped` (present,
not terse's), or `orphaned-stash` (terse has a stash entry but the `mcpServers`
entry it should match is gone — usually a sign the config was hand-edited after
wrapping; `uninstall-mcp --all` won't touch it either since there's nothing to
restore it *into*). For a `wrapped` server a second line shows **what it actually
fronts** (`wraps=…`), whether the cross-call **diff** tier is on/off/default, and
whether the **stats** ledger is on — the things you need when a wrapped server
misbehaves and the bare `wrapped` line can't tell you why. A policy file that has
gone missing since install is flagged `(MISSING)` (the proxy would fail to launch
without it); a *relative* policy path is never flagged, since it resolves against
the launcher's cwd, which a status scan can't know. `--file`/`--repo-path` override
project/local scope the same way `install-mcp` does.

### Let terse write the policy for you (`policy generate`)

Authoring `policy.json` by hand is the main chore: add a server, see no compression, then
capture → measure → read the report → edit JSON. `policy generate` does that loop for you
from a corpus of captured outputs (collect one with `proxy --capture-dir` or `install-mcp
--capture-dir`):

```bash
# print a policy to stdout (per-tool decisions go to stderr):
uv run terse policy generate --corpus ~/.config/terse/session-corpus

# or write it straight to a file:
uv run terse policy generate --corpus <dir> --out policy.json
```

It is **conservative and lossless**: for each tool it measures the real per-tier token
savings and enables a tier only where the saving clears `--threshold` (default 5%) *and*
every payload round-trips exactly — otherwise that tool is left as passthrough. The
dictionary tier is added only where its *marginal* saving clears the bar too (so a tool
that tabularizes well but has no repeated values won't carry the dictionary cost). Each
rule is commented with the measured savings, and the summary prints highest-win tools
first:

```
# terse policy generate — 6 tool(s), threshold 5.0%
  gh.api.items        minify,tabularize,dictionary  40.4% saved (dictionary +30.2%)
  kb.read.search      minify,tabularize             31.5% saved (dictionary +0.0% below threshold — dropped)
  status.rate_limit   (passthrough)                 1.2% < 5.0% threshold
```

It also **suggests** (never enables) `drop-to-retrieve` candidates: fields that are large
*and* near-unique, where the lossless tiers are powerless (nothing repeats to fold) but the
field dominates the payload — the classic case being an `embedding`/vector field. These ride
along as an **inactive** `_suggested_fields` block and print under the tool:

```
  kb.read.list_nodes  (passthrough)                 2.6% < 5.0% threshold
      ↳ drop-candidate result[].embedding (~84% of tokens, 100% unique, ~2293 tok/value) — suggested, off by default
```

Because drop is lossy, you opt in by renaming `_suggested_fields` → `fields` in the output —
then confirm the model still answers with `terse fluency`. Until you do, it is a no-op (the
loader ignores `_suggested_fields`). On the measured corpus, enabling the `embedding` drop
turns a +2.6% tool into a +77% one.

terse measures *tokens*, not comprehension — before relying on a generated policy, confirm
the model still reads the compressed form with `terse fluency --corpus <dir>` (see below).

#### Rule names carry the server

A generated rule is named the way the proxy will **look it up**, which is `{server}.{tool}`
whenever the corpus recorded a server — so a corpus captured from a `--server-name runecho`
wrap authors `runecho.structure`, not `structure`.

That is not cosmetic. `Policy.select` tries the qualified name against *every* rule before
it tries the bare one against any, so with a `runecho.*` rule deployed, a bare `structure`
rule never matches — no matter where in the file it sits. Authoring the qualified name is
the only thing that reaches it.

The corollary is that a corpus which records **no** server can only produce bare rules. If
`policy generate` or `autotune` prints:

```
  [note] 1674/1674 payload(s) record no server, so their rules are authored under the bare
         tool name — which a deployed server-scoped rule (`runecho.*`) shadows at runtime
         whatever its position. Re-capture, or wrap with `proxy --server-name`, to author
         reachable rules.
```

…the rules in that diff for a server-scoped tool are inert. Fix it at the source: make sure
every wrap passes `--server-name` (`install-mcp` does), then capture again. For a
hand-captured payload, `terse capture --server <name>` records it directly.

A second note reports how many payloads predate `result_id` and therefore had their
*results* reconstructed from capture timing rather than read. Neither field can be recovered
after the fact, which is why both are reported rather than quietly assumed away — see
[TECHNICAL.md](TECHNICAL.md) for what the timing heuristic can get wrong.

### Re-tune a deployed policy (`policy autotune`)

`policy generate` authors from nothing and **overwrites**. Run it on a policy that is
already governing the wire and you silently lose every decision the corpus cannot see.
`policy autotune` is the re-run: it *merges*, prints a diff, and writes nothing without
`--apply`.

```bash
# the common case — --policy and --corpus resolve from what install-mcp wired:
terse policy autotune

# ...then, once you have read the diff:
terse policy autotune --apply
```

Who owns what, on a re-run:

| key | decided by | on autotune |
|---|---|---|
| `tiers` | the corpus | proposed, **including removals** |
| `_suggested_fields` | the corpus | regenerated (stays inactive) |
| `capture`, `structured`, `never_lossy_servers` | you | preserved verbatim |
| active `fields` | you | preserved verbatim |
| rules for tools absent from the corpus | you | preserved, in position |

A corpus cannot see that a tool returns a plaintext credential, or that your client
validates `outputSchema`. Those three keys are fail-safe everywhere else in terse, and
regeneration is not allowed to be the one path that reverses them.

Path resolution is fail-safe too: defaults are used only when the wrapped servers agree on
exactly one policy (and one capture dir). A disagreement lists the paths and refuses rather
than guessing which policy governs the wire.

#### It reads the savings ledger too

Two things the corpus structurally cannot show, so autotune cross-checks `terse stats`'
ledger (`--ledger FILE`; defaults to the one `terse stats` reads, and is skipped in silence
if you have none):

```
  not tunable from this corpus — live in the ledger, no captured payloads (~/.local/state/terse/stats.jsonl):
  ! kb.read.search                 122,431 tok at   0.0% saved over 84 block(s)
    (capture is off for these, or they were never captured — autotune cannot tune what it cannot see)
```

* **Coverage.** A `"capture": false` tool appears *only* in the ledger. Without this
  section, a diff covering everything terse could see reads as "the install is tuned now"
  while a fifth of your tokens were never an input at all.
* **Frequency.** The corpus is idempotent by sha — it holds each payload's first sighting,
  not every call — so a tool called twice and one called 800 times weigh the same in it. A
  proposed tier *removal* now prints the live block count beside the rule, because giving
  back measured savings on a hot tool is a different decision from doing it on a cold one.

Fix a blind spot by capturing the tool, not by pointing autotune at something else: if
`capture: false` is there on purpose, the blind spot is the correct outcome.

### Reaching a long-text tool (`$text.code_blocks`)

Some tools don't return JSON at all — they return markdown whose bulk is verbatim source
(`codegraph_explore` is the canonical case). The lossless tiers have nothing to fold in
prose, so those payloads pass through at **0% saved** no matter how you set `tiers`.

A `$`-sigil field path selects *spans* of the raw text instead of fields of an object:

```json
{
  "match": { "tool": "codegraph.*" },
  "tiers": ["minify", "tabularize", "dictionary"],
  "fields": {
    "$text.code_blocks": { "lossy": "drop-to-retrieve", "min": 400 }
  }
}
```

Every fenced code block of at least `min` characters leaves the wire as a `terse.retrieve`
handle; the exploration summary, blast radius, and file headings stay resident, so the model
keeps the *intelligence* and fetches the *source* only if it actually needs it. On 60 real
captured `codegraph_explore` payloads this took a 0.0%-saved tool to **87.0%**.

This is **lossy and opt-in** — the same bar as any drop:

- Everything is recoverable, and the gate proves it: terse restores the entire payload from
  what it emitted plus the session store and requires byte-for-byte equality with the
  original, or it emits the untouched text and warns.
- It is suppressed — with a warning, never silently — on a `--never-lossy` server, on a
  selector marked `{"critical": true}`, and on a rule with `"tiers": []`.
- A misspelled selector (`$text.codeblocks`) warns rather than quietly compressing nothing.
- Confirm the behavior, not just the token count, with `terse fluency --drop-eval` (below):
  the question that matters for this shape is whether the model *retrieves* the source
  instead of answering from the surrounding prose.

**That behavioral question is now measured, not assumed.** 30 captured `codegraph_explore`
payloads, 5 models (see caveat below) over an OpenAI-compatible gateway, table dated
2026-07-23:

| model | called retrieve when needed | asked for the right block | left it alone when not needed | answered correctly |
|---|--:|--:|--:|--:|
| `claude-sonnet-5` | 100% | 100% | 100% | 97% |
| `deepseek-v4-pro` | 100% | 100% | 100% | 100% |
| `qwen3.7-max` | 100% | 100% | 87% | 100% |
| `glm-5.2` | 100% | 100% | 100% | 83% |
| `minimax-m3` | 100% | 100% | 100% | 97% |

No model answered from the retained prose as if it had read the source, and (bar one
qwen over-fetch) none fetched a block it did not need. The "answered correctly" column is
a comprehension check on the *retrieved* content and is deliberately un-guessable — it is
lower than the retrieve columns because the model has to read the fetched source, not
because it failed to fetch it. Reproduce with
`terse fluency --drop-eval --corpus <dir> --policy <file> --base-url ... --models ...`.

**Caveat added 2026-08-19 (#249), corrected same day after review:** on this project's own
eval gateway, `claude-sonnet-5` is a confirmed DeepSeek alias, not real Anthropic — the
same routing this table's `deepseek-v4-pro` row already names honestly (`answerers.py` and
the 0.26.0 changelog entry both state this outright, not as a hedge). The two rows' numbers
are *not* identical (97% vs. 100% on the last column) — do **not** read that as evidence
they're different backends: #249's own stress-panel data shows two confirmed-identical
DeepSeek aliases producing non-identical numbers on the same corpus (a documented
run-to-run noise floor, #295), so a numeric mismatch here is the *expected* outcome for a
duplicate, not counter-evidence against one. The genuinely open question is dating, not
identity: this table predates #249's alias discovery by nearly a month (git blame:
2026-07-23), and the gateway's alias config isn't version-controlled, so whether the
aliasing was already in effect when this table was measured can't be checked from this
repo. Treat "5 models" as unconfirmed on that basis, most likely 4.

**State the downside too.** 87% is a *gross* number: a retrieve returns the span in full,
so the conversation then holds both the stub and the original. Best case, the model never
retrieves and you keep the full saving; worst case it retrieves every block and you are
~13% *worse* off than not compressing, plus the extra round-trips. The measured retrieve
behavior above is what makes that asymmetry a good bet — not the token count.

**It does not generalize to all long text.** The selector reaches markdown *scaffolding
around* code, not raw source. On the third-party `filesystem/read_text_file` corpus —
long-text, 34k tokens — the only fences are docstring examples and it saves **1%**. That is
why `policy generate` proposes the selector only when droppable blocks are ≥40% of a tool's
tokens across ≥2 payloads; a tool like `read_text_file` is correctly left alone.

### One-command lossy tuning (`terse tune`)

`terse tune` chains the whole lossy-adoption loop into one command: it runs the generator,
then presents drop-to-retrieve candidates **safe-first**, classified by field role:

```bash
terse tune --corpus corpus/ --out policy.json
# # terse tune — 40 payload(s), 6 tool(s), 3 drop candidate(s)
# SAFE candidates — supporting prose, enable after a dropeval pass:
#   kb.read.nodes    result[].description   ~41% tok, 100% uniq  [prose]
#   → enabling all 1 here: ≈12,400 tok, ~18% of corpus (gross, before the per-record retrieve-handle cost)
# REVIEW candidates — role unknown, may be LOAD-BEARING; verify carefully:
#   kb.read.list_principles  result[].principle  ~36% tok, 100% uniq  [unknown]
#   codegraph_explore        $text.code_blocks   ~89% tok, 100% uniq  [unknown]
#   → enabling all 1 here: ≈9,800 tok, ~14% of corpus (gross, before the per-record retrieve-handle cost)
```

Text tools are included in that scan. A tool whose payloads never parse as JSON has no
fields to profile, so it used to produce no candidates at all — which is exactly how a
0.0%-saved, 89%-droppable tool stayed invisible to the generator. `$text.code_blocks` is
proposed when droppable fenced blocks are >=40% of the tool's tokens over >=2 payloads, and
always as `[unknown]`: a fenced block is source, and source is load-bearing until a
`--drop-eval` pass says otherwise.

Each bucket ends with a **rollup** — the estimated gross tokens dropping that whole
bucket would evict (`mean field tokens × record count`, summed) and its share of the
corpus's raw tokens. That's the number that answers "is turning on the SAFE set worth
it?" — a per-field `~% tok` alone can't, since it's relative to each tool's own record
list. It's a gross estimate: each dropped field leaves a small `terse.retrieve` handle
behind, so the realized saving is slightly lower.

- **`[prose]`** (evidence, rationale, description, notes, body…) — supporting text, the safe
  drop candidate.
- **`[unknown]`** — the name doesn't reveal the role, so it *may be load-bearing* (dropping it
  forces the model to call `terse.retrieve` for it). A field the model reasons over, like a
  `principle` or a `verdict`, lands here on purpose — a name heuristic can't know it's the
  essence of the record.
- **`[identity]`** fields (id, name, key, path, title…) are never suggested — the record
  needs them in-line.

Verify before enabling — the token win is real, but only a live model can tell you whether a
dropped field was actually needed:

```bash
# runs the real 2-turn retrieve eval on the suggested drops and prints the verdict:
terse tune --corpus corpus/ --out policy.json --drop-eval \
  --base-url $URL --models glm-5.2,deepseek-v4-flash
```

If the worst-case model **PASSES**, enable a field by renaming that tool's
`_suggested_fields` → `fields`. Start with `[prose]`; leave any `[unknown]` that fails.
The dropeval gate — not the role guess — is the real safety net.

### Forbidding lossy on a credential/personal server (`--never-lossy`)

Lossy transforms are **structurally forbidden** on a *never-lossy* server (a credential or
personal store), even if a policy marks one of its fields lossy — a policy typo can't leak a
credential payload through a truncate/drop. Two layers:

- A **built-in name floor** always forbids lossy on servers whose name looks secret-shaped
  (`secret`/`credential`/`vault`/`token`/`password`/`key`/`auth`) — non-overridable.
- For a sensitive store the floor can't catch (a personal KB, a launcher alias), **declare it
  at install** so it's baked into the policy:

```bash
terse install-mcp kb --policy policy.json --never-lossy
# never-lossy: baked kb into the policy's never_lossy_servers — lossy is now forbidden on it
```

`install-mcp` also prints a *hint* when a server looks sensitive but wasn't marked. The
enforcement keys off the server's verified identity (the `--server-name` terse bakes into the
wrap), so it can't be defeated by a mislabeled rule.

### Reaching a payload the server double-encoded (`"embedded"`)

Some servers return their whole body as a **JSON string** rather than as JSON:

```json
{ "response_text": "{\"results\": [{\"id\": 1, ...}, ...]}" }
```

`minify`, `tabularize` and `dictionary` all walk *parsed* structure, and a string is a leaf —
so the record array terse is best at is invisible to every tier. Measured on identical data:
**41.9% saved as a real array, 0.0% inside a string.** This is not one tool's quirk; #143
measured ~21.6% of one fleet's tokens sitting at 0.0% from a single return convention.

The `embedded` tier reaches it — add it to a rule's `tiers`:

```json
{ "match": { "tool": "secret-broker.*" },
  "tiers": ["minify", "tabularize", "dictionary", "embedded"] }
```

A folded string ships as `{"__terse_json__":1,"f":F,"v":<the document>}`, and the other tiers
then apply *inside* `v` normally — that is the point, the fold exists to let them in.

**It only fires when it can rebuild the original string byte-for-byte.** `f` names the exact
serialization that reproduces it. terse tries a fixed set of forms (`json.dumps` defaults,
minified, `indent=2`, `indent=4`, each with and without `ensure_ascii`) and **declines
whenever none of them match**, because `json.dumps(json.loads(s))` is not `s` in general:

| the string contains | why terse declines |
|---|---|
| `{"a": 1, "a": 2}` | duplicate keys — `json.loads` keeps only the last |
| `{"n": 1.50}` | re-encodes as `1.5`; the value survives, the spelling does not |
| `{"n": 1.5e0}` | re-encodes as `1.5` |
| unusual spacing, leading/trailing whitespace | no registered form emits it |
| a `__terse_*` key | would be misread as one of terse's own envelopes |

Declining is always safe; folding a string terse cannot reproduce exactly would not be. The
bar is byte-equality rather than "same data" on purpose — the guarantee terse sells is
byte-faithfulness, and a tier that quietly relaxed that to "equivalent JSON" would be
selling something else.

It is **opt-in**, like every tier that costs a primer paragraph: a form the client is told
about enlarges that server's primer (#168), so a server that never double-encodes should
not pay for the explanation. `policy generate` / `policy autotune` will add it where the
measurement justifies it.

### Compressing `structuredContent` (`"structured": "compress"`)

Some MCP servers return a typed `structuredContent` field beside a text block that
serializes the same data. terse compresses the text block — but **some clients read the
typed field and throw the text block away**, so on those tools terse can be doing work and
delivering nothing.

Measured against `claude` 2.1.218 (`scripts/probe/structured_content/`, a read-only proxy
watching what actually lands in the model's context) that is exactly what happens. Turning
this on puts the codec where the payload really is:

```json
{ "match": { "tool": "kb.*" },
  "tiers": ["minify", "tabularize", "dictionary"],
  "structured": "compress" }
```

On the reference fixture: **2,596 → 1,008 chars of the model's real context (61.2%)**,
against 0% with the default.

**You usually do not need to set this.** The default is `"auto"`, which decides per
connected client: the MCP handshake carries a `clientInfo` name the client *declares*, and
terse compresses the typed field only for clients measured not to validate it — currently
Claude Code. Every other client, a client that omits `clientInfo`, and a library caller
that never handshakes all keep the server's own object untouched. Set `"compress"` or
`"leave"` explicitly only to override that.

Why it is gated at all: the MCP spec says clients *should* validate `structuredContent`
against the tool's `outputSchema`. Leaving it alone merely makes terse a no-op; rewriting
it for a validating client would **break tools that worked**. The allowlist is measured,
not assumed — neither a wrong type nor a terse envelope in place of the declared record
array draws a complaint from the reference client.

Check whether it applies to you before bothering: a tool that declares an `outputSchema`
is the one that will emit the field. Among common servers, filesystem (14/14 tools),
memory (9/9) and kb (27/27) all do; git, fetch, serena and playwright do not.

#### `"structured": "replace"` — and why you almost certainly don't want it

`"replace"` compresses the typed field **and deletes the text block that mirrors it**.
Measured against the same client, it saves nothing:

| mode | what reaches the model |
|---|--:|
| default (`auto` → `leave` for an unlisted client) | 2,596 chars |
| `"compress"` | **1,008 chars** |
| `"replace"` | 1,008 chars — *no further change* |

Claude Code had already thrown the block away, so removing it takes ~2,596 chars off a
local stdio pipe and nothing off the model's context. It ships for the client that
forwards *both* fields — for which it roughly halves the result, and which under
`"compress"` can receive a cross-call diff in the block that contradicts the full envelope
in the typed field. No such client has been measured. If you are on Claude Code,
`"compress"` is the end of the line.

It is never what `"auto"` resolves to, and it drops nothing unless all five guards hold:
the mode is explicitly `"replace"`, the rule has non-empty `tiers`, the result is not an
error, there is exactly one text block, and that block's parsed JSON is **equal** to
`structuredContent`. A block that says anything the typed field doesn't is not a mirror
and is kept.

### Keeping a tool's output off disk (`"capture": false`)

Some tools return things that should never be written to a file — a credential, a token,
a private key. terse's two disk sinks (`--capture-dir`'s corpus and `--debug-log`'s
replay trace) both store **raw** payloads, so for those tools you want them off:

```json
{ "match": { "tool": "secret-broker.*" },
  "tiers": [],          // don't compress it (nothing to gain, no state kept)
  "capture": false }    // ...and never write it to disk
```

Two things worth knowing:

- **`"tiers": []` alone is not enough.** Passthrough stops compression, but the capture
  tee runs *before* the tiers and would still write the payload out. You need
  `"capture": false` to stop that.
- **You still get the measurement.** The savings ledger (`terse stats`) records sizes and
  decisions only — never content — so a gated tool is still counted, just never quoted.

Why put it in the policy rather than just leaving `--capture-dir` off that server? Because
the policy is durable and reviewable: it survives a re-wrap, it's visible next to
everything else you decided, and it's the only way to express the exclusion when one
proxy fronts several servers (`--config` has a single capture dir). A flag you have to
remember not to pass is one copy-pasted command away from writing secrets to disk.

Terse only ever sees what's already in a tool's result — i.e. what your model was going
to read anyway — so wrapping a server doesn't expose anything new to the *model*. What
`capture: false` controls is what **survives on disk** afterwards.

### When a result looks wrong: the replay log

If a compressed result ever looks misshapen, add `--debug-log FILE` to the proxy and it
appends one JSON line per intercepted result — the raw payload, the tier decision, and
exactly what terse emitted:

```bash
uv run terse proxy --debug-log /tmp/terse-audit.jsonl -- uvx some-mcp-server
```

Each line has `{tool, id, diff_mode, tiers, changed, blocks:[{raw, emitted}]}`. Even a
no-op (`changed:false`) is logged, so you can confirm terse left a suspect payload alone.
Replay any line through `terse compress --tool <tool>` on its `raw` to reproduce. Opt-in
and side-effect-only: a log-write failure never affects what the client receives.

If a sink is dead — `--capture-dir` pointing at a regular file, `--stats-log` at a
directory, a full disk — the proxy says so on stderr the **first** time each one fails,
without needing `--debug`, then goes quiet so it can't flood the hot path:

```
[terse-proxy] gh.api.items: capture skipped: [Errno 17] File exists: '/tmp/not-a-dir' (further occurrences silenced unless --debug)
```

The run itself is unaffected — that is the point of the sinks being fail-open — but an
empty corpus makes a later `terse measure --corpus` report a percentage over whatever
subset happened to land, so this one is worth reading.

Under `proxy --config` (multi-server mode) the bookkeeping is per **peer**, so a dead shared sink prints
one line per downstream that hit it — each naming its own peer-qualified tool — rather
than one for the whole run. That is deliberate: it tells you *which* downstreams were
actually affected, and the flood guard still holds within each peer.

Because those records embed the raw payload, a tool with `"capture": false` in the policy
(above) is skipped here too — one declaration covers both disk sinks.

### How much is terse actually saving me? (`terse stats`)

Every proxy keeps a **live savings ledger** by default — one small JSON line per tool
result recording sizes, token counts, and what terse did (`compressed`, `diff`,
`unchanged`, `passthrough`). It stores **no payload content whatsoever** (that's what
makes always-on safe — unlike the replay log above), it's bounded (rotated at 10 MB,
one prior generation kept), and a write failure can never affect a tool call.

```bash
uv run terse stats               # all recorded history
uv run terse stats --since 7d    # just the last week (30m / 24h / 7d / 1w forms)
uv run terse stats --recommend   # one wrap/don't-wrap verdict per installed entry
uv run terse stats --json        # the raw aggregate, for scripts
```

`--recommend` rolls the break-even table's two numeric columns into one word per installed
entry — `KEEP` (cleared its own primer this window), `TUNE` (arithmetically reachable, just
short of the bar), `UNWRAP` (no block volume ever clears it), `INSUFFICIENT` (the ledger
cannot answer yet) — printed beside the coverage ratio it came from. It **replaces** the
ledger tables rather than adding to them; drop the flag for the arithmetic behind the words.
`TUNE` is a reachability statement and nothing more: terse models no policy change here (the
ledger is payload-free by design, so it structurally cannot) — `terse policy autotune` is the
command that answers the what-if. The verdict is in `--json` either way, so a script never
needs both flags.

`--json` is a **contract**, not a debug dump: `tests/test_stats_json_contract.py` pins every
field name and type at seven shapes — the top level, `total`, each `tools[]` row, each value
of the `versions` object, the `primer_liability` blob, each of its `servers[]` rows, and each
of those rows' `contributors[]` — so a rename or a removal fails CI rather than a consumer's
script. Four things worth knowing before you parse it:

- **`versions` is an object, not an array**, keyed by the terse version string that wrote
  those records (`{"0.17.1": {"blocks": …}}`). Iterating it directly gives you version
  strings; iterate `.items()` or `.values()` for the rollups.


- **`null` is a real answer and never means zero.** `blocks: null` on a liability row means
  no ledger label was recoverable — *we never found the rows to ask* — which is a different
  claim from `blocks: 0`, "installed and never called". Same for `turns_covered`,
  `session_covered`, `saved_per_block` and `primer_tokens`.
- **`primer_liability` itself can be `null`**, when the install could not be sized (a
  malformed MCP config). The ledger half of the document is still complete and correct; the
  reason is on stderr.
- **The `verdict` is per installed entry, never per peer.** A multiproxy router pays ONE
  union primer for its whole fleet, so `primer_liability.servers[]` already pools its peers
  into a single row and the verdict is computed against that pool. The per-peer breakdown is
  in `contributors[]`, which carries evidence (blocks, tokens saved, rate) and deliberately
  no primer, break-even or verdict of its own — there is no honest per-peer primer to divide
  by, and charging each peer the full shared one would recommend unwrapping peers that are
  collectively paying for themselves.

The report shows total tokens saved, the decision mix (how often the cross-call diff
actually fired), and a per-server/per-tool breakdown — with a `diff%` column (that
tool's cross-call diff hit rate) so you can see *which* tools the diff tier is actually
paying off on. When tiktoken wasn't available at record time the per-tool columns fall
back to characters (`chr raw`/`chr out`), matching the header rather than showing zeros.
Real numbers from your real sessions, complementing the synthetic-corpus `measure` report. The ledger lives at
`$XDG_STATE_HOME/terse/stats.jsonl` (usually `~/.local/state/terse/stats.jsonl`);
redirect it with `proxy --stats-log FILE` or disable it with `proxy --no-stats`
(bake the opt-out into a wrapped entry with `install-mcp --no-stats`).

### See how well it does across many tools

If you've collected sample outputs (see "Building a sample set" below), these produce
markdown reports:

- `uv run terse measure` — how many tokens are saved, per tool and per data shape.
- `uv run terse probe` — whether there's more to gain from future features.
- `uv run terse validate` — confirms the savings hold across different token counters.
- `uv run terse fluency` — the one that matters for the proxy: does a model *read* the
  compressed form as accurately as raw JSON? (see below.)

Add `--html` to `measure` or `verify` for a charted companion report next to the
markdown (e.g. `reports/verify-report.html`) — inline SVG bar/stacked-bar charts,
zero JS, zero CDN, so it stays offline like everything else in terse. Every chart
has a `<details>` table-view fallback underneath it.

Add `--bars` to `measure` or `verify` for the same savings charts as unicode bars
printed straight to the terminal — no new file, color only when stdout is a tty
(honors `NO_COLOR`). `fluency` also has a `--bars` flag — see below.

Add `--json` to `verify` for a machine-readable aggregate on stdout instead of the
report — the lossless-gate verdict (`lossless_gate.ok`), cl100k savings totals, and
per-shape/coverage breakdown, from the same numbers as the markdown. Handy in CI:
`terse verify --corpus corpus --json | jq -e .lossless_gate.ok`. It writes no
file/HTML/bars (machine mode), mirroring `stats --json` / `mcp-status --json`.

Add `--history <file.jsonl>` to `measure` to track savings over time, not just one
snapshot — is the win improving, flat, or regressing as the corpus grows? Each run
appends one line (timestamp, payload count, lossless gate, token totals) to the file,
then prints a trend table plus a sparkline across every run recorded there so far:

```
uv run terse measure --corpus corpus --history reports/measure-history.jsonl
# ...
# ## Trend across runs
# | # | timestamp | label | payloads | lossless | raw tok | terse tok | saved % | Δ pts |
# |---|---|---|---|---|---|---|---|---|
# | 1 | 2026-07-01T09:00:00+00:00 | corpus | 12 | 12/12 | 4102 | 2380 | +42.0% | — |
# | 2 | 2026-07-02T09:00:00+00:00 | corpus | 14 | 14/14 | 4890 | 2650 | +45.8% | +3.8 |
#   ▁█   +42.0% -> +45.8%  (range +42.0% .. +45.8%)
```

The file is plain JSONL (one JSON object per line, append-only) — diffable, greppable,
safe to commit if you want savings tracked in git history. It's not currently wired
into `verify` — verify's no-`--corpus` fallback uses a synthetic, deterministic sample,
which would just log the same numbers every time; point `--history` at `measure`
runs over your own captured corpus for a trend that means something.

### Check that the model still understands the compressed output

Saving tokens is pointless if the model reads the compressed form worse than raw JSON.
`fluency` measures that directly: it asks a model deterministic questions (count a
field, look one up, list them all, take a max) over both forms and scores the answers
against known-correct values — no second model judging.

```
# 1. (optional) build a synthetic corpus that stresses the hardest cases
python scripts/gen_stress_corpus.py corpus-stress

# 2a. keyless: writes an eval pack you can drive by hand, then score
uv run terse fluency --corpus corpus-stress

# 2b. with models (one OpenAI-compatible endpoint, e.g. OpenRouter):
TERSE_FLUENCY_BASE_URL=https://openrouter.ai/api/v1 \
TERSE_FLUENCY_API_KEY=sk-... \
TERSE_FLUENCY_MODELS=google/gemini-2.5-flash,deepseek/deepseek-chat \
  uv run terse fluency --corpus corpus-stress

# 2c. tighten the verdict: repeat each question N times for a confidence interval
TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
  uv run terse fluency --corpus corpus-stress --trials 5
```

The report shows accuracy per model for raw vs compressed vs compressed-with-a-one-time
format note ("primer"), flags which transform (if any) costs comprehension, and gives a
PASS/FAIL gated on the *worst* model. A model that scores 0% on raw JSON is a setup
error (wrong model id, no key) and is excluded from the verdict, not counted as a
comprehension failure. `--trials N` repeats each question N times and reports each
accuracy with a `±` 95% bound, so the verdict is a tight bound rather than directional.

Add `--bars` for the same verdict as a terminal forest plot — a point + 95% CI track
per model (best terse-form vs raw, or diff-form vs full-terse under `--diff`) with a
pass/fail badge, printed straight to the terminal. For a charted HTML version of that
forest plot, add `--html` to any of the paired diff-family evals (`--diff`,
`--diff-soak`, `--text-diff-eval`) — it writes next to `--out` with a `.html` suffix,
same inline-SVG/no-JS/no-CDN form as `measure --html`.

### Does the model actually use `terse.retrieve`? (`fluency --drop-eval`)

A field marked `{"lossy":"drop-to-retrieve"}` is provably recoverable — but only *if* the
model calls the synthetic `terse.retrieve` tool when it needs that field. `--drop-eval`
measures the actual behavior with a live tool-calling model, not just the round-trip gate:

```
# needs a policy with a drop-to-retrieve field, and a tool-capable model:
TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
  uv run terse fluency --drop-eval --policy drop-policy.json --corpus corpus
```

The report scores retrieve-recall (did it call retrieve when the answer needed the dropped
field), no-overfetch (did it leave the tool alone when the answer didn't need it), and
final-answer accuracy — gated on the worst model. Run this before enabling
`drop-to-retrieve` in a policy you'll actually deploy.

### Cross-call diffing and its fluency check

The proxy emits a lossless **delta** against the prior same-tool result instead of
the full payload — big in agent loops that call the same tool repeatedly (~91% overlap).
It is stateful and **OFF by default** — opt in with `--diff` (or `"diff": true` in a policy).
Its validation program passed (pair fluency, nested-record coverage, drift soak; see
TECHNICAL.md), so the default is a *cost* decision, not a confidence one: the primer
paragraph explaining the diff format adds 190 cl100k tokens to that server's primer — the
largest single section — attached once per session to each wrapped server that emits
a terse form (#211), against a measured **0.38% hit rate** (7 diffs in 1,828 blocks over
13.3 days, banking 5,052 tokens). Those 5,052 tokens are erased by about 27 primer
attaches: roughly two session-server pairs a day across the whole 13.3-day window, which
an active fleet passes immediately. That is the standalone cadence; behind a router the
paragraph rides `initialize` and is re-read every turn, which only widens the gap. So
#170 flipped it off. The ~900–2,700x multiplier
this paragraph used to quote was the same comparison against the pre-#211 per-turn charge;
#211 shrank the cost side by the session turn count and left the verdict intact. Turn it
on for a workload that genuinely re-calls the same tool with the same arguments —
`terse stats` shows your own hit rate under `diff reasons`:

```
# nothing happens without opting in — a plain proxy does NOT diff. Opt IN per proxy:
uv run terse proxy --diff -- uvx some-mcp-server
# or per policy file: {"diff": true, ...}

# multi-block results (one record per content block) are joined into one record array
# before compressing — folds records across blocks AND makes the result diff-eligible
# (ON by default, #116). This changes N content blocks to 1. Opt OUT per proxy:
uv run terse proxy --no-join-blocks -- uvx some-mcp-server
# or per policy file: {"join_blocks": false, ...}; bake into a wrapped server with
# install-mcp --no-join-blocks

# re-run the fluency gate against your own consumer/model anytime.
# needs same-tool PAIRS in the corpus (capture a tool 2+ times) + a configured model:
TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
  uv run terse fluency --diff --corpus corpus
```

`fluency --diff` reports diff-form accuracy vs full-result accuracy on the same
questions and PASS/FAILs on the worst model — re-run it for a new/weaker consumer
before trusting the default. The diff is always lossless and only sent when smaller;
it falls back to the full compressed form whenever no diff applies or the prior
result isn't available.

### The depth dimension: `fluency --diff-soak`

`--diff` tests one hop (full result + one diff). In production a model reads up to
`--diff-keyframe-interval` (default 5) **consecutive** diffs off one full anchor
before the proxy re-anchors — so the question that gated the default-flip (and gates
raising the keyframe interval) is whether comprehension *drifts* with chain depth:

```
# real corpus runs, depths 1..5 (6 chain windows per depth, round-robin across tools):
TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
  uv run terse fluency --diff-soak --corpus corpus --trials 3
```

The report shows accuracy by depth (chain form vs full-terse control on the same
final-state questions) and PASS/FAILs both overall and at the deepest tested depth,
worst model. The mechanical half of the same soak — hundreds of chained hops
reconstructed exactly, keyframe cadence, reconnect resets — is pinned in
`tests/test_diff_soak.py` and runs in CI.

### Text diffing and its fluency check (`fluency --text-diff-eval`)

The diff above only reasons about JSON. Non-JSON tool output (file reads, source
excerpts, log tails) gets its own diff codec (Tier 0.7, `text_diff.py`) — same
lossless/on-by-default/falls-back-to-full contract, applied to unstructured text
instead of records. `--text-diff-eval` is its behavioral check, the text-payload
analogue of `--diff`:

```
# needs same-tool TEXT payload PAIRS in the corpus (capture a text-producing tool
# 2+ times) + a configured model:
TERSE_FLUENCY_BASE_URL=... TERSE_FLUENCY_API_KEY=... TERSE_FLUENCY_MODELS=... \
  uv run terse fluency --text-diff-eval --corpus corpus
```

It asks whether a model reconstructs the current text as accurately from (previous text
+ text-diff) as from the full current text, and PASS/FAILs on the worst model — re-run
it for a new consumer of text-heavy tools. There's no separate switch: the same
diff tier emits a text diff instead of a JSON diff whenever the payload isn't JSON
(and `--diff` turns both on together), so this eval is a risk-item check on the
opt-in path.

### Building a sample set

To measure your own tools, capture their outputs first:

```
your-tool | uv run terse capture --tool your.tool.name -
```

Add `--server <name>` if that tool comes from a named MCP server, so the rule generated for
it is authored under the same qualified name the proxy looks it up by (see *Rule names carry
the server* above). A payload captured this way records no result id — it has no result to
belong to — so it is scored as a single-block result, which is the honest reading.

Each capture is saved locally. **Only capture output you're comfortable storing** —
captured files can contain whatever the tool returned. Do not capture anything with
passwords, personal data, or private documents in it.

### Adjusting the policy

The policy file (`policy.example.json`) is a list of rules. Each rule says: for tools
whose name matches this pattern, run these compression tiers. Patterns use `*` as a
wildcard (`gh.*` matches every GitHub tool). The first matching rule wins, so put more
specific rules first. An empty tier list means "leave this tool's output alone."

A pattern is matched against the tool's own name — so `gh.*` works only because those
tools are *called* `gh.something`. If you're scoping a rule to a **server** whose tools
have plain names (runecho's is just `structure`), the proxy needs `--server-name` to
connect the two, or the rule silently never fires. `install-mcp` handles this
automatically; see "Tell terse which server it's fronting" above.

## What to Do When Something Breaks

- **"round-trip lossless: FAIL"** — Stop and report it. This should never happen; it
  means the compression and decompression disagree. Note the tool and input that
  triggered it. (The test suite checks this on every change, so a FAIL in normal use
  is a bug worth filing.)

- **"It saved 0%" or a tiny number** — Not a failure. That payload was already compact
  or had little repetition (single objects and already-tidy tools do this). The
  per-tool report will show which tools are worth compressing and which aren't.

- **A `[warn] field ... not implemented yet` message** — The policy asked for `summarize`
  on a field; that mode isn't built yet, so terse safely ignored it and kept everything.
  Nothing was lost.
- **A `[warn] lossy: truncated marked field(s) — output is NOT lossless` message** — Expected
  when a field is marked `{"lossy":"truncate"}`. terse capped that field on purpose and
  annotated the cut (`…⟨+N chars⟩`). Only fields you marked (never `{"critical":true}` ones)
  are affected; if the gate can't prove that, terse falls back to the lossless output and
  says so (`lossy step skipped`).
- **A `[warn] lossy: dropped marked field(s) to retrieve handle(s)` message** — Expected
  when a field is marked `{"lossy":"drop-to-retrieve"}` and terse is running as the proxy.
  terse replaced that field with a `__terse_dropped__` handle and stored the original; the
  model gets it back by calling the injected `terse.retrieve` tool. Outside the proxy (e.g.
  the one-shot CLI) there is no store, so the field is left lossless with a
  `needs the proxy store` warning instead.

- **"no payloads in corpus/"** — You ran a report before capturing any samples. Capture
  some tool outputs first (see "Building a sample set").

- **A token count shows "unavailable"** — The token counter's data file didn't load
  (usually no internet on first run). Reconnect and try again.

For anything else, see the [Technical Reference](TECHNICAL.md) or [README](README.md).

## FAQ

**Does terse delete any of my data?**
By default, no — the compressed output always reconstructs the exact original. Two
opt-in lossy modes exist for fields you explicitly mark in the policy: `truncate`
(caps a field's length) and `drop-to-retrieve` (replaces a field with a handle you
can fetch back via the injected `terse.retrieve` tool). Nothing is dropped unless you
mark it, and `{"critical":true}` fields are never touched by either mode.

**Why didn't my payload get smaller?**
Because there was nothing safe to remove. terse shrinks repetition (repeated field
names, repeated values, extra spacing). A single small object or an already-minimal
response has none of that, so it's left as-is.

**Will the model still understand the compressed output?**
Yes. The compressed form is still readable text — a table with a header, or values
with a small legend at the top — not an encoded blob. The model reads it in place;
it never has to "fetch" anything that was removed.

**Why does it compress some tools and not others?**
Because it was measured to only pay off on some. Compressing already-tidy output wastes
effort, so the policy turns it off there. You control this in the policy file.

**Do I need an API key?**
No. Everything runs locally. A key is only needed for the optional live fluency eval
(`fluency --diff`/`--drop-eval`), which calls any OpenAI-compatible endpoint you point
it at (broker pool or a loopback gateway). You never need it for normal use.

<!-- docvet:anchors
drop-to-retrieve -> src/terse/transforms.py
proxy --config -> src/terse/cli.py
truncate -> src/terse/transforms.py
-->

