# Technical Reference: terse

## Architecture

terse is a pure-Python library with a thin CLI. The core is a deterministic,
fully-lossless transform pipeline; everything else (measurement, probes, policy)
is built around it.

```
raw tool output (JSON text)
        │
        ▼
 json.loads ──► compress_structure  (Tier 0: tabularize, recursive nested-key fold;
        │              │            Tier 0.6 `embedded`: fold a string leaf that is
        │              │            itself a JSON document, byte-exact only)
        │              ▼
        │        dict_encode        (Tier 0.5: fold repeated string values -> legend)
        │              │
        │              ▼
        │          minify           (Tier 0: strip whitespace, serialize)
        ▼              │
   (policy decides ◄───┘
    which tiers run per tool)
        │
        ▼
   compressed JSON text  ──► decompress ──► byte-identical original
```

**Design decisions:**

- **Lossless gate is non-negotiable.** `transforms.roundtrip_ok(obj)` runs
  `decompress(compress(obj)) == obj`. Token availability changes *which* values get
  aliased (a performance choice) but never correctness. The test suite is a
  parametrized battery of this gate.
- **Representation transform by default, not offload.** The lossless tiers produce valid
  input the model reads directly (a table, an inline legend) with no `retrieve` step — the
  core difference from lossy-offload approaches. The one deliberate exception is the opt-in
  `drop-to-retrieve` lossy mode (#10): where you explicitly mark a field, terse evicts it to
  a handle and serves it back through a synthetic `terse.retrieve` tool. Off by default —
  lossless-first is the rule, retrieve is the marked exception.
- **Row-major tables.** Tabularization keeps rows as rows (positional cells mapped to
  a `cols` header), including nested `subcols`, so the table stays as legible as
  CSV/markdown — the model already does position→header mapping for the outer table.
- **Aliases are collision-proof.** Dictionary aliases come from a `~`-sigil namespace
  that is checked disjoint from every literal string (keys and values) in the
  payload, so decode is an exact legend lookup with no ambiguity.
- **Selective, fail-closed policy.** Value is per-tool, so a policy gates which tiers
  run. An unmatched tool gets the lossless default and never a lossy op.
- **Determinism.** No clock/random in the transform path; same input → same output.
- **Prior art on the primitive vs. the bundle.** Lossless tabularization of uniform
  JSON arrays is independently published elsewhere (e.g. [TOON](https://toonformat.dev/),
  MIT-licensed, ~40% reduction, its own benchmark). terse doesn't claim novelty on
  that transform in isolation. The differentiated surface is the combination: an
  MCP-transport-agnostic proxy (no client/server changes required), cross-call
  diffing, a `drop-to-retrieve` lossy escape hatch, and a behavioral fluency gate
  (`terse fluency`) that every diff/lossy tier had to pass before it shipped at all —
  none of the comparable formats or proxies bundle all of that.

### The guarantee ladder

"Lossless" is four separate claims, not one, and each is proved by a different artifact.
They are not interchangeable — a lower rung passing says nothing about the rung above it,
which is why every tier here ships with its own gate instead of one blanket assertion.
Each rung is already documented in full somewhere below; this is the index, so a reader
does not have to assemble the ladder from four documents.

```
  rung                     the claim                          proof artifact
  ────────────────────────────────────────────────────────────────────────────────────
  4. net economics         wrapping this server banks more    `terse stats` break-even
                           than its own primer costs
        ▲ not implied by 3
  3. task success          a model stays as accurate over a   `terse fluency --diff-soak`
                           CHAIN of diffs as over full ones
        ▲ not implied by 2
  2. model comprehension   a model READS the compressed       `terse fluency`
                           form as accurately as the original
        ▲ not implied by 1
  1. mechanical            `decompress(compress(x))` is `x`   `transforms.roundtrip_ok`
     reconstruction        again, by value
```

Read it upward. Rung 1 is a property of the bytes and is settled in CI for everyone; rung
4 is a property of *your* traffic and can only be settled by your own ledger. Nothing
above rung 1 is implied by rung 1 — that gap is the entire reason the behavioral evals
exist, and #170 is the case in point: the diff tier cleared rungs 1–3 and was still turned
off by default because it failed rung 4 at the measured hit rate.

- **1 — mechanical reconstruction: `transforms.roundtrip_ok`.** See *Lossless gate is
  non-negotiable* above and `transforms.py` under File Descriptions. Total and cheap, so
  it runs as a parametrized battery over the whole corpus. Note its exact claim — the
  gate is byte-faithful **by value** (`values_equal`), which is `==` plus a NaN contract,
  not string equality of the re-serialized text. Where a lossy mode replaces this gate it
  substitutes a *stronger* one for the affected span (`acceptable_loss`,
  `droppable_loss`, `text_droppable_loss` — see Known Limitations); no mode simply drops
  a rung.
- **2 — model comprehension: `terse fluency`.** See *Proxy: the model must understand
  terse's format* under Known Limitations (the eval lives in `src/terse/fluency/`; its
  renderers are `report.py` / `html_report.py` / `terminal_report.py` under File
  Descriptions). A round-trip gate proves the bytes reconstruct; it cannot prove a model
  reads the reconstruction as well as it read the original, which is a separate,
  live-model question.
- **3 — task success: `terse fluency --diff-soak`.** See *Cross-call diffing is built and
  OPT-IN* under Known Limitations. Single-hop comprehension does not imply comprehension
  at chain depth, so this measures accuracy as a function of how many diffs deep the
  chain is, against the keyframe bound; its mechanical twin is `tests/test_diff_soak.py`,
  which asserts exact reconstruction hundreds of hops deep. `fluency --drop-eval` is the
  same rung for the lossy path — a real 2-turn tool loop scored on whether the model
  actually reaches for `terse.retrieve`. Honest scope: these are the strongest shipped
  artifacts on this rung, not a general agentic-task benchmark.
- **4 — net economics: `terse stats` break-even.** See `stats.py` under File Descriptions
  (`_break_even`, `primer_liability`) and the break-even rule in `docs/POSITIONING.md`. A
  tier can be perfectly lossless, perfectly legible, and still lose tokens on your traffic
  — the primer is paid whether or not the tier ever fires. This is the only rung terse
  cannot settle for you.

## File Descriptions

- **`transforms.py`** — the lossless core.
  - `minify` / `compress_structure` (+ `_fold_records`) / `dict_encode` and their
    exact inverses (`decompress_structure` + `_unfold_row`, `dict_decode`).
  - `compress_with(obj, tabularize, dictionary)` applies a selectable subset of
    tiers; `compress`/`decompress`/`roundtrip_ok` are the full pipeline + gate.
  - Markers: `TABLE_MARKER`, `DICT_MARKER`, `ALIAS_SIGIL` (`~`).
  - Depends on `tokenize.count_cl100k` for the tokenizer-aware aliasing threshold.
- **`policy.py`** — `Rule`/`Policy` dataclasses, `load_policy` (JSON parse + validate),
  `default_policy`, `Policy.select` (first tool-glob match wins), and `apply()` which
  returns an `Applied` record (text, tiers run, skipped, warnings). The only module
  that knows about lossy field modes (which it parses and warns about, never executes).
- **`proxy.py`** — MCP stdio middleware. `Interceptor` is the pure message logic
  (records request `id → tool name`, compresses the matching `tools/call` result via
  `policy.apply`); it is transparent (non-result messages forwarded unchanged),
  fail-open (any error forwards the original), and frame-safe. `run_proxy` launches
  the downstream server as a subprocess and wires `Interceptor` to stdio with two
  pump threads.
- **`stats.py`** — the live savings ledger + `terse stats`. `build_record` /
  `classify_decision` produce one payload-FREE line per result (ts, version, server,
  tool, decision, char + cl100k token sizes — never content, which is what makes
  always-on safe where capture/audit must stay opt-in); the decision label is derived by
  sniffing the emitted text's envelope head (diff/textdiff marker, `== raw`,
  policy-tiers-empty), not by threading state out of the compression paths.
  `append_stats` rotates at 10 MB (one prior generation kept; single O_APPEND writes
  under PIPE_BUF stay atomic across concurrent proxies, and a torn tail from a
  crashed writer is self-healed with a newline before the next append).
  `load_stats`/`aggregate`/`build_stats_report` back the `stats` subcommand; token
  totals sum only fully-tokenized records (`untokenized` counts the rest — reported,
  never blended). `aggregate` also rolls up `versions` (per-writer-version blocks and
  token sums, so "did that release help?" is answerable without a payload-mix-confounded
  time slice) and `total.unversioned` (records predating the field — never bucketed
  under a `None` key), and canonicalizes tool names at READ time via `canonical_tool`,
  which strips a router's `<server>__` qualification when it merely repeats the record's
  own server label. Read-time, not write-time, so it also repairs records already on
  disk, where the same tool split into two rows depending on which topology handled it. The record's `server` is `proxy --server-name` when given (the config's
  own name — truthful, and what `install-mcp` bakes in), else `server_label`'s guess from
  the command basename, which misreads a launcher-wrapped server (kb behind secret-broker
  labels itself `sb-run`) — #83. Default path `$XDG_STATE_HOME/terse/stats.jsonl`; the
  proxy wires it via `build_stats_writer` (same never-load-bearing contract as
  capture/audit: the writer owns I/O only and lets failures out, and `Interceptor`
  swallows them — announcing the first of each kind unconditionally, further ones only
  under `--debug`, so a dead sink can't go silent *or* flood stderr; #131. The
  bookkeeping lives on the `Interceptor`, which multiproxy builds per peer, so there
  the guard is once per (peer, kind) — attribution over a single line). ON by
  default in `cli.py` (`--no-stats` / `--stats-log FILE`);
  the `run_proxy`/`run_multi_proxy` API default stays None (disabled) so library
  callers opt in explicitly.
- **`capture.py`** — `classify_shape` (pretty/compact JSON, array-of-records,
  long-text), `capture_payload` (writes a sha-idempotent envelope to `corpus/`),
  `load_corpus`, `coverage`, `extract_records`, `bare_and_server` / `qualify` /
  `qualified_tool`.
  The envelope records `server` and `result_id` when the proxy knows them, so a tune-time
  reader can tell *which downstream* sent a payload and *which result* its blocks arrived
  in, rather than inferring both. Both are optional and omitted when unknown — the format
  is additive, so a corpus captured before them stays loadable, and both are preserved
  together on an idempotent rewrite (a first sighting's timestamp beside a later
  sighting's result id would place the same block in two different calls at once).
  `qualify(bare, server)` reproduces `Policy._match_candidates`' first candidate and is
  the single definition of what a tool is called, shared by the corpus and the generator
  so they cannot drift. That name is not a preference: `select` tries the qualified
  candidate against *every* rule before the bare one against any, so a bare `structure`
  rule is dead behind a deployed `runecho.*` wherever in the file it sits (#152).
- **`measure.py`** — `measure_payload` (per-tier cl100k decomposition: `minify +
  tabularize + dictionary == tier_total`, re-runs the gate), `measure_corpus`, and
  `cross_tokenizer_savings` (cl100k vs o200k invariance).
- **`probes.py`** — `value_redundancy` and `cross_call_overlap`: upper-bound
  estimators for whether higher-ceiling levers (dictionary, cross-call diffing) are
  worth building. They measure, they do not compress.
- **`tokenize.py`** — `count(text, encoding)` over named tiktoken vocabs (cl100k,
  o200k) and `encode_cl100k` (token ids for probes).
- **`report.py`** — markdown renderers: `build_report` (savings by shape + per-tool +
  tier attribution + coverage + gate banner), `build_probe_report`,
  `build_tokenizer_report`.
- **`html_report.py`** — `build_html_report`'s charted HTML counterpart to
  `build_report`: inline-SVG diverging bars (savings), stacked bars (tier
  attribution), and a forest plot (`forest_plot`, per-model accuracy + 95% CI).
  `build_html_diff_report` wraps the forest plot into the comprehension-gap page
  behind `fluency --diff --html` (and `--diff-soak` / `--text-diff-eval`, whose
  paired results share the same shape). Pure stdlib string templates — no JS, no
  CDN, no new dependency — reuses `report.py`'s `_form_stats` / `_worst_case_gap`
  so the verdict never diverges from the markdown. Wired via `--html` on
  `measure`/`verify`/`fluency` (writes next to `--out`, same basename, `.html`
  suffix).
- **`terminal_report.py`** — zero-new-artifact bar-chart counterpart to `report.py`'s
  markdown. `build_terminal_report` covers `build_report`'s savings-by-shape /
  savings-by-tool / tier-attribution sections (gate/coverage stay markdown-only —
  already glance-readable as text). `build_terminal_diff_report` /
  `build_terminal_fluency_report` render a two-line-per-model forest plot (point + 95%
  CI whisker track, pass/fail badge) for `build_diff_report`'s / `build_fluency_report`'s
  verdict sections, fed by `report.py`'s `diff_gap_rows`/`fluency_gap_rows` so the gap a
  chart shows can never diverge from the markdown's gate. Unicode glyphs always print;
  ANSI color only when stdout is a tty and `NO_COLOR` is unset (piped/CI output keeps
  the shape of the win, just uncolored). Wired via `--bars` on `measure`/`verify`/
  `fluency`, printed straight to the terminal (nothing written to disk).
- **`cli.py`** — argparse dispatch for the subcommands.

## API Integrations

- **tiktoken (local)** — token counting under `cl100k_base` and `o200k_base`. No
  network at runtime after the one-time vocab download. Used for measurement and for
  the dictionary coder's cost-aware aliasing threshold. There is **no public local
  tokenizer for Claude 3+**, so cl100k is an estimate; cross-tokenizer invariance
  (cl100k vs o200k) is the keyless robustness check rather than a true Claude count.
- No other external services. terse does not call any tool APIs itself; it compresses
  output that is piped or passed to it.

## Configuration

### Policy file (JSON)

```jsonc
{
  "version": 1,                       // only 1 is supported; anything else is rejected
  "defaults": { "tiers": ["minify", "tabularize", "dictionary"] },
  "policies": [
    {
      "match": { "tool": "gh.*" },    // fnmatch glob on the tool name
      "tiers": ["minify", "tabularize", "dictionary"],   // [] = passthrough (skip)
      "capture": false,               // optional; never PERSIST this tool's payloads
      "structured": "compress",       // optional; "auto" (default) | "compress" | "leave" | "replace"
      "fields": {                     // optional, per-field
        "result[].id":   { "critical": true },           // denylist: never made lossy
        "result[].body": { "lossy": "drop-to-retrieve" },// evicted to a handle; served by terse.retrieve
        // `$`-sigil paths select SPANS of a non-JSON payload instead of fields of an
        // object — the only lossy reach into long-text results. v1 selector: code_blocks.
        "$text.code_blocks": { "lossy": "drop-to-retrieve", "min": 400 }
      }
    }
  ]
}
```

- **Matching:** rules are evaluated in order; the first whose `match.tool` glob
  matches wins. No match → `defaults.tiers`. A rule is matched against, in order: the
  server-qualified name `{server}.{tool}` (when the server is known — see below), the
  tool name as given, then the bare part after multiproxy's `__` peer prefix. The first
  rule matching any candidate wins, so a server-scoped rule outranks a bare-name one.
- **Server-scoped globs need the server name (#83).** A rule like `runecho.*` is
  matching a *server*, but MCP tool names don't inherently carry one: kb names its own
  tools `kb.read.search` (so `kb.*` matches unaided), while runecho calls its tool plain
  `structure` (so `runecho.*` matched nothing and silently fell through to `defaults` —
  the rule looked authored and did nothing). Tell the proxy who the downstream is and
  the qualified candidate makes such a rule work for every server: `proxy --server-name
  runecho`, which `install-mcp` bakes in automatically from the config's own server name,
  and which multiproxy supplies per-peer from each peer's `name`. Without it, matching is
  unchanged (bare names only). A tool that already self-prefixes is never double-qualified
  (`kb.read.search` under server `kb` stays `kb.read.search`, not `kb.kb.read.search`).
- **Tiers:** any subset of `minify` / `tabularize` / `dictionary` / `embedded`. `minify` is
  implied by serialization (a warning is emitted if omitted). `[]` = passthrough.
  `VALID_TIERS` and `DEFAULT_TIERS` are deliberately distinct: a policy file that omits
  `defaults.tiers` gets `DEFAULT_TIERS`, so adding a tier to the valid set never silently
  switches it on for configs already on disk.
- **`embedded` (opt-in):** folds a string leaf that is itself a JSON document into
  `{"__terse_json__":1,"f":F,"v":...}`, so the other tiers can reach a body the server
  double-encoded (`{"response_text": json.dumps(body)}` → measured 0.0% → 41.4%). Fires
  ONLY when one of a fixed registry of serializations (`_EMBED_FORMS`) reproduces the
  original string byte-for-byte; `f` records which. Declines duplicate keys, `1.50`/`1.5e0`
  renormalization, unusual whitespace, and any embedded `__terse_*` key. The form ids are a
  wire contract — append, never edit, or payloads already encoded decode wrong.
- **`structured` (default `"auto"`, #128):** `"compress"` also runs this tool's
  `structuredContent` through the codec, replacing the typed field with a terse envelope.
  MCP 2025-06-18 lets a tool return that field beside a text block mirroring it, and
  terse historically compressed only the block — but measured against `claude` 2.1.218
  the client forwards the **typed field** to the model and discards the block entirely
  (`scripts/probe/structured_content/`; BENCHMARKS §6's scope note). On such a tool terse
  therefore delivered ~0%; with `"compress"` the same fixture measures **61.2%** of the
  model's actual context, captured end to end rather than inferred from the ledger.
  The risk of compressing it unconditionally is asymmetric: any client that validates the
  typed field against the tool's `outputSchema` (which the spec says clients SHOULD do)
  would see terse **break tools that worked**, while leaving it alone merely makes terse
  a no-op. `"auto"` — the default — resolves that **per connected client** rather than by
  guessing: the MCP `initialize` request carries `params.clientInfo`, a name the client
  *declares*, and the proxy sees it. terse compresses the typed field for clients measured
  not to validate (`policy.STRUCTURED_SAFE_CLIENTS`) and leaves it alone for every other
  client, for one that omits `clientInfo`, and for a library caller that never handshakes
  — **fail-closed in all three cases**. Explicit `"compress"`/`"leave"` always override it.
  The list is keyed on name, not name+version: a version ceiling would block every future
  release on one measurement, and a listed client that started validating would fail
  *loudly* (tool calls error visibly), with `"leave"` as a one-line override. The client
  version is logged under `--debug` so such a regression is diagnosable from a log.
  Codec only, no diff: diffing the typed field needs its own per-tool base and keyframe
  accounting, and is deliberately left unbuilt rather than half-built.
- **`structured: "replace"` (#128 option 2)** compresses the typed field *and deletes the
  text block that mirrors it*. **Measured benefit against the reference client: zero.**
  Context cost goes 2,596 → 1,008 chars under `"compress"` and 1,008 → 1,008 under
  `"replace"`, because `claude` 2.1.218 had already discarded the block — what it removes
  is stdio bytes, not context. It is documented rather than deleted because it is the
  correct behavior for a client that forwards *both* fields (which would otherwise halve
  its saving, and which under `"compress"` can receive a cross-call **diff** in the block
  contradicting a full envelope in the typed field). No such client has been measured.
  `"auto"` never resolves to it: every mode up to `"compress"` is invisible to a client
  that ignores `structuredContent`, and this is the first that removes information from
  the wire. Five guards, each independently tested, must all hold or the block stays —
  the mode resolves to `replace`, the rule has non-empty `tiers` (`[]` is the hands-off
  switch), the result is not an `isError`, there is exactly one text block, and that
  block's parsed JSON **equals** `structuredContent`. That last one is what makes it safe
  rather than merely measured: a block carrying anything the typed field does not is not
  a mirror and is never dropped. Deliberately *not* a guard: whether the tool declared an
  `outputSchema` — the expected gate, measured false (the `noschema` probe's tool declares
  none and the client read the typed field anyway).
- **`capture` (default `true`, #85):** `false` means *never persist this tool's
  payloads* — the proxy skips both disk sinks that write raw content: the
  `--capture-dir` corpus tee and the `--debug-log` replay trace (whose records embed the
  raw payload too, so gating only the tee would be half a guard). The payload-free stats
  ledger still counts the tool — sizes and decision, never content — so a gated tool is
  measured, just never quoted. Nothing about what the client receives changes.
  This is **not** implied by `"tiers": []`: passthrough only stops compression and diff
  state; the capture tee sits above the tier logic and would still write to disk.
  Its purpose is to make "this tool's output must never hit disk" a declarative property
  of the policy — surviving re-wraps, reviewable in one place, and expressible under
  `--config` multiproxy (one process, one `capture_dir`, so omitting the flag can't
  exclude a single peer) — rather than an operator remembering never to pass
  `--capture-dir` to one particular wrapper. The motivating case is a
  credential-returning server: secret-broker's `reveal_credential` returns a plaintext
  value by design. Parsing is deliberately strict (`_coerce_capture`): a non-bool raises
  at load rather than coercing, because every wrong-typed value in Python is truthy
  (`bool("false") is True`) and a lax coercion would silently re-enable the guard — the
  one direction a typo must never fail in.
- **`critical`:** a denylist against lossy ops — a `critical` field is never truncated
  or dropped, even if also marked `lossy`.
- **`lossy`:** `truncate` and `drop-to-retrieve` are implemented (opt-in, off by default);
  `summarize` is accepted by the schema but deferred — it emits a warning and leaves the
  field lossless.
- Validation: unknown tiers and unsupported versions raise `ValueError` at load time.

`policy.example.json` ships a policy that encodes the measured insight (gh/runecho
full tiers, kb drops dictionary, `*.rate_limit` skipped).

### Environment

- `TERSE_FLUENCY_BASE_URL` / `TERSE_FLUENCY_API_KEY` / `TERSE_FLUENCY_MODELS` — the
  OpenAI-compatible endpoint, key, and model list for the live fluency eval (broker
  pool or a loopback gateway). Absent by default; the pure core runs without them.
- `CLAUDE_CONFIG` — overrides the Claude Code config path `install-mcp`/`uninstall-mcp`/
  `mcp-status` read and write (default `~/.claude.json`).
- `TERSE_MCP_CMD` — overrides the launcher command `install-mcp` wraps a downstream
  server with (whitespace-split; for unusual installs, e.g. pointing at a non-default
  `terse` binary). Must be an absolute path — spawned via `execve`, no shell to expand `~`.
  Unset, `install-mcp` prefers the `terse` console script on `PATH` from outside its own
  virtualenv and falls back to `<interpreter> -m terse` (`terse_invocation`, #275).
- `XDG_STATE_HOME` — overrides where `terse stats` persists its ledger
  (`$XDG_STATE_HOME/terse/stats.jsonl`, fallback `~/.local/state`).
- `NO_COLOR` — when set, disables ANSI color in terminal reports (`terminal_report.py`).

## Deployment

terse is a library + CLI, not a service.

```bash
uv sync                       # create the venv, install pinned deps (uv.lock)
uv run terse <subcommand>     # run the CLI
```

To use it as a library, import `terse.policy.apply` / `terse.transforms.compress`.

To run a downstream MCP server behind terse, wrap its launch command with the proxy:

```bash
terse proxy --policy policy.example.json -- <downstream server command and args>
```

In an MCP client config (e.g. Claude Code's `mcpServers`), set the server's command
to `terse` and prefix the original command/args after `proxy --policy <file> --`. The
proxy speaks plain MCP stdio, so the client needs no changes. Note: the policy's tool
globs match the **downstream tool names** as that server defines them (not any client-
side `mcp__<server>__` prefix). There is no remote infrastructure and nothing to roll
back; reverting is `git checkout` of a prior commit.

## Maintenance Commands

```bash
uv run pytest                 # full suite (the lossless gate + measurement + policy)
uv run pytest -q tests/test_roundtrip.py   # just the lossless gate
uv run terse measure          # token savings report over corpus/ -> reports/
uv run terse probe            # ceiling probes (value redundancy, cross-call overlap)
uv run terse validate         # cross-tokenizer invariance (cl100k vs o200k)
uv run terse stats --since 7d # live savings from real proxy sessions (payload-free ledger)
```

Reports are written under `reports/` (gitignored). The corpus under `corpus/` is
gitignored because captured tool output may contain real data.

## Releasing (automatic, with one manual follow-up)

Publishing is **zero-touch** — there is no manual tag or version bump.
`.github/workflows/release.yml` runs on every push to `main` and:

1. Derives the next version from the Conventional-Commit types since the last tag —
   `feat:` → minor, `fix:`/`perf:` → patch, a breaking `type!:`/`BREAKING CHANGE` → minor
   while 0.x. A push whose commits are only `docs:`/`chore:`/`test:`/`ci:`/etc. releases
   nothing.
2. Creates and pushes the `vX.Y.Z` tag (hatch-vcs makes that tag the version).
3. Builds the sdist+wheel, cuts a GitHub Release with auto-generated notes, and publishes
   `terse-mcp` to PyPI via Trusted Publishing (OIDC — no stored token), after re-running
   the full test suite against the tagged tree as a publish gate.

The **one manual step** is the changelog: in the first pull request opened after a release,
run

```bash
python3 scripts/release/graduate_changelog.py <tag> CHANGELOG.md
```

to move `[Unreleased]` into its versioned section. This is not automated because the
workflow cannot push to protected `main` on a user-owned repo without a stored PAT (the
header of `release.yml` carries the full reasoning). It is enforced, not remembered:
`tests/test_changelog_covers_every_release.py::test_unreleased_does_not_describe_work_that_already_shipped`
fails that pull request until the section moves.

Two manual overrides remain: push a `vX.Y.Z` tag by hand (released verbatim), or use the
Actions **Run workflow** button with a forced bump level. To require a human click before
each PyPI upload, add a required reviewer to the `pypi` environment in
Settings → Environments; leaving it unprotected keeps releases fully automatic. There is
no remote infrastructure and nothing to roll back at runtime; a bad release is superseded
by the next patch (PyPI versions cannot be re-uploaded).

## Known Limitations

- **Tier 1 (lossy) — `truncate` and `drop-to-retrieve` built; `summarize` deferred.**
  A field marked `{"lossy":"truncate","max":N}` (and not `{"critical":true}`) is capped +
  annotated, gated by `lossy.acceptable_loss`: only marked, non-critical fields may differ,
  each only as a valid truncation. A field marked `{"lossy":"drop-to-retrieve"}` (with an
  optional `min` size floor) is replaced by a `__terse_dropped__` handle; the original is
  stored per session (proxy-only, LRU-bounded by count and bytes, cleared on reconnect) and
  served back by a synthetic `terse.retrieve` tool the proxy injects into `tools/list` and
  answers itself. Its gate `lossy.droppable_loss` accepts a drop only if the handle resolves
  to the exact original (recoverable == acceptable); store writes are staged and committed
  only on gate pass, so a failure leaves no orphan handles. A retrieve miss (evicted, or a
  pre-reconnect handle) returns a legible `isError`, never a protocol error.
  **`"$text.code_blocks"`** applies the same mode to a payload that never parsed as JSON,
  addressing spans instead of fields: every fenced code block over `min` chars becomes the
  same one-line handle marker (one drop form for the primer to teach, not two). It exists
  because the lossless codec has nothing to fold in prose — 60 captured `codegraph_explore`
  results measured 0.0% saved with 89.2% of their tokens in fenced source; the selector
  takes that corpus to 87.0%. Its gate `lossy.text_droppable_loss` is strictly stronger
  than `droppable_loss`: it restores the WHOLE payload from the emitted text plus the store
  and demands byte-for-byte equality, so nothing outside a dropped span can have moved. A
  payload that already contains a marker line therefore fails the gate and passes through
  untouched — the marker-collision guard falls out of the gate rather than needing its own
  check. A dropped text payload is never used as a CDC diff base (the base would depend on
  which spans cleared the size floor); the next raw text re-anchors as a full. `summarize`
  (needs a model in the proxy) is parsed but deferred — warned and left lossless. All lossy
  is off by default; each mode replaces the round-trip gate with its own acceptable-loss
  gate only where a field is explicitly marked.
  **Behavioral gate (#46):** `droppable_loss` proves a drop is recoverable *if* the model
  retrieves it — it says nothing about whether a real tool-calling model actually reaches
  for `terse.retrieve` when the dropped field is needed, or leaves it alone (no over-fetch)
  when it isn't. `terse fluency --drop-eval --policy <policy with a drop field>` answers
  that directly: a live 2-turn tool-loop eval (system primer + the same `RETRIEVE_TOOL_DEF`
  the proxy injects, the model decides whether to call it, the harness resolves the handle
  exactly as `Interceptor.answer_retrieve` would) scored on retrieve-recall (called when
  needed), no-overfetch (didn't call when not needed), and final-answer accuracy — gated on
  the worst model, same honesty bar as the rest of the `fluency` package. Live-model-only,
  no offline/pack mode (mirrors `fluency --diff`'s precedent) — run it before recommending
  `drop-to-retrieve` in a generated policy.
- **Proxy: the model must understand terse's format.** The proxy compresses tool
  results in place, so the model receives the table/legend form. It is self-describing
  (a `cols` header, an inline legend) and needs no decode step, but a model that has
  never seen the format might read it less fluently than raw JSON. The proxy delivers a
  one-time format primer by augmenting the MCP `initialize` result's `instructions`
  field (which compliant clients add to the model's system context) — paid once per
  session, not per call. This is load-bearing: `terse fluency --diff` with NO system
  primer (the inline-note-only condition) showed `gemini-2.5-flash` regressing ~20% on
  diff reconstruction, while the same model with a system primer held 100% — i.e. the
  *inline* note can't substitute for the system-level primer for weaker models. Caveat:
  not all clients surface `instructions`; the inline self-describing forms remain the
  fallback. This was the main open question for proxy *usefulness* (correctness is
  covered by the round-trip tests).
  **Measured (`terse fluency`):** the "Claude Haiku 4.5" figure previously quoted here
  (96–100% paired, ~38% token saving, jointly attributed alongside Gemini 2.5 Flash and
  a DeepSeek alias-resolution figure of 83% bare / 100% with primer) predates the fix
  (#249, 2026-08-19) that let this harness reach real Anthropic models at all — the
  gateway's `claude-sonnet-5`/`claude-fable-5`/`claude-haiku-4-*` ids were DeepSeek
  aliases. Those figures were panel-level, not cleanly separable by model, and the panel
  included the mislabeled Haiku entry — treat the whole paragraph as unconfirmed legacy
  data, not as individually reattributable to Gemini or DeepSeek, and not as flatly
  wrong either (a `cli:`-backend result, or an id that never carried a `claude-*` alias,
  such as a direct Gemini/DeepSeek route, is unaffected by this specific bug). The ~38%
  token saving is deterministic tokenizer arithmetic and still holds regardless.
  **Real measurement, via `claude -p` OAuth,
  on the stress corpus (`--trials 1` throughout):** Opus 5 shows 0 regressions with or
  without the primer; Haiku 4.5 shows a real 8-point gap without the primer (92% vs.
  100% raw); Sonnet 5 shows a smaller gap (96% -> 100%). Opus and Sonnet are both
  single-trial reads and unconfirmed — a later same-day deepening pass on the same
  issue re-ran four *other* (substitute, non-Anthropic) models at `--trials 3-5` and
  found three of four flipped conclusion under more trials, including one where the
  primer measurably *hurt* a smaller substitute model. Only Haiku, among the named
  models, was re-run at `--trials 3`, and it held (92% -> 99%, closed to within noise
  by the primer) — the one figure here that is independently replicated.
  The decisive lever is the `~N`
  dictionary alias — especially the whole-subtree variant where `~N` expands to an
  entire OBJECT (the `deref` question probes this). Takeaway: the primer's benefit is
  real but **model-dependent**, not universal, so "always inject the
  primer" is not the correct blanket policy — nor, on this evidence, is "never inject
  it." Verdict gates on the worst model, not the
  mean. See #249 for the full panel, the replication story, and its still-open
  real-payload-corpus precondition.
  Run-to-run noise at
  temperature 0 is no longer eyeballed: `terse fluency --trials N` repeats each question
  N times and reports accuracy with a pooled binomial confidence interval — though #295
  (open) found that bound collapses to `±0` at `--trials 1` regardless of true accuracy,
  which is exactly the failure mode #249's deepening pass demonstrated three times above;
  read a low-trial interval as directional at best, not as the tight bound this sentence
  used to promise unconditionally. (Parametric SE over N×Q×P Bernoulli draws is
  used rather than the std of N whole-eval runs, which is itself noise at small N.)
- **Proxy transport and fan-out (#5).** A `terse proxy` downstream is either a stdio
  command or an MCP Streamable-HTTP `url` (`transport.py`'s `Transport` abstraction —
  `Interceptor`/`pump` are transport-agnostic; drop-to-retrieve's swallow-and-reply logic
  never touches the downstream transport at all, so it needed zero HTTP-specific
  reimplementation). `--config peers.json` fronts N peers (any mix of stdio/HTTP) behind
  one policy/primer/process, exposing each peer's tools under their own names (`{peer}__`
  qualification only on a genuine cross-peer name collision, #168) and sharing one
  content-addressed drop-to-retrieve store across all of them (`multiproxy.py`). v1 scope
  is proportionate, not exhaustive: HTTP is synchronous POST-then-drain (no cross-request
  pipelining) with no standalone GET-SSE listener for unsolicited server pushes; multi-peer
  broadcasts (`initialize`/`tools/list`) wait on every peer up to a bounded timeout before
  merging, and any client method outside `initialize`/`tools/list`/`tools/call` falls back
  to peer 0 (debug-logged). Each is a documented v1 limitation, not a silent gap — revisit
  if a real workload needs it.
- **Cross-call diffing is built and OPT-IN (`proxy --diff` / policy
  `"diff": true`).** Default-off since #170: the tier is correct, but its primer
  paragraph is 34% of the primer (190 of 555) and the live ledger shows a 0.38% hit rate.
 The probe shows 91% overlap between successive
  same-tool calls; the proxy emits a lossless delta against the prior result (keyed row
  diff for record arrays, shallow key diff for objects) instead of the full payload. It
  is stateful (per-tool last result), self-verifying (a diff is sent only when it
  provably reconstructs the result), and fail-open (full form whenever a diff doesn't
  apply or isn't smaller — the dangling-reference fallback). Its two model-side risks were
  both measured and cleared — the tier is correct; #170 returned it to opt-in on cost
  grounds (above), not on a fidelity finding:
  (1) the round-trip gate proves the diff reconstructs but **not** that a model *reads*
  it as well as the full form — since PASSED by `terse fluency --diff` (4-model panel,
  incl. the nested-record surface, #72 — per the 0.26.0 changelog entry (#249), this
  panel's gateway ids are confirmed DeepSeek aliases, so it was two DeepSeek models
  measured twice under Anthropic names, not four; the PASS itself is unaffected by
  which name each run carried) and at chain depth by `fluency --diff-soak`
  (#75); (2) the diff references the prior result in the model's context, which a
  context compaction could evict. Risk (2) is bounded by **keyframes**: the proxy
  forces a self-contained full result after every K consecutive diffs per tool, so a
  chained diff can never drift more than K turns from an anchor a model can reconstruct
  from scratch (`diff_keyframe_interval` policy field / `proxy --diff-keyframe-interval K`,
  default 5; 0 disables). A client *reconnect* (a new MCP `initialize`) is a stronger
  reset signal: the proxy drops every per-tool diff base on it, so the next result of
  each tool re-anchors as a full rather than a delta against a base the rebuilt context
  no longer holds (#20). Both properties are **soaked, not just unit-tested**: the drift
  soak (`tests/test_diff_soak.py`) drives the real Interceptor hundreds of chained hops
  deep — interleaved tools, error interludes, a mid-soak reconnect, and a 300-diff
  unbounded chain — with an independent client-side reconstructor asserting exactness at
  every hop; `terse fluency --diff-soak` measures the model-side analogue (accuracy vs
  chain depth, up to the keyframe bound). The residual gap — a context *compaction* with
  no reconnect — is unobservable over stdio; keyframes cap its blast radius at K turns,
  and `--no-diff` remains the escape hatch for a consumer where that residual matters.
- **Cross-block join is built and ON by default (`proxy --no-join-blocks` / policy
  `"join_blocks": false` to opt out, #116).** Several MCP servers return one record per
  content block, so each block is a single object: `tabularize` never sees an array, and
  the diff tier — which reasons about one logical payload — is skipped for the whole
  result (`_compress_or_diff` is entered only when `len(text_blocks) == 1`). On terse's
  own live ledger that shape was **71% of all results** and the single largest token sink
  (`kb.read.list_principles`, 37% of proxied traffic) was 100% of it. When every text
  block parses as a JSON object, the proxy joins them into one record array before
  compressing (`policy.apply_joined`), then runs the same lossless codec and diff decision
  over the array. Measured on a realistic 80-record `list_principles` payload: per-block
  +9.6% → joined codec +24.9%, and a near-identical repeat call collapses from ~6900
  tokens to ~100 (the diff unlock — an order of magnitude the per-block ceiling can't
  reach). Design points that keep it safe:
  (1) lossy field rules run **per block, before the join** (`_lossy_stage`), so a path like
  `$.results[*].body` still resolves against one record's shape rather than the joined
  array; (2) the join fires only when every block is a JSON **dict** (a mixed bag of dicts,
  scalars, and arrays is not a record sequence) and none carries a reserved terse marker or
  nests past the depth cap — otherwise it falls back to the per-block path and records
  *why* (`multiblock_non_json` / `_heterogeneous` / `_marker` / `_depth` / `_passthrough` /
  `_off`); (3) the per-result verify-before-emit gate proves the joined payload round-trips
  before it ships; (4) N → 1 changes the content-block count, which the MCP spec (2025-06-18
  server/tools) permits — block count carries no meaning, blocks have no index a payload can
  reference — while non-text blocks (image/audio/resource) keep their positions; (5) the
  per-tool diff base tracks whether it came from a joined result, and a join↔single **shape
  flip** re-anchors as a full instead of diffing an array against an object. Independent of
  `diff`: with diffing off, joining still folds records into one compressed block.
- **Text diff (Tier 0.7 text, #25) covers non-JSON results, but only the codec side is
  measured so far.** File reads, source excerpts, and log tails get `applicable: False`
  in `measure_payload` — zero Tier-0 compression — and used to get zero cross-call
  diffing too, since the row/key diff above only reasons about JSON. `text_diff` adds a
  second, independent diff codec for exactly that case: a rolling-hash content-defined
  chunker (`text_diff._chunk`) splits the prior and current text into position-independent
  chunks, and the diff references unchanged chunks instead of resending them — same
  self-describing/fail-open/keyframe contract as the JSON diff, with its own per-tool
  base (`Interceptor.last_text`) so a tool that alternates JSON and text results never
  mixes bases. Round-trip correctness is test-covered (`tests/test_text_diff.py`) and a
  live proxy run confirms ~90-93% token savings on a mostly-unchanged 200-line log
  re-read or append.
  **Behavioral gate:** `terse fluency --text-diff-eval` answers for the text-diff codec
  the same question `--diff` answers for the JSON row/key diff — does a model reconstruct
  the current text as accurately from (previous text + text-diff) as from the full
  current text? Two deterministic questions (line count, exact last line — the part most
  likely new in an append-only log tail) over any same-tool TEXT payload pair in the
  corpus, paired scoring against the worst model, same honesty bar as the rest of the
  `fluency` package. Live-model-only, no offline/pack mode (mirrors `--diff`'s precedent) — run
  it before enabling `proxy --diff` for text-heavy tools against a real model consumer.
- **Marker collision.** A payload that genuinely contains a reserved
  `__terse_table__` / `__terse_dict__` / `__terse_diff__` key (at any depth) can't be
  compressed without the consumer mis-reading the user's own dict as a terse envelope —
  the codec has no escape convention. `policy.apply` detects this
  (`transforms.has_terse_marker`) and passes the payload through uncompressed with a
  warning, so the proxy stays lossless. The `~`-alias namespace can't be exhausted: the
  base-36 generator is unbounded against a finite avoid-set. Real tool output contains
  none of these.
- **Dictionary coding trades some direct legibility for tokens.** A `~0` reference is
  resolved by reading the inline legend in the same payload (no retrieve step), but it
  is less immediately readable than the literal value. It is gated on real token
  savings, so it only fires where it pays.
- **No true Claude token counts without a key.** All shipped numbers use cl100k;
  robustness is established via cross-tokenizer invariance (cl100k vs o200k, ~0.5 pt
  mean divergence), not a direct Claude measurement.
- **Shape classifier is shallow.** `classify_shape` inspects one level; deeply-nested
  record structures may bucket as `compact-json` even though terse folds them. The
  per-tool report exists precisely because shape buckets can mislead.

<!-- codeshot:arch:start -->
![Architecture — generated by codeshot](docs/architecture.svg)
<!-- codeshot:arch:end -->

<!-- docvet:anchors
CLAUDE_CONFIG -> src/terse/install_mcp.py
NO_COLOR -> src/terse/terminal_report.py
TERSE_MCP_CMD -> src/terse/install_mcp.py
XDG_STATE_HOME -> src/terse/stats.py
install-mcp -> src/terse/cli.py
mcp-status -> src/terse/cli.py
policy generate -> src/terse/cli.py
tune -> src/terse/cli.py
uninstall-mcp -> src/terse/cli.py
verify -> src/terse/cli.py
-->

