Metadata-Version: 2.4
Name: agentmeasure
Version: 0.5.0
Summary: AgentMeasure — check-up report for coding agent runs + conformance pack for telemetry fixture verification. Zero external dependencies.
License: MIT
Keywords: agent,codex,claude,observability,measurement,healthcheck,conformance
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# AgentMeasure Healthcheck (pre-release)

**A local check-up report for your coding agent runs.**
Point it at the logs your agent runtime already writes. It answers three
questions with evidence — *which records are duplicated, where did the agent
pay retries for one logical operation, and where did the same tool fail over
and over* — and writes a terminal summary plus a local HTML report.

Same honesty rules as the [conformance pack](../conformance/pack/README.md):
every verdict is `OK / FINDING / UNPROVABLE`, and UNPROVABLE is a first-class
result — when the logs cannot decide, that is disclosed, never zeroed.

> Status: engineering preview (v0.3.0). Adapters: **Codex rollout logs** and
> **Claude Code session logs** (`--runtime claude`; auto-detected when Codex is
> absent). Claude v1 scope: duplicate / retry / error-run checks fully supported;
> session token accounting is honestly **UNPROVABLE** — Claude transcripts carry
> per-request usage only, no cumulative counter, and per-request sums are never
> presented as session totals.
> Distribution name decided: **`agentmeasure`** on PyPI (command of the same
> name). PyPI publish is armed and goes live with tag `v0.3.0` once the PyPI
> account/trusted publisher is configured — until then use the git install.

## Install

> **Note (2026-09-21):** PyPI still serves **0.4.0**, which crashes on the
> first command (`NameError` in `_run_check`; `demo` / `check` / `compare`
> all affected). The fix is in 0.4.1 and already on main. Until the 0.4.1
> wheel lands on the index, install from git instead (option 3 below) —
> that path runs clean.

```bash
# 1) from PyPI (live since v0.4.0 — see note above; 0.4.1 pending upload)
pipx run agentmeasure demo

# 2) install the command
pipx install agentmeasure

# 3) from git (tracks main)
pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"

# 4) zero install, straight from a repository checkout
python3 healthcheck/agentmeasure demo
```

Python 3.9+ standard library only; zero runtime dependencies; no network code
(a test enforces both).

## Quick start

```bash
agentmeasure demo      # synthetic session — see it work, no data needed
agentmeasure check     # your agent logs, last 7 days (Codex or Claude Code)
agentmeasure check --all           # every local session
agentmeasure check --runtime claude   # force the Claude Code adapter
agentmeasure check --dir ~/.codex/sessions/2026/09/05
agentmeasure selftest  # adapters + checks on bundled fixtures
agentmeasure validate export.json  # check an export against its schema
agentmeasure share run.json        # PREVIEW the sanitized summary (writes nothing)
agentmeasure share run.json --out summary.md   # export after review
agentmeasure --version
```

(Without installing, prefix the commands with `python3 healthcheck/agentmeasure`.)

Options for `check` / `demo`: `--html PATH` (default `agentmeasure-report.html`),
`--json PATH` (full machine-readable export), `--share PATH` (sanitized summary,
`.md` or `.json`), `--days N`, `--save-snapshot [PATH]`, `--no-history`.
`demo` accepts the same output paths and `--no-history`.

Input filters for `check` and `compare`:

```bash
--since 2026-09-01 --until 2026-09-05   # date range (overrides --days)
--project myrepo                        # keep sessions whose project matches
                                        # (project = cwd basename, case-insensitive)
```

## Outcome-billing verification: prepare → recount → dispute → recovery

The buyer-side bill check for per-resolution vendors (Intercom Fin, Zendesk AI
Agents, …). Rules live in the [vendor-rules registry](../registry/vendor-rules.json)
(single source of truth, each rule cited and confidence-graded). Four commands,
one pipeline:

```bash
agentmeasure prepare --export native-export.csv --vendor zendesk --inspect
agentmeasure prepare --export native-export.csv --vendor zendesk --out prepared.csv
#   native Intercom/Zendesk export -> canonical skeleton. Judgement columns
#   stay EMPTY for human review; native signal columns travel as hint_*
#   cells; prov_* cells record where every value came from. No verdicts.

agentmeasure recount --export prepared.csv --vendor zendesk --inspect
agentmeasure recount --export prepared.csv --vendor zendesk --json recount.json
agentmeasure recount --export prepared.csv --vendor zendesk --contract contract.json
#   Tier 1: the vendor's own published rules, line by line. Findings in BOTH
#   directions; absent evidence is UNPROVABLE, never guessed. --contract adds
#   the buyer's outcome standard (e.g. reopen_window_hours: 72) as a separate
#   outcome lane — renewal leverage, never netted into the claim.

agentmeasure dispute --export prepared.csv --vendor zendesk --contract contract.json
#   the negotiable pack: cover letter citing the vendor's own rule per
#   finding + appendix + (optionally) Tier 2 under the AMS-1 standard.

agentmeasure recovery --pack pack/dispute-pack.json --confirmations confirmations.csv
#   what the vendor actually conceded: credited/paid per disputed line, with
#   confirmation dates and evidence refs. Realized value is counted
#   separately from the claim — an argument won is not cash.
```

`agentmeasure recount --list-vendors` shows the registry with each rule's
source and confidence (P = primary/official, S = secondary, U = unverified).
The browser-local recount at [the website](../website/) runs the same Tier 1
with no upload; a cross-language parity test keeps the two from drifting.

### The whole story in one artifact

```bash
agentmeasure crosscheck --export prepared.csv --vendor zendesk --ledger billing.csv
#   your billing ledger (计费流水) against the export: charges the export
#   doesn't flag, billed flags the ledger doesn't confirm, amounts off the
#   published price. Disagreements are named, never resolved.

agentmeasure verify --export prepared.csv --vendor zendesk --contract contract.json \
    --ledger billing.csv --confirmations confirmations.csv --out vledger
#   the Verified Ledger: Tier 1 + outcome lane + ledger cross-check +
#   realized recovery in ONE document (json + md), each lane labelled
#   claim / leverage / realized and never summed across.

agentmeasure dashboard --pack vledger/verified-ledger.json --out dashboard.html
#   offline single-file HTML: cards, tables, recovery bar. No JS, no
#   network, printable — renders, never recomputes.

agentmeasure rules-diff last-month-registry.json
#   the monthly ritual: diff last period's rules snapshot against the
#   packaged registry. Exit 1 = a material change (trigger, price, reopen
#   rule) — re-verify the period under the new rule and disclose it.

agentmeasure narrative --pack pack/dispute-pack.json --audience vendor \
    --emit-prompt prompt.txt
#   step 1 of two: the drafting prompt, built from the pack. Run YOUR OWN
#   model on it, then:
agentmeasure narrative --pack pack/dispute-pack.json --check-draft draft.txt --out letter.md
#   step 2: every number in the draft is checked against the pack; any
#   invented or rounded number refuses the draft and the rule-based
#   template ships. The model never touches the numbers. No model? The
#   template is the default.

agentmeasure delivery --log delivery.jsonl --log-event \
    --engagement ACME --vendor zendesk --period-index 1 --phase mapping --minutes 45
agentmeasure delivery --log delivery.jsonl
#   appendix-E metrics from logged phases: first look ≤10h, third period
#   ≤1h, repeat-period reuse ≥70%. Use the mapping fingerprint `prepare`
#   prints to track which engagements share the reusable mapping.
```

Outcome pricing beyond per-resolution support (qualified lead, completed
case) is defined — schema first, per the BP — at
[schemas/outcome-unit.schema.json](../schemas/outcome-unit.schema.json) with
worked examples. The outcome **engine** now ships too:
`agentmeasure outcomes --units units.jsonl` verifies them fail-closed, and the
schema covers action and usage-aggregate units — one evidence plane across the
BP's three stages.

```bash
agentmeasure crm-join --export prepared.csv --crm crm.csv --out joined.csv
agentmeasure outcomes --units outcome-units.jsonl --json outcome-ledger.json
agentmeasure period-compare last-month/verified-ledger.json vledger/verified-ledger.json
agentmeasure template --export --vendor intercom --fingerprint <fp> --contract contract.json
agentmeasure certify --pack vledger/verified-ledger.json --digest-out digest.txt
agentmeasure incrementality --events treatment-holdout.csv
```

### Agent commerce: the receipt where the click disappears

Traditional attribution starts at the click; agent commerce has none. The
merchant-side measurement receipt (F2.1-F2.9):

```bash
agentmeasure commerce-ledger \
    --events agent-events.jsonl --orders merchant-orders.csv \
    --payments merchant-payments.csv --policy-version 1.0.0
#   metric + value + evidence table; retries reconcile to 1 Operation /
#   2 Attempts / 1 verified order; net GMV is strong; incremental is
#   UNPROVABLE (no holdout, no claim); materiality prints the discrepancy
#   vs the naive dashboard in currency AND percent.
```

live / replay / synthetic never mix inside one aggregation (error, not a
blend); metric names carry their strength prefix and are linted against the
evidence that backs them; every input is sha256-anchored. The operating layer on top:

```bash
agentmeasure pnl-trees --orders orders.csv --campaigns campaigns.csv   # Organic / Paid, ROAS in Paid only
agentmeasure cm-ledger --orders orders.csv --costs costs.csv --period 2026-09
agentmeasure demand-audit --measurements m.csv --thresholds t.json     # Launch / Watch / Not ready
agentmeasure intent-taxonomy --file clusters.json                      # Operating Cell health
agentmeasure decision-audit --policy policy.json --executions log.jsonl
agentmeasure benchmark-export --rows brand-rows.csv --min-brands 5     # aggregates only, ever
```

 Schema:
[schemas/commerce-profile.schema.json](../schemas/commerce-profile.schema.json);
canonical retry vector pinned in CI at
`conformance/vectors/commerce-retry-baseline/`.

`certify` grades the artifact on the AMS-1 adoption ladder (Stage 0..3) and
emits the deterministic digest; hand the digest to your own signing tool —
this package never runs one. `incrementality` produces statistics about a
difference, never a verdict: the BP's line between rule judgement and causal
inference stays visible in the output itself.

## Sharing results: preview first, export second

The recommended flow never writes a shareable file until you have read it:

```bash
agentmeasure check --json run.json          # the report export embeds the summary
agentmeasure share run.json                 # terminal PREVIEW — nothing is written
agentmeasure share run.json --out summary.md  # export only after you reviewed it
```

`share` re-checks the summary against its whitelist before showing or writing
anything, so a tampered or future-format export is refused rather than leaked.
`check --share PATH` remains available as the one-step explicit alternative.

## Snapshots & compare — did the change help?

The re-run preview: save a snapshot, change something (a prompt, a Skill, a
dependency), run the agent again, compare.

```bash
python3 healthcheck/agentmeasure check --save-snapshot before.json
# …make your change and run the agent…
python3 healthcheck/agentmeasure check --save-snapshot after.json
python3 healthcheck/agentmeasure compare before.json after.json
# or skip the second snapshot: compare runs a fresh check as side B
python3 healthcheck/agentmeasure compare --all before.json
```

`compare` prints metric deltas (failed executions, retry chains, …), check
verdict transitions (`HC-02 OK → FINDING`), and honest caveats: token totals
are differenced only when **both** sides are provable, and version / window /
mode mismatches are disclosed because they change what a delta is allowed to
mean. Snapshots are local artifacts (they contain project names and session
short-ids); the snapshot schema is versioned and validated on load.

## Versioned export schemas — how other tools integrate (R8)

Everything a third party needs is a JSON document with a version on it:

| document | written by | schema |
| --- | --- | --- |
| report export | `check --json PATH` / `demo --json PATH` | `report-v1` |
| snapshot | `--save-snapshot PATH` | `snapshot-v1` (schema field = 1) |
| comparison | `compare --json PATH` | `compare-v1` |

Reference schemas live in [`schemas/`](schemas/). Compatibility policy:
**within a schema version, fields are only added — never renamed, retyped, or
removed**; a consumer that meets an unknown newer version must stop and say so.
`agentmeasure validate FILE…` checks any export against its schema, and
[`examples/track-weekly.py`](examples/track-weekly.py) is a complete external
consumer: it reads snapshots through the schema only, imports nothing from the
package, and refuses future schema versions. MCP-style servers can wrap the
same exports later; the exports, not internal modules, are the integration
surface.

## What it checks

| id | check | question it answers | verdict basis |
| --- | --- | --- | --- |
| HC-01 | Duplicate records | byte-identical lines, repeated call/execution ids, sessions split across files, and the format's **by-design dual recording** when model-side calls and command events are both present | exact line hashes, id counts |
| HC-02 | Retry amplification | same command failed then re-run: how many executions carried one logical operation | consecutive same-command blocks whose first attempt failed |
| HC-03 | Tool error runs | ≥3 back-to-back failures of the same tool with no success between | consecutive failed executions, same tool kind |

Plus a coverage overview: sessions, turns, executions by outcome, token
consumption (with subset discipline), compactions, sub-agent activity, corrupt
lines, and a **per-project breakdown** (project = cwd basename, canonical
deduplicated counts). Every finding carries evidence — file, line number, exit
codes — and a concrete next step.

## Honest-number rules (the part most tools get wrong)

- **Executions are counted from exactly one stream.** Codex Desktop can write a
  tool call as both a `response_item` record and an `item_completed` event.
  Summing both — or grepping the file — double-counts when both are present.
  HC-01 discloses the observed ratio instead of hiding it.
- **Token subsets are never added into totals.** `cached_input` ⊆ input,
  `reasoning_output` ⊆ output in Codex logs. The report shows subsets
  alongside, never inside, totals.
- **Token totals use the last cumulative snapshot per session**, not sums of
  per-event deltas (which double-count after context compaction). The aggregate
  is marked UNPROVABLE unless every logical session has one valid snapshot;
  missing or malformed snapshots are reported separately.
- **Corrupt or truncated input lowers every count.** All numbers are lower bounds
  where lines failed to parse or a file hit the safety cap; the limitation is
  reported, never dropped.
- **UNPROVABLE, not zero.** Sessions whose format lacks outcome events make
  HC-02/HC-03 UNPROVABLE for that part — reported as such.

## Statistics definitions (for support and accounting)

| term | definition |
| --- | --- |
| execution | one `CommandExecution` / `McpToolCall` event; its outcome is `ok`, `failed`, or `unknown` |
| retry chain | a maximal block of consecutive executions of the same command whose first attempt failed; one chain = one logical operation executed N times |
| resolved chain | a chain whose last attempt succeeded |
| tool error run | ≥3 consecutive failed executions of the same tool kind with no success or unknown in between |
| token total | last cumulative `total_token_usage` snapshot per session, summed only when every session has a valid snapshot; otherwise UNPROVABLE |
| project | basename of the session's recorded working directory (`session_meta.cwd`); used for the breakdown, `--project` filtering, and snapshots — never included in the share summary |
| snapshot | versioned JSON record of one run's window, verdicts, and aggregate counters (schema 1); local artifact |
| comparison delta | `B − A` per metric; tokens are differenced only when provable on both sides, and version/window/mode mismatches are disclosed as caveats |
| demo run | executed on the bundled synthetic session; counted separately (`synthetic-demo`) |
| own-data run | executed on real local logs (`own-data`) |
| run number | local run history at `~/.agentmeasure/history.jsonl` distinguishes first vs repeated runs; delete the file to reset |

## Privacy

- **No network.** The package contains no network code; a test enforces this by
  scanning every module for network imports.
- **Raw data stays local.** Prompts, message text, and reasoning content are
  never read into the model (only envelope types, ids, exit codes, durations,
  and token counters).
- **Two artifacts, two levels.** The HTML report is personal (local paths and
  evidence metadata — raw commands are represented by hashes, and the banner
  says so). The `--share` summary
  is built from a fixed whitelist of aggregate counts: no prompts, paths,
  commands, repo names, or session ids can appear in it, and a planted-string
  test enforces this.
- **Run history** (`~/.agentmeasure/history.jsonl`) stays on the machine and is
  deletable.

## Verified runtime support

| runtime | status | verified against |
| --- | --- | --- |
| Codex rollout JSONL (Desktop originator) | supported | cli 0.142.3 – 0.153.0; review snapshot 54 local files, 48 sessions, ~83k lines (2026-06-30 → 2026-09-05). Sessions without `CommandExecution` events (older format) run with HC-02/HC-03 UNPROVABLE, disclosed. |
| Codex CLI (`codex_cli_rs` originator) | expected compatible, **not yet verified** — no local samples | pending real samples |
| Claude Code (`~/.claude/projects`) | detected, **not yet supported** | adapter ships after validation against real samples |

New cli versions are handled defensively: unknown event types are accounted
(`unknown type` counts), never silently dropped.

## Exit codes

| code | meaning |
| --- | --- |
| 0 | the check ran and produced a report (findings or not) |
| 1 | selftest fixture mismatch (development only) |
| 2 | input/output error: no readable logs in the window, bad `--dir`/arguments, or an output path cannot be written |

## Development

```bash
python3 -m unittest discover -s healthcheck/tests   # 248 tests
python3 healthcheck/agentmeasure selftest           # fixtures + redaction + compare + schemas
bash healthcheck/scripts/smoke_install.sh           # clean-venv install smoke (packaged artifact)
```

The install smoke is also a CI job (Python 3.9/3.11): it builds the package,
installs it into a fresh venv, and runs `selftest`, `demo`, `validate`, and the
external example from the installed artifact — the repository checkout is not
used.

Python 3.9+ standard library only. Zero dependencies, by policy: if any step
asks you to register, connect to a network, or pay — that is a bug.

## Feedback

- [Open an issue](https://github.com/roy-tong/AgentMeasure/issues) — include
  the `--json` export (it contains local paths; strip them first if the log
  location is sensitive).
- [Email](mailto:tongroy18@gmail.com?subject=AgentMeasure%20Healthcheck%20feedback)
