Metadata-Version: 2.5
Name: evidence-readiness
Version: 0.1.2
Summary: Checks whether an agent trace contains the evidence a post-hoc causal reconstruction would require. Verifies evidence is PRESENT, not that it is true: a fabricated-but-internally-consistent trace passes.
License: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# evidence-readiness

**Checks whether an agent execution trace contains the evidence a post-hoc
causal reconstruction would require.**

When an AI agent causes a loss — a bad config change, a cross-session data
leak, contaminated output cascading between agents, a prompt injection —
someone eventually has to reconstruct *why* from the trace. Traces from
general-purpose agent instrumentation cannot support that reconstruction: the
evidence was never recorded. This repository contains the evidence-readiness
specification ([SPEC.md](SPEC.md)), a checker that evaluates a trace against
it, and reference fixtures showing what conformant instrumentation looks like.

## What this is — and is not

- **It checks that evidence is *present*, not that it is *true*.** A
  fabricated-but-internally-consistent trace passes. This tool authenticates
  nothing, and its result must never be relayed as "this agent is safe."
- **It never determines whether a failure occurred.** It answers "does this
  trace contain what a reconstruction of class X would require," never "did
  class X occur." It contains no detection or scoring logic.
- **This repository does not contain the reconstruction engine.** The
  specification is published so the method is assessable; the engine is
  available for inspection under the terms in `SOURCE_AVAILABILITY.md`.
- **Four failure classes** are covered: `config_drift`,
  `session_isolation_failure`, `cascade_contamination`, `prompt_injection`.
  A fifth (`tool_misuse`) was investigated and declined for stated
  structural reasons — a documented negative result, not a gap (SPEC.md §8).
- The spec is **a profile of the OTel GenAI semantic conventions: core OTel
  plus documented extensions** (SPEC.md §2). It carries no standards-body
  endorsement.

## Who it is for

- **Engineers instrumenting agents** who want incidents to be
  reconstructable after the fact — diff your trace against a conformant
  fixture and see what to add.
- **Reviewers** (insurers, auditors, counterparties) evaluating whether an
  agent system's telemetry would support retrospective loss adjudication.
- **Coding agents**: see `llms.txt`, `spec.json` (the machine-readable
  spec export, generated — never hand-maintained), and the MCP server
  in `src/evidence_readiness/mcp_server.py` (configuration below).

## Quickstart (60 seconds)

```bash
# from a clone of this repository
pip install .
evr check path/to/trace.json        # or a directory of traces
evr check --json path/to/traces/    # machine-readable report
evr check --require-class prompt_injection path/to/trace.json
```

The package has **zero runtime dependencies**; `pip install .` in any
Python ≥3.10 environment is the whole setup. Equivalent `uvx` forms, both
exercised: `uvx --from . evr check ./traces` from a clone, and
`uvx --from evidence-readiness evr check ./traces` from the package index.

**Both need `--from`, and the trailing `evr` is not optional.** A bare
`uvx evidence-readiness ...` runs the console script *named for the
distribution*, and that script is the MCP server (below), not the checker
— it reads standard input, ignores the arguments, and exits successfully
without checking anything.

Exit codes carry no reconstruction vocabulary: `0` all requirements met ·
`1` one or more requirements partially met · `2` one or more requirements
unmet · `3` malformed input (SPEC.md §9 is the source of record).
Universal requirements (EVR-R1..R6) drive the exit code; per-class evidence
readiness is reported separately, and `--require-class` folds a named
class's readiness into the exit code. For every failed requirement the
output names the specific missing field and the span kind it belongs on.

Every run ends with the same statement this README opened with: **presence,
not truth** — evidence being present does not make it authentic, and a
passing trace is not a safe agent.

## CI: check every PR (GitHub Action)

A spec is read once; a check that runs on every PR changes instrumentation
behavior. Add this to a workflow in the repository that produces your
traces (`evidence-readiness` is this repository's owner once published; from
within this repository itself, `uses: ./.github/actions/readiness-check`):

```yaml
steps:
  - uses: actions/checkout@v4
  - uses: evidence-readiness/spec/.github/actions/readiness-check@main
    with:
      trace-path: traces/
```

That is the whole setup — the default GitHub runner's `python3` is
sufficient, and the Action installs the (zero-dependency) checker itself.
The step fails when the checker's exit code exceeds
`minimum-passing-exit-level` (default `0`; exit-code semantics as in the
Quickstart above). Optional `required-classes` folds named classes'
evidence requirements into the exit code. Outputs: `exit-code`,
`exit-meaning`, `report-path` (full JSON report), and
`per-class-readiness`. PR annotations name each incomplete requirement's
specific missing field and span kind. Every annotation is a statement
about the **trace telemetry**, never about the agent: an annotated PR has
incomplete evidence, not a detected failure — and a clean run is not a
safe agent. This repository's own CI (`.github/workflows/ci.yml`)
dogfoods the Action against `fixtures/`, asserting the expected —
deliberately nonzero — exit codes per bucket.

## MCP server (agent surface)

`src/evidence_readiness/mcp_server.py` exposes the checker and the
requirement data to agents over the Model Context Protocol (stdio).
Three tools: `check_trace_readiness` (file, directory, or inline JSON —
inline content is processed in memory and never written anywhere),
`get_requirement`, and `list_class_requirements`.

Installing the package delivers **three** console scripts, and it is worth
being precise about which is which before you copy anything below:

| script | what it runs |
|---|---|
| `evr` | the checker CLI — this is the one a human wants |
| `evr-mcp` | the MCP server, on stdio |
| `evidence-readiness` | the MCP server, on stdio — same target as `evr-mcp`, named for the distribution so that per-invocation runners which resolve `<package>` to a like-named script find it |

So `evidence-readiness` is **not** the checker. Typing it at a shell gets a
server waiting for JSON-RPC on stdin, which looks like a hang. Use `evr`.

If the package is installed, the `evr-mcp` console script is the whole
configuration:

```json
{
  "mcpServers": {
    "evidence-readiness": {
      "command": "evr-mcp"
    }
  }
}
```

From a clone, with nothing installed, run the module by path instead:

```json
{
  "mcpServers": {
    "evidence-readiness": {
      "command": "python3",
      "args": [
        "/absolute/path/to/evidence-readiness/src/evidence_readiness/mcp_server.py"
      ]
    }
  }
}
```

Either way it needs no dependencies, and the second form needs no
installation. It is **local-only**: no network calls, no phone-home,
nothing persisted.
It deliberately has **no submission tool** — the server never transmits
anything; submitting a trace anywhere is a human decision gated on the
`CONTRIBUTING.md` disclaimer and a redaction judgment an agent cannot
make. Every tool description carries the presence-not-truth statement,
so a readiness result cannot reasonably be relayed as "this agent is
safe."

<!-- mcp-name: io.github.BrienBounded/evr -->


## Status

The specification is versioned ([SPEC.md](SPEC.md) carries its current
version and correction log) and the checker is implemented against it,
with reference fixtures (`fixtures/`), the machine-readable export
(`spec.json`), the agent surface (`llms.txt`, the MCP server), and the CI action
(`.github/actions/readiness-check/`) in place. The methodology and
limitations paper is drafted ([PAPER.md](PAPER.md)); its front-matter
banner records which sections have had an adversarial read and which
have not. The standing
disclosure policy is [SOURCE_AVAILABILITY.md](SOURCE_AVAILABILITY.md);
reporting and trace-submission terms are
[CONTRIBUTING.md](CONTRIBUTING.md), with redaction guidance in
[REDACTION.md](REDACTION.md) and release verification in
[RELEASING.md](RELEASING.md). Corrections are logged in `CHANGELOG.md`.
The spec is descriptive: every requirement is backed by an observed outcome
in a validation record of 240 reconstruction attempts, **222 of them
synthetic** — backing is per-requirement, from the specific cases that
exercised it, not a property measured across all 240 (18 real-world
Who&When cases validate the abstention direction only; the split is stated
wherever the count appears — SPEC.md §1).

## Attribution

Brien Buckman

Licensed under Apache-2.0 (see `LICENSE`).

The substance of this repository's development record is public:
corrections that affect published content are logged in
[CHANGELOG.md](CHANGELOG.md), and the development defect record, as it
bears on the method, is summarized in PAPER.md §8.6. The commit history
in which these files were drafted and revised is not published.
