Metadata-Version: 2.5
Name: postern-conformance
Version: 0.1.2
Summary: Conformance checker for the Postern protocol — points at a running runner and reports which of SPEC.md section 3's levels it actually meets.
Project-URL: Specification, https://github.com/sigrix-io/postern/blob/main/SPEC.md
Project-URL: Source, https://github.com/sigrix-io/postern
Project-URL: Issues, https://github.com/sigrix-io/postern/issues
Author: Sigrix
License-Expression: Apache-2.0
Keywords: agents,conformance,postern,protocol
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: jsonschema==4.26.0
Requires-Dist: rfc3339-validator==0.1.4
Requires-Dist: rfc3986-validator==0.1.1
Description-Content-Type: text/markdown

# postern-conformance

Points at a running [Postern](https://github.com/sigrix-io/postern) runner
and reports which of the specification's three conformance levels it
actually meets — and which of its MUST rules the runner breaks getting
there.

```console
$ pip install postern-conformance
$ postern-conformance http://127.0.0.1:8787
```

```
Postern conformance · http://127.0.0.1:8787
schemas: checkout

  §4.4
    PASS  status answers without credentials
    PASS  status matches its schema
  §3
    PASS  declares Level 3
          Stream — describe, status, run, stream
    PASS  run is implemented at Level 3
  §2.3
    PASS  no wildcard Access-Control-Allow-Origin by default
    PASS  run rejects a non-JSON media type
  …

Declared level: 3
34 pass  0 fail  0 warn  2 skip

Conformant at Level 3.
```

Postern is four HTTP endpoints, so this checker speaks to a runner over the
wire and has no idea what it is written in. A Go, Rust or TypeScript runner
is checked exactly as well as a Python one.

## It does not run your agent unless you ask

A run may invoke tools that spend money and mutate state outside the
workspace ([§4.1.2](../../SPEC.md#412-capabilitieswrite_tools)), and an
abort is not a rollback ([§4.5](../../SPEC.md#45-the-life-of-a-run)). A
conformance checker that ran an agent to find out whether it conforms would
be charging you for the answer, and would have done so before you could
read this.

So by default it checks only rules a runner applies **before** the agent
starts — which is most of the specification:

- the level rule, in full, because a verb above the runner's level must
  refuse and so executes nothing
- every refusal path: a malformed body, a missing required input, a
  revoked entitlement, a media type the runner must not parse
- `describe` and `status` entirely, along with their schemas
- the CORS rules, none of which involve running anything

`--execute` opts into the rest — the `run` response shape, the SSE framing,
the `delta` concatenation invariant, `run_id` uniqueness, and — for a runner
declaring `status.idempotent_retry` — that a key already answered is
refused rather than replayed when it arrives carrying different `inputs`.
There is no way to check those without a real run, which is why they are a
decision rather than a default.

The last of those spends nothing of its own: it presents the key the
uniqueness check already bound, and a conformant runner refuses it without
running the agent. It rides on `--execute` because the key has to have been
bound by a real run first, not because it buys another.

Two refusals ride on it for the opposite reason — not because they need a
run, but because a runner that breaks them performs one. A request failing
a `validation` the agent's own `describe` declares, and a request sent
while `status` reports a declared credential unset
([§4.6](../../SPEC.md#46-the-order-of-refusals) step 5), are both bodies a
conforming runner rejects before the agent starts, so against one they cost
nothing. There is no way to ask that is free against a runner answering
wrongly: the run *is* the answer, and you have to have agreed to pay for
it.

The second is askable only where the runner said so first.
`status.credentials` is OPTIONAL ([§4.4](../../SPEC.md#44-get-posternv0status)),
so a runner publishing no credential state is conformant and its
environment cannot be read from outside — the report says so rather than
passing it quietly.

## Usage

```console
postern-conformance http://127.0.0.1:8787
postern-conformance http://127.0.0.1:8787 --origin https://app.example.com
postern-conformance http://127.0.0.1:8787 --execute
postern-conformance http://127.0.0.1:8787 --json
```

`--origin` names an origin the runner is configured to allow. Without it
the CORS *header* rules are skipped: which origins a runner allows is the
runner's own decision, and the specification fixes only the two ends of it
([§2.3](../../SPEC.md#23-browser-clients)). The rules that hold for *any*
origin — no wildcard default, no `Origin: null` — are checked either way.

| Exit | Meaning |
|---|---|
| 0 | No MUST rule was broken |
| 1 | At least one MUST rule was broken |
| 2 | The runner could not be checked at all |

**A SHOULD cannot fail the run.** It reports as `WARN` and the exit status
stays 0. A checker that failed a runner for declining an option the
specification left open would be ignored, and its MUSTs would be ignored
with it.

## What a PASS is worth

Two limits, both of which the tool states in its own output rather than
leaving you to discover:

- **A pass is not a proof for a rule with two right answers.** The default
  `text/plain` probe sends a body the runner must reject anyway, so a
  runner that wrongly parses `text/plain` still answers `400` and still
  passes. `--execute` is what makes that one conclusive. The self-test
  below asserts this rather than assuming it — the fault is planted, and
  the default probe is confirmed *not* to catch it.
- **`describe` being side-effect free cannot be observed from outside.**
  Two identical calls returning identical bytes is evidence, so a
  difference warns; it is not a breach this tool can stand behind.

Rules are checked against the specification's own
[`schemas/`](../../schemas) rather than restated in Python, so a schema and
this checker cannot drift. Running from a checkout uses that checkout's
schemas; an installed wheel carries a copy. Every report says which it
used.

## Proving the checks can fail

The failure mode of a conformance checker is a false green: every check
reads correctly, passes against a real implementation, and would have
passed just as happily against a runner that did none of it.

```console
$ python tools/conformance/selftest.py
postern-conformance self-test

  22 conformant baselines, none failing
  11 error codes, table agrees with the schema
  6 schemas, the build hook bundles each one
  2 declared formats, every one asserted
  3 stream shapes, each read to a bounded end
  39 planted faults, each caught by its own check

Every check can fail.
```

It runs the checker against a deliberately conformant fake runner, where
nothing may fail, and then against the same runner with exactly one rule
broken — asserting that the *named* check catches it, not merely that
something did. Standard library only; it needs no runner and no network.

## Why this is not called `postern`

[CONTRIBUTING.md](../../CONTRIBUTING.md) puts a language SDK out of scope,
and the reasoning holds: Postern is HTTP, any language can serve or call
it, and publishing a Python client under the specification's own name would
make one language the blessed one for a document whose whole claim is that
none is.

This is a test suite rather than a client library, and it deliberately
leaves the name a Python client would want free for whoever writes one
under their own.

## Changes

### 0.1.2

**Verdicts move in both directions.** A runner that reported clean under
0.1.1 may report findings here, and one that reported a finding may come
back clean. Almost all of it is checks that were narrower than the rule
they named — asked on one surface where the specification binds several —
so read a new finding the way 0.1.1 asked you to: as a rule that was always
there and is only now being checked. 10 conformant baselines became 22 and
24 planted faults became 39.

- **`idempotent_retry` moved from `describe.capabilities` to `status`.**
  §4.1 opens "`capabilities` describes the **agent**", and whether a repeat
  executes it a second time is a fact about the runner serving it. The
  checker reads `status` and deliberately does not fall back to the old
  location; a runner still declaring it under `capabilities` validates and
  means nothing by it.
- **§2's media type is checked on every body it binds**, not on `status`
  alone. `describe`, `run`'s success body and every error body went
  unasked — and an error body is the one most likely to get it wrong, since
  a failure path often leaves the serializer that would have set the header.
- **The §2.3 preflight rules are asked on both preflighting verbs.** A
  runner admitting `Content-Type` on `run` and not on `stream` passed,
  while no browser could stream from it at all.
- **§4.1.3 is read as written** — the *whole* `describe` response is
  scanned for a credential value, where the check had read the
  `credentials` block alone.
- **A stream's `start` and `done` must name the same run**, which
  `stream-event.schema.json` has always said and nothing checked.
- **§4.6 step 5 is a check a runner performs, not only one it orders.**
  Where `describe` declares a credential the environment does not carry, a
  conforming runner answers `424 missing_credential`.
- **`status.agent`, `status.entitlement` and `describe.output` are
  required**, and `status.agent` requires its own members — `{"agent": {}}`
  used to validate while `describe` required `id`, `name` and `version`.
- **A held-open stream is reported rather than hanging the report**, and
  the bounded read still sees what a runner sends after `done`.
- **Five checks a non-conformant runner was passing are closed**, and
  separately the checker stopped failing runners the specification permits.
- **A rebuilt wheel no longer ships the previous build's schemas.** A wheel
  built over an earlier build's artifacts validated against a specification
  nobody was reading — the one entry here that is a packaging fault rather
  than a check.

### 0.1.1

**A runner that reported clean under 0.1.0 may report findings here.** That
is the intent rather than a regression: the specification was clarified in
four places and the suite followed. Read a new finding as a rule that was
always there and is only now being checked.

- **§4.6 is checked at all.** A `run` missing a required input, sent to a
  runner whose environment is incomplete, earns both `bad_request` and
  `missing_credential`, and the specification did not order them — so this
  suite *skipped* §4.2's rule whenever a runner answered `424`, which is the
  ordinary state of one being brought up. §4.6 now orders them, and the
  `424` is a finding.
- **A reused `Idempotency-Key` carrying different inputs must be refused**,
  not replayed.
- **`run_id` is per execution**, quoting §4.2's own wording.
- **`capabilities.streaming` is withdrawn** and no longer checked.

### 0.1.0

First release. 5 conformant baselines, 22 planted faults.

## Status

The specification is a draft and nothing in it is stable yet, so neither is
this. It tracks Postern 0.1.

Apache-2.0, same as the specification.
