Metadata-Version: 2.4
Name: conformance
Version: 0.1.0
Summary: Verify that AI agent runtimes hold nine operational invariants, under injected faults, with reproducible evidence
Keywords: agents,verification,determinism,conformance,testing,llm
Author: Conformance
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Dist: conformance-harness==0.1.0
Requires-Dist: textual>=8.2.8,<9
Requires-Python: >=3.12
Project-URL: Changelog, https://github.com/RohitBind123/conformence_evaluation/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/RohitBind123/conformence_evaluation/blob/main/docs/14_Onboarding_Walkthrough.md
Project-URL: Homepage, https://github.com/RohitBind123/conformence_evaluation
Project-URL: Issues, https://github.com/RohitBind123/conformence_evaluation/issues
Project-URL: Source, https://github.com/RohitBind123/conformence_evaluation
Description-Content-Type: text/markdown

# Conformance

**Verify what your AI agent *did* — not what it said.**

Conformance injects deliberate faults at your agent's effect boundary
(a redelivered queue event, a crash mid-write, a withheld approval, a
raced turn), records what actually happened, and evaluates nine
deterministic invariants over that record. No language model judges
anything. Every finding comes with a content-addressed witness a
sceptic can re-derive offline, without you.

```
$ conformance verify --scenario shipbot --seed 17

seed 17 · spec conformance-1.0 · 8 schedule(s) run

✕ 1 custody                  VIOLATED    wit_05b69e04ec177d5209056ab82d
⚠ 2 one-advance              inconclusive (no attachment point for staged_race…)
⚠ 3 no-scarce-hold           inconclusive (no pool acquisitions in record…)
✕ 4 quarantine               VIOLATED    wit_5fde8bf97de2a976d8a57fbd21
✕ 5 exactly-once-effect      VIOLATED    wit_d223f1f2cfec1b7eca4e89552a
⚠ 6 plan-identity            inconclusive (needs its own recording…)
⚠ 7 timeout-equals-cancel    inconclusive (no attachment point for delay_response…)
⚠ 8 structural-authority     inconclusive (no attachment point for withhold_approvals…)
⚠ 9 state-separation         inconclusive (no database schema discoverable…)

Result
  Level Achieved: —
  Target:         L1 (DURABLE)
  Depth:          V2 (DETERMINISTIC)
  Gate:           FAIL
```

Every invariant returns `hold`, `violated`, or `inconclusive`.
`inconclusive` is never styled as a pass and never counts toward a
level: unmeasured is not passed. That is the whole design.

## Install

```bash
pip install conformance          # or: uv tool install conformance
```

Python 3.12+. No account, no API key for verification, nothing uploaded.

## Start where you are

```bash
conformance init                      # detect the stack, find effect sites, run static checks
conformance doctor --explain-depth    # what each invariant needs to be decidable
conformance verify --path .           # V0 verdicts, no instrumentation required
```

`init` needs no SDK, no credentials, and no network. It reports what it
found and — for anything it cannot decide at this depth — exactly what
would decide it.

## Going deeper: record once, verify forever

Declare a scenario naming an entrypoint that runs one bounded episode
of your agent, then:

```bash
conformance record --scenario myagent --seed 1   # your key, your machine, once
conformance verify --scenario myagent --seed 1   # sealed, offline, byte-deterministic
```

Your credentials are used exactly once. Everything after that runs
against the captured cassette: CI needs no model key and no network. A
cassette miss is a loud failure (exit 3) — never a live call, never a
guess.

Same commit, same seed, same cassette produces identical verdict bytes
and identical witness ids: 1,000 out of 1,000 runs on the crash
schedules, Wilson 95% CI ≤ 0.38% flake.

## Checking a finding

A violation produces a witness. Export it and it becomes a file
anybody can check, offline, without trusting this tool:

```bash
conformance export <witness-id>
conformance replay <witness-id>       # re-derives the verdict from the evidence
```

Editing the evidence breaks its content address and replay refuses.

## In CI

```yaml
- run: pip install conformance
- run: conformance ci --scenario myagent --seed 1 --level-target L1
```

A violation of an invariant your target level requires fails the check.
Exit codes: 0 pass, 1 gate fail, 2 cannot execute, 3 fail-closed breach.

## Depth, honestly

| Depth | What it needs | What it decides |
|---|---|---|
| V0 | nothing | discovery, configuration, invariant 9 |
| V2 | SDK attached + a recorded cassette | all nine, under fault |

Most systems start and stay useful at V0. Moving to V2 means
instrumenting the effect boundary — usually a single keyword argument
if your effects are HTTP. `conformance doctor --scenario <name>
--explain-depth` lists the exact step per invariant.

## Documentation

- [Onboarding walkthrough](https://github.com/RohitBind123/conformence_evaluation/blob/main/docs/14_Onboarding_Walkthrough.md)
  — install to sealed verdict
- [Worked examples](https://github.com/RohitBind123/conformence_evaluation/tree/main/examples)
  — manual, HTTP, LangGraph, and CI
- [CLI reference](https://github.com/RohitBind123/conformence_evaluation/blob/main/docs/09_CLI_Specification.md)
- [The specification](https://github.com/RohitBind123/conformence_evaluation/blob/main/docs/spec/conformance-1.0.md)
- [CHANGELOG](https://github.com/RohitBind123/conformence_evaluation/blob/main/CHANGELOG.md)
  — including what is knowingly incomplete

Apache-2.0.
