Metadata-Version: 2.4
Name: verdictkit
Version: 0.1.0
Summary: Non-custodial verification for agent work: did the agent do what it said?
Author: agentbuilt
License: MIT
Project-URL: Homepage, https://github.com/mattedwardseo/verdictkit
Project-URL: Documentation, https://github.com/mattedwardseo/verdictkit#readme
Project-URL: Repository, https://github.com/mattedwardseo/verdictkit
Project-URL: Issues, https://github.com/mattedwardseo/verdictkit/issues
Keywords: agents,verification,ai-safety,auditing,deterministic-checks
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.28
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build; extra == "dev"

# verdictkit

Non-custodial verification for agent work. Answers one question: **did the
agent do what it said it would?**

The product is the *verification layer*, not custody: a machine-readable
contract (intended action + checkable success criteria), deterministic
checks against the world, and a rubric verdict — `PASS` / `FLAG` / `FAIL`
— with evidence. No money moves through it, so there is no money-transmitter
licensing wall. That is the whole strategic point.

**Status: 0.1.0, launch-ready, NOT published to PyPI.** Do not run the
publish command without approval.

## Quickstart

```bash
pip install verdictkit        # not on PyPI yet; use: pip install .
```

### Local verification (free, unlimited, no key, no network)

```python
import verdictkit as vk

contract = vk.define_contract(
    task_id="research-042",
    agent_id="agent-7f3a",
    intended_action="Summarize the Q3 incident reports into incidents.md",
    success_criteria=[
        {"id": "c1", "type": "file_exists",
         "path": "incidents.md",
         "description": "summary file was written"},
        {"id": "c2", "type": "pattern_count_gte",
         "path": "incidents.md",
         "pattern": r"^## Incident",
         "min_count": 3,
         "description": "covers at least 3 incidents"},
    ],
    unverifiable_claims=["tone is executive-appropriate"],  # -> FLAG, honestly
)

result = vk.verify_run(contract, base_path="./output")
print(result["verdict"])   # PASS | FLAG | FAIL
print(result["reason"])    # evidence, always
```

Or from the shell:

```bash
verdictkit verify contract.json --base-path ./output -v
# exit code: 0 = PASS, 1 = FAIL, 2 = FLAG, 3 = usage/IO error
```

### Hosted verification (reputation ledger + re-verification)

Local mode checks *your* machine. The hosted tier lets a third party trust
the verdict: you capture evidence locally, the server **re-runs the
deterministic checks against the evidence** — it never trusts the agent's
claims alone.

```python
import verdictkit as vk

client = vk.Client(api_key="vk_live_...")          # base_url=... to override
contract_id = client.create_contract(contract)      # -> "vc_..."
bundle = vk.collect_evidence(contract, base_path="./output")
verdict = client.submit_run(contract_id, bundle)    # server-side verdict dict

client.get_verdict(contract_id)                     # latest verdict
client.list_verdicts(agent_id="agent-7f3a")         # reputation ledger
```

Errors are clean: `vk.AuthError` on bad credentials (401/403),
`vk.VerdictkitError` on network trouble, timeouts, or bad responses.

## API reference

| Symbol | Kind | Description |
|---|---|---|
| `vk.define_contract(task_id, agent_id, intended_action, success_criteria, scope=None, unverifiable_claims=None, principal="unknown")` | function | Build a validated contract dict. Raises `ValueError` on unknown check types or criteria missing `id`/`type`. |
| `vk.verify_run(contract, base_path=None)` | function | Run all checks locally, return verdict dict. No key, no network. `base_path` resolves relative criterion paths. |
| `vk.Verdict` | class | Constants `PASS`, `FLAG`, `FAIL`. |
| `vk.collect_evidence(contract, base_path=None)` | function | Capture a content-addressed evidence bundle (file bytes + sha256, dir listings, parsed JSON, sqlite row counts). Local only, no network. |
| `vk.verify_bundle_hash(bundle)` | function | Recompute `bundle_hash`; `False` means the bundle was tampered with. |
| `vk.Client(api_key, base_url=..., timeout=30.0)` | class | Hosted API client. `create_contract(contract) -> contract_id`, `submit_run(contract_id, evidence_bundle) -> verdict dict`, `get_verdict(contract_id) -> dict`, `list_verdicts(agent_id=None, limit=100) -> list`. Raises `ValueError` without an API key. |
| `vk.VerdictkitError` | exception | Base error: network, timeout, bad server response. |
| `vk.AuthError(VerdictkitError)` | exception | 401/403 from the API. |

### Verdict shape

```python
{"contract_id": ..., "task_id": ..., "agent_id": ...,
 "verdict": "PASS" | "FLAG" | "FAIL",
 "reason": "human-readable evidence summary",
 "check_results": [{"check_id": ..., "type": ..., "status": "pass"|"fail"|"error",
                    "evidence": ...}, ...],
 "unverifiable_claims": [...]}
```

### Check types

`file_exists` · `file_nonempty` · `file_contains` (regex) ·
`pattern_count_gte` · `dir_exists` · `json_valid` · `line_count_gte` ·
`sqlite_table_has_rows` · `path_absent` (scope guard)

## The rubric

- **FAIL** — any check fails. Fail dominates everything else.
- **FLAG** — no failures, but something couldn't be verified: a check
  errored, a claim was declared unverifiable, or there was nothing
  checkable at all.
- **PASS** — every check passed cleanly, and nothing was left unverifiable.

Design rule: a check that cannot run is an *error*, never a pass. A run
with nothing checkable can at best be *flagged*. Absence of evidence is
never evidence of success.

## Trust model

1. **Contracts are machine-readable promises.** What the agent intended to
   do, written before the run, with deterministic success criteria.
2. **Evidence is content-addressed.** `collect_evidence` captures file
   bytes with their sha256, directory listings, and query results. The
   bundle carries a `bundle_hash` over canonical JSON — the server
   recomputes every hash; tampering breaks them.
3. **The server re-runs checks; it never trusts claims.** Hosted
   verification executes the same deterministic check engine against the
   bundle's content. Claims without evidence stay in
   `unverifiable_claims` and can only produce FLAG.
4. **Truncation is honest.** Artifacts are capped (256 KiB file content,
   2000 dir entries). Truncated artifacts are marked, and checks the
   server cannot fully re-run become FLAG — never PASS.

## What verification is NOT

- **Not a guarantee of quality.** PASS means the deterministic criteria
  were met — the file exists, has ≥3 sections, parses as JSON. It says
  nothing about whether the writing is good, the code is correct, or the
  summary is accurate. Put anything subjective in `unverifiable_claims`
  and accept the FLAG honestly.
- **Not an oracle.** Checks only see what they can read: files, dirs,
  sqlite. They cannot verify "the user is happy" or "the bug is fixed"
  unless that was operationalized into a checkable artifact.
- **Not custody, not escrow.** No money moves through verdictkit; it
  attests to work done, it does not hold funds or release payment.
- **Not tamper-proof on a compromised machine.** Local verification trusts
  the local filesystem. The hosted tier raises the bar (content-addressed
  bundles, server-side re-runs, reputation ledger) but a fully
  compromised agent host can still fake the inputs. Verification makes
  lying *auditable*, not impossible.

## Development

```bash
pip install -e ".[dev]"   # dev extras: pytest, build
python3 -m pytest         # 59 tests, all local, no network
python3 -m build          # wheel/sdist -- DO NOT upload without approval
```

## License

MIT. See `LICENSE`.
