Metadata-Version: 2.4
Name: groundproof
Version: 0.1.0
Summary: Verify agent-reported claims against ground truth: git state, files, commands, tests, HTTP endpoints.
Project-URL: Homepage, https://github.com/blockhaven-ai/groundproof
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,ci,claims,ground-truth,verification
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: httpx>=0.24; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: http
Requires-Dist: httpx>=0.24; extra == 'http'
Description-Content-Type: text/markdown

# groundproof

Verify agent-reported claims against ground truth.

Automated agents (LLM coding agents, bots, scripted pipelines) report
outcomes: "I merged the branch", "the file now contains X", "tests pass",
"the PR is closed". These reports are sometimes wrong. groundproof takes a
structured claims file and checks each claim against the real system —
the git repository, the filesystem, a command's exit code, an HTTP
endpoint — and produces a per-claim verdict with captured evidence. It is
designed to run as a post-agent gate in CI.

groundproof performs no model calls and has no AI dependencies. Every
check is a direct observation of system state.

## Installation

```sh
pip install groundproof            # core (pyyaml only)
pip install "groundproof[http]"    # adds httpx for http.responds
```

Requires Python 3.10+.

## Usage

```sh
groundproof init                   # write a commented example claims.yaml
groundproof verify claims.yaml     # verify, print a table, set exit code
groundproof verify claims.yaml --report report.json --annotations
groundproof types                  # list registered claim types
```

Library API:

```python
from groundproof import verify_claims

report = verify_claims("claims.yaml")
for result in report.results:
    print(result.claim.id, result.verdict.value, result.reason)
print(report.exit_code())  # 0 / 1 / 2
```

## Claims file schema

A claims file is YAML or JSON: either a top-level list of claims, or a
mapping with a single `claims` key. Each claim:

| Key           | Required | Meaning                                      |
|---------------|----------|----------------------------------------------|
| `id`          | yes      | unique string identifying the claim          |
| `type`        | yes      | a registered verifier name (see catalog)     |
| `params`      | no       | mapping of verifier-specific parameters      |
| `description` | no       | free text carried into the report            |

```yaml
claims:
  - id: feature-merged
    type: git.branch_merged
    description: The widget-cache branch was merged
    params:
      branch: feature/widget-cache
      into: main
```

## Verdicts

Every claim gets exactly one of three verdicts:

- `verified` — the claim was checked and holds.
- `refuted` — the claim was checked and does not hold.
- `unverifiable` — the claim could not be checked: missing tool, absent
  repository, unreachable network, invalid parameters.

The third value exists so that "could not check" is never reported as a
pass. Each verdict carries an evidence object recording what was checked,
what was observed, and a timestamp.

## Verifier catalog

| Type                  | Params                                                       | Refuted when                          | Unverifiable when                         |
|-----------------------|--------------------------------------------------------------|---------------------------------------|--------------------------------------------|
| `git.commit_exists`   | `sha`, `repo?`                                               | commit absent                          | git missing, path not a repo               |
| `git.branch_merged`   | `branch`, `into`, `repo?`                                    | branch not an ancestor of `into`       | git missing, ref does not exist            |
| `git.file_changed_in` | `path`, `sha`, `repo?`                                       | path not touched by the commit         | git missing, commit absent                 |
| `fs.file_exists`      | `path`                                                       | path absent                            | bad params                                 |
| `fs.file_contains`    | `path`, one of `text` \| `regex`                             | text/regex not found, file absent      | unreadable file, bad regex                 |
| `fs.file_hash`        | `path`, `sha256`                                             | digest mismatch, file absent           | malformed digest, unreadable file          |
| `cmd.succeeds`        | `argv`, `cwd?`, `timeout?`                                   | nonzero exit code                      | executable missing, timeout                |
| `test.passes`         | `framework` (pytest), `node_id`, `cwd?`, `timeout?`          | test fails (pytest exit 1)             | node id not found, collection error        |
| `http.responds`       | `url`, `method?`, `expect_status?`, `expect_body_contains?`, `timeout?` | wrong status, body text missing | httpx not installed, network error |
| `github.pr_state`     | `repo` (owner/name), `number`, `state` (open\|closed\|merged) | state differs, PR not found            | `gh` CLI missing or errored                |

`cmd.succeeds` and `test.passes` capture the exit code and the tail of
stdout/stderr as evidence. `repo` and `cwd` default to the current
working directory. Without `expect_status`, `http.responds` accepts any
2xx status.

## Exit codes

| Code | Default                  | `--strict`               | `--lenient`              |
|------|--------------------------|--------------------------|--------------------------|
| 0    | all verified             | all verified             | none refuted             |
| 1    | any refuted              | any refuted or unverifiable | any refuted           |
| 2    | any unverifiable (none refuted), or claims file malformed | claims file malformed | claims file malformed |

A refuted claim fails the run in every mode.

## CI usage

```yaml
# .github/workflows/groundproof.yml
name: groundproof
on: [pull_request]

jobs:
  verify-claims:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # git claims need history
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "groundproof[http]"
      - run: groundproof verify claims.yaml --strict --annotations --report report.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: groundproof-report
          path: report.json
```

`--annotations` emits GitHub Actions `::error` / `::warning` workflow
commands so refuted and unverifiable claims appear inline on the run.

## Extracting claims from agent output

Agents can embed claims directly in their markdown output in a fenced
block whose info string is `claims`:

````markdown
Work complete.

```claims
- id: branch-merged
  type: git.branch_merged
  params: {branch: feature/widget-cache, into: main}
```
````

`groundproof.extract.extract_claims(text)` finds these blocks, parses
them as YAML, and validates them through the normal schema. The
extraction is purely mechanical text processing.

```python
from groundproof import verify_claims
from groundproof.extract import extract_claims

claims = extract_claims(agent_markdown)
report = verify_claims(claims)
```

## Writing a custom verifier

A verifier is a callable taking the claim's `params` mapping and
returning a `VerificationResult`. Use the helpers to build verdicts and
attach evidence as keyword arguments:

```python
# myorg_pkg/verifiers.py
from groundproof.schema import VerificationResult, refuted, unverifiable, verified


def ticket_closed(params) -> VerificationResult:
    """Check that a tracker ticket is closed. Params: ticket_id."""
    ticket_id = params.get("ticket_id")
    if not isinstance(ticket_id, str):
        return unverifiable("invalid params: 'ticket_id' must be a string")
    status = my_tracker_lookup(ticket_id)   # your integration
    if status is None:
        return unverifiable(f"tracker unreachable for {ticket_id}")
    if status == "closed":
        return verified(f"{ticket_id} is closed", observed_status=status)
    return refuted(f"{ticket_id} is {status}", observed_status=status)
```

Register it via an entry point so groundproof discovers it when your
package is installed:

```toml
# pyproject.toml of your package
[project.entry-points."groundproof.verifiers"]
"myorg.ticket_closed" = "myorg_pkg.verifiers:ticket_closed"
```

The claim type `myorg.ticket_closed` is then usable in any claims file,
and appears in `groundproof types`. Entry points cannot shadow built-in
verifier names. For programmatic use without packaging, register on a
`Registry` instance and pass it to `verify_claims(claims, registry=...)`.

Conventions for verifiers:

- Return `unverifiable` for anything that prevents the check itself
  (missing tool, bad params, network failure). Never raise for expected
  failure modes; an uncaught exception is converted to `unverifiable`.
- Return `refuted` only when the system was actually observed and the
  claim does not hold.
- Put the observed values in evidence so reports are auditable.

## License

Apache-2.0. See [LICENSE](LICENSE).
