Metadata-Version: 2.4
Name: proofline
Version: 0.1.1
Summary: Verifiable run bundles for reproducible AI and agent execution
Author: Powfu
License: MIT
Project-URL: Homepage, https://github.com/Powfu-zwx/proofline
Project-URL: Repository, https://github.com/Powfu-zwx/proofline
Project-URL: Issues, https://github.com/Powfu-zwx/proofline/issues
Keywords: llm,agents,reproducibility,provenance,evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Security
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: jsonschema>=4.21; extra == "dev"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Dynamic: license-file

# Proofline

[![CI](https://github.com/Powfu-zwx/proofline/actions/workflows/ci.yml/badge.svg)](https://github.com/Powfu-zwx/proofline/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)

Proofline is a model-agnostic protocol and reference implementation for verifiable AI runs.

![proofline demo: verify a bundle, diff two runs, catch a prompt change](docs/demo.gif)

A run bundle is a single JSON document that records the inputs, code revision, model/tool steps, outputs, costs, redactions, and hashes needed to replay, diff, and audit LLM or agent executions. The core is a versioned schema; SDKs, storage backends, and framework adapters are replaceable layers around it.

## Why

- **Regression testing.** `proofline diff` compares two runs semantically. Run ids, timestamps, and derived digests never show up as noise; what changed in inputs, outputs, and costs does. See [CI regression gates](docs/ci-regression.md) for the end-to-end recipe.
- **Audit and forensics.** Bundles are tamper-evident: a stable SHA-256 digest covers everything that can affect replay decisions, and `proofline verify` re-checks every stored hash, redaction path, and secret pattern.
- **Portability.** A bundle is one JSON file with a published [schema](schemas/run.schema.json) and [spec](spec/run-bundle-v0.1.md). No server, no vendor lock-in.

## Non-goals

- Not another chat UI or agent framework.
- Not a model provider or prompt registry.
- Not a claim that reruns are bit-identical when the underlying model or tools are nondeterministic.

## Install

```bash
git clone https://github.com/Powfu-zwx/proofline.git
cd proofline
python -m pip install -e .
```

## Quickstart

Record the same command twice, then prove the runs are semantically identical:

```bash
proofline run --out artifacts/a.run.json -- python examples/code_fix_agent.py
proofline run --out artifacts/b.run.json -- python examples/code_fix_agent.py

proofline verify artifacts/a.run.json
# OK artifacts/a.run.json

proofline diff artifacts/a.run.json artifacts/b.run.json
# no semantic differences
```

Timestamps and run ids differ between the two bundles, but both are excluded from the stable digest and from semantic diffs, so identical work produces an empty diff.

## SDK

```python
from proofline import RunRecorder

with RunRecorder(out_path="artifacts/demo.run.json") as recorder:
    with recorder.step("model", "draft", input={"prompt": "hi"}) as step:
        step["output"] = {"text": "ok"}
        step["cost"] = {"input_tokens": 3, "output_tokens": 1}
```

Secrets are redacted before anything touches disk: keys like `api_key` / `credentials` and values like `sk-...`, `AKIA...`, `Bearer ...`, or JWTs are replaced with `[REDACTED]`, and each redaction site is recorded as a JSON Pointer in the bundle.

See `examples/rag_citation_check.py` and `examples/code_fix_agent.py` for full agent-shaped runs.

## Integrations

### OpenAI

```bash
python -m pip install -e ".[openai]"
```

```python
from openai import OpenAI
from proofline import RunRecorder
from proofline.openai import wrap

with RunRecorder(out_path="artifacts/openai.run.json") as recorder:
    client = wrap(OpenAI(), recorder)
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "ping"}],
    )
```

Each `chat.completions.create` call is recorded as a `model` step with the request as input, the response as output, and token usage as cost. Streaming calls are recorded too: the accumulated text is stored together with a `truncated` flag, and a failed request records an `error` step. `AsyncOpenAI` clients are wrapped by the same `wrap()` call. Run `examples/openai_chat.py` for an end-to-end recorded call.

### Anthropic

```bash
python -m pip install -e ".[anthropic]"
```

```python
from anthropic import Anthropic
from proofline import RunRecorder
from proofline.anthropic import wrap

with RunRecorder(out_path="artifacts/anthropic.run.json") as recorder:
    client = wrap(Anthropic(), recorder)
    client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=256,
        messages=[{"role": "user", "content": "ping"}],
    )
```

`messages.create` is recorded the same way, including `stream=True` event iteration, and `AsyncAnthropic` clients are wrapped by the same `wrap()` call. Run `examples/anthropic_chat.py` for an end-to-end recorded call.

## Bundle anatomy

```json
{
  "schema_version": "0.1",
  "run_id": "6f0c0f1e-…",
  "created_at": "2026-08-11T15:00:00.000Z",
  "actor": {"type": "human+agent", "name": "powfu", "version": "0.1.1"},
  "project": {"name": "proofline", "revision": "9dd5f0a…", "dirty": false},
  "invocation": {"argv": ["python", "agent.py"], "cwd": "…", "env_keys": ["PATH"], "python": "3.11.15"},
  "steps": [
    {
      "step_id": "step-1",
      "kind": "model",
      "name": "draft",
      "status": "ok",
      "started_at": "…",
      "ended_at": "…",
      "input": {"prompt": "hi"},
      "output": {"text": "ok"},
      "error": null,
      "cost": {"input_tokens": 3, "output_tokens": 1},
      "metadata": {},
      "input_digest": "…",
      "output_digest": "…"
    }
  ],
  "redactions": [],
  "metadata": {},
  "bundle_digest": "…"
}
```

## Core invariant

A bundle is portable evidence. Any field that cannot affect replay decisions, such as wall-clock timestamps or a fresh run id, is excluded from the stable digest and from semantic diffs. `bundle_digest` is SHA-256 over canonical JSON of the bundle with volatile fields removed; see the [spec](spec/run-bundle-v0.1.md) for the exact normalization and verification rules.

## FAQ

**How is this different from LangSmith, Langfuse, or other tracing platforms?**
Those are observability platforms: hosted dashboards for exploring traces at scale. Proofline is an evidence format: a single verifiable JSON file you can commit to a repo, diff in CI, attach to an incident report, or hand to an auditor. No server, no account, no SDK lock-in. If you already run a tracing platform, proofline is complementary — it is the artifact you keep when a specific run has to be provable.

**Is a bundle proof that the model would answer the same way again?**
No, and the spec is explicit about this non-goal. A bundle proves what was sent, what came back, what it cost, and that nobody altered the record afterwards. Determinism is your pipeline's job; the [CI recipe](docs/ci-regression.md) shows how to get there where it matters.

**Does redaction make bundles safe to share?**
Redaction is pattern-based and best-effort — it catches well-known key names and token formats before anything touches disk, and `verify` re-scans as a second line of defense. It is not a guarantee; review bundles like any fixture before publishing them. See [SECURITY.md](SECURITY.md) for the exact boundary.

**When should I not use proofline?**
If you want live dashboards, sampling analytics, or fleet-wide monitoring, use a tracing platform. If your pipeline has no decisions worth auditing or regressing, a bundle is overhead. Proofline earns its keep where runs are consequential: agents that touch code, money, or user data, and pipelines whose behavior changes must be caught in review.

## Development

```bash
python -m pip install -e ".[dev]"
ruff check src tests examples
pytest -q
```

## License

[MIT](LICENSE)
