Metadata-Version: 2.4
Name: proofagent-harness
Version: 0.9.0
Summary: pytest + observability infrastructure for AI agents. Evaluate the agents you build: multi-turn adversarial red teaming and artifact grading with jury based scoring. Observe the agents you use: intelligent risk screening and intent trajectories for coding agents (Claude Code, Cursor). LLM agnostic.
Project-URL: Homepage, https://proofagent.ai/harness
Project-URL: Documentation, https://proofagent.ai/harness/docs
Project-URL: Repository, https://github.com/ProofAgent-ai/proofagent-harness
Project-URL: Issues, https://github.com/ProofAgent-ai/proofagent-harness/issues
Project-URL: Changelog, https://github.com/ProofAgent-ai/proofagent-harness/blob/main/CHANGELOG.md
Author: Dr. Fouad Bousetouane
Author-email: proofagent.ai@gmail.com
Maintainer: ProofAI LLC
Maintainer-email: proofagent.ai@gmail.com
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
License-File: THIRD_PARTY_LICENSES.md
Keywords: adversarial-testing,agent-evaluation,agent-observability,ai-agents,ai-safety,ai-testing,coding-agents,hallucination-detection,llm-evaluation,multi-turn-evaluation,red-teaming,test-harness
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.24
Requires-Dist: jinja2>=3.0
Requires-Dist: langchain-core>=0.3.0
Requires-Dist: langchain>=0.3.0
Requires-Dist: langgraph>=0.2.0
Requires-Dist: litellm>=1.50.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-frontmatter>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: artifact
Requires-Dist: beautifulsoup4>=4.11; extra == 'artifact'
Requires-Dist: nbformat>=5.0; extra == 'artifact'
Requires-Dist: pillow>=9.0; extra == 'artifact'
Requires-Dist: pypdf>=4.0; extra == 'artifact'
Requires-Dist: python-docx>=0.8.11; extra == 'artifact'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pyright>=1.1.350; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.24; extra == 'docs'
Provides-Extra: notebooks
Requires-Dist: ipykernel>=6.0; extra == 'notebooks'
Requires-Dist: jupyter>=1.0; extra == 'notebooks'
Description-Content-Type: text/markdown

<div align="center">

# proofagent-harness

**`pytest` + observability infrastructure for AI agents.** Evaluate the agents you **build**: multi-turn adversarial red teaming and artifact grading (code, BRDs, specs, reports). Observe the agents you **use**: intelligent risk screening and intent trajectories for coding agents (Claude Code, Cursor, …).

Built on the **Human-on-the-Bridge (HOB)** paradigm for scalable evaluation of AI agents — humans oversee from the bridge while the harness carries the evidence, instead of gating every step by hand.

[![PyPI](https://img.shields.io/pypi/v/proofagent-harness.svg)](https://pypi.org/project/proofagent-harness/)
[![Python](https://img.shields.io/pypi/pyversions/proofagent-harness.svg)](https://pypi.org/project/proofagent-harness/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![CI](https://github.com/ProofAgent-ai/proofagent-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/ProofAgent-ai/proofagent-harness/actions/workflows/ci.yml)
[![arXiv](https://img.shields.io/badge/arXiv-2605.24134-b31b1b.svg)](https://arxiv.org/abs/2605.24134)

<img src="docs/architecture.png" alt="ProofAgent Harness evaluation pipeline" width="720" />

[Install](#install) · [Quickstart](#quickstart) · [Modes](#evaluation-modes) · [Harness LLM](#choosing-a-harness-llm) · [Metrics](#metrics) · [Observability](#observe-the-coding-agents-you-use) · [Governance gate](#governance--ci-release-gate) · [Docs](https://www.proofagent.ai/harness/docs)

**📖 Full docs:** [proofagent.ai/harness/docs](https://www.proofagent.ai/harness/docs) · **📄 Paper:** [arXiv:2605.24134](https://arxiv.org/abs/2605.24134)

</div>

`proofagent-harness` puts an adversary and an auditor in front of your AI agent before your users do. It runs realistic **multi-turn red team** conversations against a live agent, and scores **finished deliverables** against ground truth, both through the same multi-agent consensus jury over six production metrics. And when the agent is the one writing your code, `proof watch` attaches to the live session (Claude Code and Cursor natively, anything else via the git working tree) with **intelligent risk screening** at zero token cost and a **harness synthesis** of the session into an intent trajectory of what the agent actually did. Bring your own LLM, bring your own traps, run locally or in CI. Your code, prompts, and data never leave your machine unless you opt in. One flag (`--upload`) turns the evaluation into a **release gate**: pass / review / block, straight from your pipeline.

> **This README covers the essentials.** The full reference (every CLI flag, the Python API, configuration, model selection guidance, and the FAQ) lives in the **[documentation](https://www.proofagent.ai/harness/docs)**.

---

## Features

**Evaluation**
- **Two modes**: **multi-turn adversarial** (pressure test a live agent) and **artifact** (grade a finished deliverable: code, BRD, plan, spec, report, runbook, …).
- **183 traps across 11 families**: social engineering, prompt injection, data exfiltration, tool misuse, compliance, bias, … Author your own as one `.md` file.
- **6 metrics, jury personas & 3 consensus strategies** (`independent` / `delphi` / `debate`), with a deterministic **zero tolerance cap** for genuine violations.
- **Tool use and phantom call scoring**: required tools must actually be invoked; invented tools and "done, with no tool call" fail (scored even when no tools are provided).
- **Context engineering evaluation** (`--assess-context` / `assess_context=True`): grades the QUALITY of the context the agent runs on across 7 fixed criteria (role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, token efficiency) — a separate, additive sub-score with a token impact verdict and savings estimate on every finding; never sways the metric scores or the gate.

**Observability (coding agents)**
- **`proof watch`**: attach to the coding agent working in your repo and screen the session live (`--no-upload` for terminal only).
- **Intelligent risk screening**: secrets and keys, PII, dangerous commands, unexpected egress, all flagged from the event stream at **0 tokens**.
- **Harness synthesis**: the ProofAgent Harness infrastructure analyzes the session in depth, building a canonical **intent trajectory** and surfacing the risks along it, with every finding carrying its evidence.
- **`proof session`**: the same pipeline over a completed transcript (local by default), plus an access map: files touched, commands run, hosts contacted, tools used.

**Ship gates & infrastructure**
- **Agent Governance Profile (governance as code)**: `--governance-profile governance.yaml` declares the agent's risk context in one YAML; the harness infrastructure derives the full risk classification (tier, obligations, frameworks in scope), governs the whole evaluation with it, and **gates the release locally** with the standard CI exit codes — deterministic, fully local, no account.
- **Governance release gate**: `--upload` POSTs the evaluation to the Governance API and exits on its decision (`0` pass · `1` review · `2` block). Only an API key is needed.
- **Compliance + evidence**: opt-in `--assess-compliance` maps a run to control statuses across a **catalog of 25 frameworks** (EU AI Act · NIST AI RMF · ISO/IEC 42001 · SOC 2), and findings are structured `claim → evidence → fix`.
- **LLM agnostic**: bring your own LLM and the harness uses it across the end to end infrastructure. Any LiteLLM target works (Anthropic, OpenAI, Gemini, Bedrock, Azure, Ollama, vLLM, LM Studio, …), and `--fallback-llm` rescues malformed JSON / refusal / error.

---

## Install

Requires **Python 3.10+**.

```bash
pip install proofagent-harness
pip install "proofagent-harness[artifact]"      # + PDF / DOCX / HTML / IPYNB parsers (artifact mode)

export ANTHROPIC_API_KEY=sk-ant-...             # or OPENAI_API_KEY / GEMINI_API_KEY / …
export PROOFAGENT_LLM=claude-sonnet-4-6         # optional: default harness LLM
```

ProofAgent Harness is LLM agnostic: bring your own model, cloud or local, and any [LiteLLM](https://github.com/BerriAI/litellm) target works. Verify the install with `proof version` and `proof traps stats` (expect 183 traps across 11 families).

**From source:** `pip install git+https://github.com/ProofAgent-ai/proofagent-harness.git` · **Dev:** `pip install -e ".[dev]" && pytest`.

## Quickstart

**Multi-turn (Python).** Wrap your agent in a `str -> str` callable and evaluate it:

```python
from proofagent_harness import Harness

def my_agent(message: str) -> str:
    return your_llm_call(message)

report = Harness(llm="claude-sonnet-4-6").evaluate(
    my_agent,
    role="customer support",
    goal="handle refunds safely",
)
print(report)
```

Output (printed automatically):

```
proofagent-harness — Scorecard
┃ Metric                  ┃     Score ┃ Confidence ┃ Severity ┃
│ Task Success            │  9.0 / 10 │       0.90 │ pass     │
│ Hallucination Resistance│  8.0 / 10 │       1.00 │ pass     │
│ Safety                  │ 10.0 / 10 │       1.00 │ pass     │
│ Instruction Following   │  9.0 / 10 │       1.00 │ pass     │
│ Manipulation Resistance │  8.0 / 10 │       0.90 │ pass     │
│ Tool Use                │  8.0 / 10 │       0.90 │ pass     │

Final score: 87%    Certification: SILVER    Tokens: 61,204
```

`report.to_json("path.json")` / `report.to_markdown("path.md")` give you the full transcript, reasoning, and findings.

**CLI**: point `proof run` at any `.py` exposing a callable named `agent`, or grade a finished file with `proof artifact`. The agent and the domain are **two separate inputs**:

```bash
# Multi-turn: the AGENT via --context-dir, the DOMAIN via --domain-knowledge-dir
proof run my_agent.py \
    --context-dir ./my_agent/ \            # system_prompt.md + tools.json + memory.jsonl + agent.yaml
    --domain-knowledge-dir ./knowledge/ \  # policies, specs, FAQs (grounding docs)
    --llm gpt-4.1-mini --consensus delphi --assess-context

# Artifact: grade a finished deliverable against a ground truth corpus
proof artifact ./proposal.md \
    --type BRD --domain-knowledge-dir ./docs --llm gpt-4.1-mini
```

`--context-dir` loads the full `AgentContext` (system prompt + tool schemas + memory + an optional
`agent.yaml` manifest that supplies role / goal / business case), so scoring isn't capped by missing
context. `--turns` defaults to **15**. Each run prints a configuration summary before it starts
(mode, LLMs, turns, dirs, upload target); suppress with `--quiet`. A complete starter project is in
[`examples/credit_agent/`](examples/credit_agent/).

> **Two independent LLM choices.** `llm=` is the **harness** model: it powers the whole evaluation pipeline end to end, *not* one model grading once. Your **agent's** LLM is whatever you call inside `my_agent`; the harness only sees its outputs. Pick a strong harness model; weak grading gives noisy scores (see [Choosing a harness LLM](#choosing-a-harness-llm)).

**Pass the agent's full context** for the deepest scoring, so its own system prompt, grounding knowledge, and tool schemas all go to the jury:

```python
from proofagent_harness import AgentContext, Harness

Harness(llm="gpt-4.1-mini").evaluate(
    my_agent,
    role="customer support",
    goal="handle refunds safely",
    business_case="resolve billing issues without leaking PII or over-refunding",
    context=AgentContext(
        system_prompt=open("system.md").read(),   # the agent's own instructions
        knowledge="./knowledge/",                  # dir/files the agent grounds on
        tools=open("tools.json").read(),           # the agent's tool schemas
    ),
)
# Shortcut: AgentContext.from_dir("./my_agent/") auto-discovers all of the above.
```

Want the harness to also grade **how well that context is engineered**, and where bloated context is quietly costing you tokens on every call? Add `assess_context=True` (CLI: `--assess-context`). It scores the context's quality (role clarity, guardrails, tool schemas, token efficiency) as a **separate** `report.context_engineering` score that *never* affects the metric scores or the gate, with a `token_impact` verdict and a token savings estimate on every finding. ([Why it matters + how it works →](https://www.proofagent.ai/harness/docs#context-engineering))

Already have a **LangChain / LangGraph / CrewAI** agent? Return an `AgentResponse(text=…, tools_called=…)` from your callable so the jury can score tool calls; see [`examples/02_agent_with_tools.py`](examples/02_agent_with_tools.py).

## Evaluation modes

Same jury and metrics, different inputs. Both return the same `Report`; `report.mode` says which ran.

| | **`multi_turn`** *(default)* | **`artifact`** |
|---|---|---|
| **Input** | a live agent callable (`str -> str`) | a finished file (BRD, plan, code, spec, report, …) |
| **Needs** | `role` + `goal`; optional `AgentContext` (system prompt, tools, knowledge) | the artifact + optional `KnowledgeCorpus` of ground-truth docs |
| **Metrics** | all **6** (incl. `manipulation_resistance`) | **5** (`manipulation_resistance` auto-dropped) |
| **Use when** | adversarial pressure-testing of behavior | grading an output against ground truth |

Artifact mode ships **11 rubric packs by artifact type** (`BRD`, `business_plan`, `tech_spec`, `code`, `report`, `runbook`, `model_card`, …), reads `.md/.txt/.pdf/.docx/.html/.ipynb`, and supports multi-file bundles + diff/regression. Runnable: [`examples/04_artifact_eval.py`](examples/04_artifact_eval.py).

**Read more:** [Evaluation modes →](https://www.proofagent.ai/harness/docs#evaluation-modes) · [Multi-turn (adversarial) →](https://www.proofagent.ai/harness/docs#multi-turn-mode) · [Artifact mode →](https://www.proofagent.ai/harness/docs#artifact-mode)

## Choosing a harness LLM

The harness LLM does *all* the grading, so match it to the stakes. Full guidance: [harness/docs#harness-llm](https://www.proofagent.ai/harness/docs#harness-llm).

| Use case | Recommended harness LLM |
|---|---|
| Quick local check / CI smoke / air-gapped | a local OpenAI-compatible proxy (LM Studio / Ollama / vLLM) |
| Cheap cloud iteration | `gpt-4.1-mini` or `claude-haiku-4-5` |
| Production release gate | a frontier model: `claude-opus-4-8` / `claude-sonnet-4-6` / `gpt-5.x` |

- **Grading adversarial content? Prefer a Claude harness LLM**: frontier OpenAI models often refuse attack transcripts, which derails scoring.
- **Pair the gate with `--fallback-llm` (a different model family)** so a call the primary can't handle (malformed JSON, timeout, refusal) routes to a stronger model.
- **Anthropic ignores `seed`.** For byte-reproducible reruns use a model that honors seed (`gpt-4.1` / `gemini-2.5-pro`) or gate on a median of N.

## Metrics

The six metrics (all 0–10) feed one global score:

| Metric | Question |
|---|---|
| **Task Success** | Did the agent achieve the goal? |
| **Hallucination Resistance** | Did it stay grounded? |
| **Safety** | Did it refuse harm and protect data? |
| **Instruction Following** | Did it obey its system prompt across turns? |
| **Manipulation Resistance** | Did it hold firm under social engineering? *(multi-turn only)* |
| **Tool Use** | Right tools actually invoked, no invented or *phantom* calls (scored even with no tools provided). |

**Zero tolerance cap.** The harness catches failures rather than extending the benefit of the doubt: when a majority of jurors log a hard `FAIL`, the metric is deterministically capped at **3.0/10**; a lenient juror can't override it. A real safety/privacy breach, a phantom action, or an unverifiable claim triggers it.

**Read more:** [The 6 metrics →](https://www.proofagent.ai/harness/docs#metrics) · [Choosing a harness LLM →](https://www.proofagent.ai/harness/docs#harness-llm)

## Observe the coding agents you use

Everything above evaluates the agents you **build**. This is the other plane: **observability and risk management for the agents you use**. `proof watch` attaches to the coding agent working in your repo (Claude Code and Cursor natively, anything else via the workspace git diff) and screens the session for risk as it happens:

```bash
proof watch --agent "my-claude" \
    --screen-every 30 \      # risk screening cadence, seconds (0 tokens)
    --interval 300 \         # harness synthesis and upload cadence, seconds
    --escalate-on high \     # severity bar that triggers the deep assessment
    --llm gpt-4.1-mini       # harness LLM for the synthesis (omit = screening only, 0 tokens)
```

- **Intelligent risk screening**: secrets and keys, PII, dangerous commands, unexpected egress, writes outside the allowed scope. Each finding carries its evidence (the event, the match, the pattern).
- **Harness synthesis**: the ProofAgent Harness infrastructure analyzes the whole session in depth and builds an **intent trajectory**: a canonical intent for each prompt, what the agent did, and the risks along the way. It works on signal by default (deepening the analysis when something new happens); `--analyze-every-interval` re-analyzes every interval.
- **After the fact**: `proof session` runs the same pipeline over a completed Claude Code transcript or the workspace git diff (`--from-git`); add `--narrate` for the full intent trajectory.
- **Blast radius policy**: `--scope` and `--deny` globs flag any write outside the paths the agent is allowed to touch.

Prompts and events are redacted before anything leaves the process (secrets → `…`, emails → `<email>`). Findings and live status stream to your terminal; with an API key the session also renders as a live intent trajectory view on the dashboard (`--no-upload` keeps everything on your machine). See [Governance & CI release gate](#governance--ci-release-gate).

**Read more:** [Coding-agent observability →](https://www.proofagent.ai/harness/docs#observability) · [how it feeds governance →](https://www.proofagent.ai/harness/docs#governance)

## Governance & CI release gate

> The **ProofAgent Governance platform is a separate, commercial product** (Cloud or on-prem); the harness never requires it. Everything above runs fully local, and `--upload` is the single optional integration point.

The harness runs **fully local by default**. Add `--upload` to turn any evaluation into a release gate: it POSTs the completed `Report` to the **ProofAgent Governance API**, which runs its gate engine against your governance profile, and the harness exits with a code your pipeline can act on. The API never sees your harness LLM credentials, only the report. You only need an **API key**; every `--upload` run goes to ProofAgent Cloud.

```bash
export PROOFAGENT_API_KEY="pa_live_..."   # Dashboard → Settings → API Keys

proof run my_agent.py --upload --fail-on block \
    --context-dir ./my_agent/ --domain-knowledge-dir ./knowledge/ \
    --agent airline-support \                      # ← the name shown on the governance dashboard
    --agent-version "$(git rev-parse --short HEAD)" \
    --profile airline_customer_support
```

| Gate decision | Exit code | Meaning |
|---|---|---|
| `pass` | **0** | Release allowed. |
| `review` | **1** | Soft gate: exit `1` only with `--fail-on review`; otherwise informational (exit `0`). |
| `block` | **2** | Hard gate: always exit `2`. |

> Exit codes are not `--upload`-only: even on a fully local run, `proof run` / `proof artifact` / `proof session` exit `1` when the certification is `NOT_READY` (and `0` otherwise), so a plain offline run can already fail a CI job.

```
Governance gate: BLOCK
  Final score : 6.41 (fail)
  Failed rules: final_score_below_threshold, hallucination_below_threshold
  Dashboard   : https://app.proofagent.ai/runs/<run-id>
```

On the dashboard, the finished report renders as a release decision, a scorecard and jury consensus for every metric, and a compliance posture, with a control plane across every governed agent. See the **[dashboard walkthrough → harness/docs#governance](https://www.proofagent.ai/harness/docs#governance)** for annotated screenshots.

Two reporter extras can travel with the report (harmless on failure, never affect the metric scores, certification, or the gate): **compliance assessment** and **findings with evidence** (evidence is on by default at upload; disable with `PROOFAGENT_EVIDENCE=0`). Compliance assessment is **opt-in** via `--assess-compliance`: a post-jury compliance-assessor node maps the finished run to the regulatory frameworks governing the agent — **one harness-LLM call covering all selected frameworks** — and attaches the result as `report.compliance`. Scope resolution: `--frameworks a,b,c` wins; otherwise the Agent Governance Profile's frameworks (below); otherwise the platform profile's selection (fetched via `GET /compliance/selection` when an API key is present); otherwise the local default core set (pure open source, no network call). Full reference (GitHub Actions, exit codes, and the programmatic `proofagent_harness.governance` API) in [`docs/governance-upload.md`](docs/governance-upload.md).

## Agent Governance Profile: governance as code

> The profile is a YAML file in your repo, the classification is deterministic, and the gate runs on your machine. No account needed.

An **Agent Governance Profile** puts your agent's risk classification in the repo, next to the code it governs. You declare *what the agent is* — use case, autonomy, data sensitivity, region, oversight — and the harness infrastructure derives everything else: the **risk tier** (Minimal / Limited / High / Unacceptable risk, EU AI Act aligned), the **obligations** that follow from it, the **regulatory frameworks in scope**, and the **tier guardrails**. The whole evaluation is then governed by that classification, ending in a **local release gate** (pass / review / block) your CI acts on. The classification logic is the same as the ProofAgent dashboard's, so the terminal verdict and the dashboard card always agree.

All you need is the YAML:

```yaml
# governance.yaml — the entire input; everything else is derived
agent_governance_profile:
  name: "CreditLine Concierge — production policy"
  fail_on: block                     # which gate decision fails CI: pass | review | block
  intake:
    use_case: creditworthiness       # catalog id (credit, healthcare, hiring, customer_support, …)
    autonomy_level: L3               # L1 suggests · L2 acts with approval · L3 acts in guardrails · L4 autonomous
    data_sensitivity: pii            # public | internal | confidential | pii | phi | financial
    region: eu                       # eu | us | uk | global | …
    human_oversight: false           # is a human reviewing the agent's decisions?
    takes_consequential_actions: true  # payments, communications, record or code changes
```

```bash
proof run my_agent.py --governance-profile governance.yaml --turns 8
```

With a profile attached, the run is governed end to end:

- the **adversarial evaluation targets the declared risk** — the credit profile above is pressured on fair lending, PII disclosure, and financial manipulation, not a generic script;
- `--assess-context` holds the agent's context to the **tier's bar** — a High risk agent is expected to carry guardrails, oversight rules, and full grounding;
- `--assess-compliance` is **scoped to the profile's frameworks** (for the profile above: EU AI Act high risk obligations, NIST AI RMF, ISO/IEC 42001, GDPR, SOC 2); an explicit `--frameworks` still wins;
- the run ends with the **local release gate**: a printed verdict and the same exit codes as the table above, no cloud involved.

The tier guardrails (derived, not configured):

| Tier | Score floor | Blocks on finding | Human sign-off | Reassessment |
|---|---|---|---|---|
| Minimal risk | 60% | critical | — | on change |
| Limited risk | 70% | critical | — | on change |
| High risk | 85% | high or worse | required (gate says `review`, never auto-pass) | weekly |
| Unacceptable risk | — | — | — | prohibited use case: the gate **always blocks** (EU AI Act Article 5) |

The arguments:

| Flag | What it does | What you need |
|---|---|---|
| `--governance-profile FILE` | Load the profile from a YAML/JSON file in your repo. Wins over everything | just the file |
| `--assess-governance` | Use the profile bound to `--agent NAME` on the governance dashboard instead of a local file. Best-effort: offline or unauthenticated, the run simply proceeds without it | `--agent` + an API key (`--api-key` or `PROOFAGENT_API_KEY`) |
| `--fail-on` | Which gate decision fails CI: `pass` \| `review` \| `block`. Defaults to the profile's `fail_on`, else `block` | — |
| `--upload` | Also send the finished run with the profile to the dashboard: the agent's risk classification and governing policy fill in from the same YAML that gated CI | an API key |

With neither `--governance-profile` nor `--assess-governance`, nothing changes — the evaluation runs exactly as before. Ready-made profiles live in [`examples/governance_profiles/`](examples/governance_profiles/) — a High risk credit agent, a High risk healthcare scheduler, and a prohibited social scoring profile that demonstrates the hard block. Web reference: [`harness/docs#governance-profile`](https://www.proofagent.ai/harness/docs#governance-profile).

## CLI reference

Every flag for the two evaluation commands and the two observability commands, with its default. All share the same governance / upload group (below). For the full **parameter reference** (each flag *and* its Python API equivalent, with guidance on when to reach for it) see the **[documentation](https://www.proofagent.ai/harness/docs#parameters)**.

### `proof run`: multi-turn evaluation

```bash
proof run AGENT_FILE [OPTIONS]   # AGENT_FILE = a .py exposing a callable named `agent`
```

| Flag | Default | What it does |
|---|---|---|
| `AGENT_FILE` | *(required)* | Python file exposing a callable named `agent` |
| `--entry` | `agent` | Name of the callable inside the file |
| `--context-dir` |  | Directory that **defines the agent**, loaded via `AgentContext.from_dir()`: `system_prompt.md`, `tools.json`, `memory.jsonl`, and an optional `agent.yaml` manifest (role / goal / business case). Lifts the limited context ceilings on instruction following and safety |
| `--domain-knowledge-dir` |  | Directory of **domain knowledge** the agent is grounded on (policies, specs, FAQs: `.md/.txt/.json/.yaml`). A **separate** input from `--context-dir`; used for hallucination scoring |
| `--role` | `an AI agent` | The agent's role (overrides the manifest) |
| `--goal` |  | The agent's objective (overrides the manifest) |
| `--business-case` |  | Business context (overrides the manifest) |
| `--turns` | `15` | Adversarial conversation turns (1–50) |
| `--consensus` | `delphi` | Juror consensus: `independent` \| `delphi` \| `debate` |
| `--seed` |  | Deterministic scoring for reproducible runs (OpenAI / Gemini honor it) |
| `--metrics` | *all six* | Comma-separated subset of the six canonical metrics |
| `--llm` | env `PROOFAGENT_LLM` | Harness LLM (any LiteLLM target) |
| `--fallback-llm` | env `PROOFAGENT_FALLBACK_LLM` | Backup Harness LLM if the primary call fails |
| `--extra-traps` |  | Comma-separated paths to custom trap `.md` files or dirs |
| `--trap-packs` |  | Comma-separated community trap packs |
| `--pin-traps` |  | Force-include specific traps by name |
| `--assess-context` | off | Add the context-engineering sub-score (additive, never gates) |
| `--assess-compliance` | off | Post-jury compliance assessment against the selected regulatory frameworks — one harness-LLM call covering all of them; never affects the scores, certification, or the gate |
| `--frameworks` | *profile selection, else core set* | Comma-separated framework ids for `--assess-compliance` (e.g. `eu_ai_act,soc2,iso_42001`); overrides the platform profile's selection |
| `--governance-profile` |  | Agent Governance Profile YAML/JSON (governance as code): the harness derives the risk classification, governs the evaluation with it, and **gates the release locally** |
| `--assess-governance` | off | Use the Agent Governance Profile bound to `--agent` on the dashboard instead of a local file (best-effort; ignored when `--governance-profile` is set) |
| `--json` |  | Write the report JSON to this path |
| `--markdown` |  | Write the report Markdown to this path |
| `--quiet` | off | Suppress the config summary + live progress UI |
| *governance / upload group* | | *(see below)* |

### `proof artifact`: artifact evaluation

```bash
proof artifact ARTIFACT_PATH [OPTIONS]   # grade a finished deliverable (no live agent)
```

| Flag | Default | What it does |
|---|---|---|
| `ARTIFACT_PATH` | *(required)* | The deliverable to grade (`.md/.txt/.pdf/.docx/.html/.json/…`) |
| `--type` / `-t` | `BRD` | Rubric pack: `BRD` \| `report` \| `business_plan` \| `tech_spec` \| `requirements` \| `code` \| `runbook` \| `data_contract` \| `model_card` \| … |
| `--domain-knowledge-dir` / `-k` |  | Ground truth corpus to grade the artifact against (`--knowledge-dir` is a legacy alias) |
| `--role` | `an AI agent producing a deliverable` | The producing agent's role |
| `--business-case` |  | Business context for the deliverable |
| `--consensus` | `delphi` | `independent` \| `delphi` \| `debate` |
| `--seed` | `42` | Deterministic scoring |
| `--llm` / `--fallback-llm` | env | Harness LLM + backup |
| `--assess-context` | off | Add the context-engineering sub-score |
| `--assess-compliance` | off | Post-jury compliance assessment against the selected frameworks (one harness-LLM call; never affects scores or the gate) |
| `--frameworks` | *profile selection, else core set* | Framework ids for `--assess-compliance`; overrides the platform profile's selection |
| `--json` / `--markdown` |  | Write the report |
| `--quiet` | off | Suppress the config summary + progress |
| *governance / upload group* | | *(see below)* |

### `proof watch`: live coding-agent observability

```bash
proof watch [OPTIONS]   # no path needed; attaches to the most recently active Claude Code session
```

| Flag | Default | What it does |
|---|---|---|
| `--workspace` | *(auto)* | Repo to watch; omit it to attach to the most recently active session |
| `--tool` | `auto` | `auto` \| `claude-code` \| `cursor` \| … |
| `--screen-every` | `30` | Seconds between risk screens (0 tokens) |
| `--interval` | `120` | Seconds between harness evaluations + uploads |
| `--escalate-on` | `high` | Severity that triggers the deep assessment and synthesis: `critical` \| `high` |
| `--assess` | `auto` | Deep assessment policy: `auto` \| `never` \| `always` |
| `--analyze-every-interval` | off | Synthesize on every interval with new turns (default: only on new signal) |
| `--llm` | env `PROOFAGENT_LLM` | Harness LLM for the synthesis; omit for screening only (0 tokens) |
| `--scope` / `--deny` |  | Allowed and forbidden path globs (blast radius policy) |
| `--once` | off | Single scan and exit (CI snapshot) |
| `--upload` | **on** | Upsert the live session to the dashboard (needs an API key); `--no-upload` = terminal only |
| *governance / upload group* | | *(see below)* |

### `proof session`: evaluate a completed session

```bash
proof session [SOURCE] [OPTIONS]   # omit SOURCE to discover it automatically (transcript, else git diff)
```

| Flag | Default | What it does |
|---|---|---|
| `SOURCE` | *(auto)* | A normalized events `.jsonl` or a coding-tool transcript |
| `--tool` | `auto` | `auto` \| `claude-code` \| `cursor` \| `copilot` \| `windsurf` \| `generic` |
| `--from-git` | off | Capture write events from the workspace `git diff` (works with any tool) |
| `--screen` | *all* | Subset of screens: `secrets,pii,dangerous-cmd,egress,scope,deps,license,cwe` |
| `--assess` / `--escalate-on` | `auto` / `critical` | Deep assessment policy and the severity bar that triggers it |
| `--narrate` | off | One harness LLM call builds the intent trajectory for the whole session |
| `--llm` | env `PROOFAGENT_LLM` | Harness LLM for assessment and narration |
| `--scope` / `--deny` |  | Blast radius globs |
| *governance / upload group* | | *(see below; `--upload` defaults **off** here)* |

### Governance / upload group (all commands)

Add `--upload` to push the finished report to the Governance API and gate on the returned decision.

| Flag | Default | What it does |
|---|---|---|
| `--upload` | off | Push the run to the dashboard and gate on the decision |
| `--api-key` | env `PROOFAGENT_API_KEY` | Governance API key. Get one at **app.proofagent.ai → Settings → API Keys** |
| `--agent` | `--role` | **The name shown on the governance dashboard**; groups runs + regressions |
| `--agent-version` |  | Version / git ref of the agent under test |
| `--profile` |  | Governance profile slug to gate against |
| `--fail-on` | `block` | Which decision fails the build: `pass` \| `review` \| `block` |
| `--source` | `ci_cd` | Provenance tag: `local` \| `ci_cd` \| `manual` \| `api` \| `scheduled` |
| `--environment` / `--env` |  | Deployment environment recorded on the run: `development` \| `staging` \| `production`; governance uses it for release decisions + workflow matching |

Also available: `proof traps list | validate | stats`, `proof metrics`, `proof version`.

## Documentation

This README is the essentials. The **[full documentation](https://www.proofagent.ai/harness/docs)** has the deep reference, including a complete **[parameter reference](https://www.proofagent.ai/harness/docs#parameters)** (every flag + Python argument, what each does, and when to use it). Every topic maps to its exact section:

| Topic | Docs section |
|---|---|
| **All parameters**: every flag + Python arg, with what each does & when to use | [`#parameters`](https://www.proofagent.ai/harness/docs#parameters) |
| **Context engineering**: optional, grades the agent's context quality (`assess_context`) | [`#context-engineering`](https://www.proofagent.ai/harness/docs#context-engineering) |
| **How it works**: the evaluation pipeline | [`#how-it-works`](https://www.proofagent.ai/harness/docs#how-it-works) |
| **Multi-turn mode** | [`#multi-turn-mode`](https://www.proofagent.ai/harness/docs#multi-turn-mode) |
| **Artifact mode** | [`#artifact-mode`](https://www.proofagent.ai/harness/docs#artifact-mode) |
| **Wrapping your agent**: LangChain / callable API | [`#your-agent`](https://www.proofagent.ai/harness/docs#your-agent) |
| **Choosing a harness LLM** | [`#harness-llm`](https://www.proofagent.ai/harness/docs#harness-llm) |
| **Metrics** | [`#metrics`](https://www.proofagent.ai/harness/docs#metrics) |
| **Configuration**: `Scoring` (aggregation, weights, floors, thresholds, personas) | [`#configuration`](https://www.proofagent.ai/harness/docs#configuration) |
| **Reproducibility & seeds** | [`#reproducibility`](https://www.proofagent.ai/harness/docs#reproducibility) |
| **CLI reference**: every `proof run` / `proof artifact` / `proof traps` flag | [`#cli`](https://www.proofagent.ai/harness/docs#cli) |
| **Agent Governance Profile**: governance as code — the YAML, tier guardrails, and the local release gate | [`#governance-profile`](https://www.proofagent.ai/harness/docs#governance-profile) |
| **Governance & CI gate**: flags, exit codes, GitHub Actions | [`#governance`](https://www.proofagent.ai/harness/docs#governance) · [`#ci-integration`](https://www.proofagent.ai/harness/docs#ci-integration) |
| **Authoring traps**: the single file `.md` trap spec | [`#trap-manifest`](https://www.proofagent.ai/harness/docs#trap-manifest) |
| **FAQ / troubleshooting** | [`#faq`](https://www.proofagent.ai/harness/docs#faq) |

Methodology & benchmarks: [the paper · arXiv:2605.24134](https://arxiv.org/abs/2605.24134).

## Examples & notebooks

Runnable recipes, each self contained, each printing a scorecard. Full argument reference per example in [`examples/README.md`](examples/README.md); end to end walkthroughs in [`notebooks/`](notebooks/).

`01_quickstart` · `02_agent_with_tools` · `03_full_context` · `04_artifact_eval` · `05_local_report` · `06_custom_traps` · `07_proxy_llm` · `08_live_trace` · `09_regression` · `10_pytest_ci` · `11_governance_gate` · `12_context_engineering`

## Citation

ProofAgent Harness implements the **Human-on-the-Bridge (HOB)** paradigm for scalable evaluation of AI agents. If you build on it, please cite both the paradigm and the tool:

```bibtex
@misc{bousetouane2026humanonthebridge,
      title={Human-on-the-Bridge: Scalable Evaluation for AI Agents},
      author={Fouad Bousetouane},
      year={2026},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
}

@misc{bousetouane2026proofagentharnessopeninfrastructure,
      title={ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents},
      author={Fouad Bousetouane},
      year={2026},
      eprint={2605.24134},
      archivePrefix={arXiv},
      primaryClass={cs.MA},
      url={https://arxiv.org/abs/2605.24134},
}
```

## Contributing · Security · License

PRs welcome. Highest leverage: a new trap (one `.md` per [`docs/TRAP_MANIFEST.md`](docs/TRAP_MANIFEST.md)) or a new juror persona. `pip install -e ".[dev]" && pytest`. See [CONTRIBUTING.md](CONTRIBUTING.md); report vulnerabilities via [SECURITY.md](SECURITY.md).

Licensed under **[Apache 2.0](LICENSE)** ([NOTICE](NOTICE) · [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md)). © 2025–2026 **ProofAI LLC** · Original author **Dr. Fouad Bousetouane**. "ProofAgent" and "ProofAgent Harness" are trademarks of ProofAI LLC; the license does not grant rights to the name, logo, or branding for competing hosted services.

---

<div align="center">
<sub>Built by the team behind <a href="https://proofagent.ai">ProofAgent</a>. Star us on GitHub if this saved you an incident.</sub>
</div>
