Metadata-Version: 2.4
Name: corrobo
Version: 0.1.5
Summary: Outcome verification for AI agents: when a tool says done, Corrobo reads the real system and rules VERIFIED, SAY-DO GAP or UNVERIFIED.
Author-email: Fahad <fahadrahiman10@gmail.com>
License: PolyForm-Noncommercial-1.0.0
Project-URL: Homepage, https://pypi.org/project/corrobo/
Keywords: ai-agents,verification,mcp,say-do-gap,guardrails,observability,postgres,attestation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: Other/Proprietary License
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3; extra == "postgres"
Provides-Extra: mcp
Requires-Dist: mcp>=2; extra == "mcp"
Provides-Extra: attest
Requires-Dist: cryptography>=41; extra == "attest"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: cryptography>=41; extra == "dev"
Dynamic: license-file

# Corrobo

**When an AI agent says "done", Corrobo reads the real system and rules VERIFIED, SAY-DO GAP or UNVERIFIED.**

An outcome verification layer for AI agents. The question it answers is not "did the tool return
success?" but **"did the promised outcome actually happen?"** Let the agent call the tool exactly as it
does today; Corrobo then independently asks the system of record whether the effect the call promised
is really there. It never trusts the tool's own response. It reads the world.

To be precise about the word: Corrobo verifies **observable effects**. It answers "did the rows this
contract names end up the way the contract says", not "did the agent behave correctly in some absolute
sense". The contract is yours; Corrobo holds the agent to it.

Free for noncommercial use (PolyForm Noncommercial 1.0.0). Commercial use needs a license: see
[License](#license).

## The problem, in one table

| Caller asked for | What the agent told the caller | What the database said | Verdict |
| --- | --- | --- | --- |
| 9:30 AM, free | "your cleaning is all set for October 5th at 9:30 AM" | a row for this caller, this slot | **VERIFIED** |
| 9:00 AM, already taken | "it looks like 9:00 AM is already taken" | no row for this caller | **UNVERIFIED**: honest failure, no accusation |
| taken slot; tool refused; caller told "booked" anyway | "You're all set!" | no row | **SAY-DO GAP** |

Row three is the failure nobody's logs show. The constraint held, the tool said no, the log said
DENIED, and the caller was still told "all set". Every layer below the reply did its job; none of them
tests the sentence. Corrobo is the line that compares the sentence to the table.

## See it in thirty seconds

No key, no network, nothing to configure:

```bash
pip install corrobo
corrobo demo
```

A receptionist agent, three callers, one booking table. For each caller it prints what your stack saw
and what the caller heard:

```
3) a taken slot, the agent says it is booked anyway
   constraint:  UNIQUE(date, time)  -> held: insert refused
   tool said:   unavailable
   log said:    TOOL -> book_appointment DENIED (taken) date='2026-10-05' time='3:00 PM'
   agent said:  "You're all set! See you on October 5th at 3:00 PM."
   table says:  3:00 PM -> someone else
   CORROBO:     SAY-DO GAP  - the real state did not match the expected effect
```

The replies in callers 1 and 2 are verbatim from a live run of a real agent (below); caller 3 is
scripted and labelled so on screen, because in that run the agent was honest 4 times out of 4. It is
the conversation Corrobo exists for on the day the agent is not.

## The whole idea, in code

Here is the entire idea: let the agent call the tool normally, then independently ask the database
whether the promised outcome actually happened.

```python
import sqlite3
from corrobo import Corrobo, SQLiteStateSource

con = sqlite3.connect("demo.db")
con.execute("CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY, slot TEXT UNIQUE, who TEXT)")
con.commit()

def book(slot, who):                      # a tool that LIES: says ok even when the slot was taken
    try:
        con.execute("INSERT INTO bookings (slot, who) VALUES (?, ?)", (slot, who)); con.commit()
    except sqlite3.IntegrityError:
        pass
    return {"ok": True, "message": f"You're all set for {slot}!"}

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="demo.db"))

for who in ("Alex", "Jordan"):            # second caller gets the same slot
    vd.check("book", {"slot": "Tue 3pm", "who": who}, lambda: book("Tue 3pm", who),
             succeeded=lambda r: r["ok"],                                   # what the tool CLAIMED
             reads=lambda a: [("bookings", {"slot": a["slot"]})],           # the rows this call touches
             effect=lambda before, after, a: any(r["who"] == a["who"] for r in after["bookings"].values()))
    print(who, "->", vd.last()["status"])

# Alex -> VERIFIED
# Jordan -> SAY-DO GAP        (told "all set"; the table says otherwise)
```

Two things you declare, and only two: `reads(args)`, the exact rows the action touches, and
`effect(before, after, args)`, what "done" means on the real state. Corrobo snapshots the footprint,
runs the tool, snapshots again, and judges the effect against real state, never the return value.

## Proven on real agents

**A live receptionist.** On 2026-09-24 Corrobo was wired to a production booking agent (Gemini + a
live Postgres) and run over eight real conversations, four of them asking for a slot that was already
taken. Verdicts matched the table in 8 of 8; the table at the top of this page is from that run. It
also caught its author twice on the way: a contract that said "done = *any* booking on this slot" was
ruled `VERIFIED_VACUOUS` because the effect already held before the action, and a claim-reader that
counted "that slot has *just been booked*" as a booking claim was exposed by printing its
disagreements. A verifier that flags a lazy contract is the point.

**Public benchmarks.** The same check, with gemini-2.5-flash as the agent, over third-party agent
benchmarks. A *claim* is the agent's final message asserting the task is complete; a claim is *false*
when the benchmark's own state assertions fail. Corrobo's effect predicate is written independently of
those assertions, so the two cannot agree by construction.

| suite | tasks | claimed success | of those, false |
| --- | --- | --- | --- |
| Box (Agent-Diff enterprise) | 47 | 17 | **10 (59%)** |
| Linear (Agent-Diff) | 65 | 27 | 4 (15%) |
| τ²-bench retail (Sierra), 2 runs | 39 x 2 | 41 | 1 (2%) |

What the Box result taught: every one of the ten false claims **changed real state**. The agent acts,
rows move, and the result is not what was asked: wrong file, wrong destination, half the job. So the
cheap check "if it claims success, something must have moved" catches 0 of 10, and neither the tool's
return value nor a transcript-reading LLM judge caught them either. A per-tool declared effect checked
against the database caught 9 of 10. The retail number is low because most of what first looked like
a gap (13%) was the benchmark's own user simulator renegotiating the task mid-conversation, which the
transcripts show and the re-score excludes.

Limits of the measurement: one model, three suites here (the smaller AgentDojo suites had one to five
claims each, too few to quote), and the tasks and grading are the benchmarks' own. Method, per-task
records and full transcripts are committed under `benchmarks/` in the repository (private during beta;
email the author for access).

**Real systems the engine has been validated against:** GitHub REST, PostgreSQL, Jira Cloud, MCP over
stdio, and the receptionist above.

## Install

```bash
pip install corrobo                 # core: pure standard library, no dependencies
pip install "corrobo[postgres]"     # + PostgresStateSource   (psycopg 3)
pip install "corrobo[mcp]"          # + drive a real MCP server over stdio (mcp)
pip install "corrobo[attest]"       # + mint/verify Ed25519 success attestations (cryptography)
```

Python 3.10+ (tested on 3.10, 3.11, 3.12). The core imports nothing outside the standard library; the
extras are imported lazily, so installing the core never pulls a driver you do not use.

## How it works

```
agent calls tool
      |
      v
 read real state  (only the rows `reads` names, through Corrobo's own read-only connection)
      |
      v
 run the tool     (untouched; its return value is recorded, not trusted)
      |
      v
 read real state again
      |
      v
 evaluate `effect(before, after, args)`  +  the tool's own claim (`succeeded`)
      |
      v
 VERIFIED  |  SAY-DO GAP  |  UNVERIFIED  |  BLOCKED
```

| Status | Meaning |
| --- | --- |
| **VERIFIED** | the promised effect is actually present in real state |
| **SAY-DO GAP** | the tool **claimed success** but real state does not match the promise (it lied, or silently failed) |
| **UNVERIFIED** | the result could not be established: the tool **reported failure** (an honest failure, not a gap), the state could not be read, or a **concurrent modification** made the outcome unattributable |
| **BLOCKED** | a pre-check (policy or precondition) refused the action before it ran (enforce mode) |

An honest tool failure is distinguished from a lie: pass `succeeded=lambda r: ...` (an HTTP 2xx check,
an MCP `is_error` flag) and a tool that reports failure becomes **UNVERIFIED (action-failed)**, never a
say-do gap.

**Monitor vs enforce.** `mode="monitor"` (the safe default) runs the tool, verifies, records evidence
and never interferes. `mode="enforce"` lets a pre-check block the action, raises `CorroboSayDoGap` on a
gap so the false "all set" never reaches the user, and raises `CorroboUncertain` when the outcome could
not be established. An honest tool failure does not raise; its own failure result is returned.

## Start here: the six names you need

The package exports a lot (mining, invariants, attestations, the MCP proxy). Ignore all of it until
you need it. Day one is:

| Name | What it is |
| --- | --- |
| `Corrobo(mode, state)` | one instance per app; `"monitor"` observes, `"enforce"` blocks and raises |
| `SQLiteStateSource` / `PostgresStateSource` / `RestStateSource` | where the truth lives; read-only |
| `@vd.verify(reads=..., effect=...)` | decorate your own function |
| `vd.check(name, args, thunk, reads=..., effect=...)` | the same, for a tool you only have a handle to |
| `vd.last()` | the verdict: `["status"]`, `.explain()`, `.get("reason")` |
| `mine_tool(vd, name, fn, arg_list)` | let Corrobo draft the contract by watching the tool (below) |

`help(Corrobo)` and `help(Corrobo.check)` describe every argument.

## Integrations

### Already have tools? Wrap them where they are

Nothing needs rewriting. Each adapter takes the tool you have and returns the same tool, verified;
name, description, schema and return value are untouched. None of them import a framework.

```python
from corrobo import wrap_function, wrap_langchain_tool, wrap_openai_function_tool, wrap_tools

BOOKING = dict(reads=lambda a: [("bookings", {"slot": a["slot"]})],
               effect=lambda before, after, a: any(r["who"] == a["who"] for r in after["bookings"].values()))

book  = wrap_function(vd, book, **BOOKING)                      # plain function, sync or async
tools = [wrap_langchain_tool(vd, book_tool, **BOOKING), search]  # LangChain @tool / StructuredTool
agent = Agent(tools=[wrap_openai_function_tool(vd, book, **BOOKING)])   # OpenAI Agents SDK
tools = wrap_tools(vd, tools, {"book": BOOKING, "cancel": CANCEL})      # a whole toolset, by name
```

`wrap_tools` raises if a contract names a tool that is not in the list: a misspelled contract that
quietly verified nothing is the failure this library exists to catch. Async tools stay async.
Details and per-framework notes: [docs/INTEGRATIONS.md](docs/INTEGRATIONS.md).

### Where the truth lives: one engine, any backend

The engine reads state through a `StateSource`; only the adapter changes.

```python
from corrobo import SQLiteStateSource       # a SQLite file or connection
from corrobo import PostgresStateSource     # real Postgres (consistent READ ONLY snapshot, PK auto-detect)
from corrobo import RestStateSource         # any REST/HTTP API (scoped GETs, Retry-After, retries)
from corrobo import MultiStateSource        # several systems of record in one contract
from corrobo import CorroboMCPProxy         # sit in front of an existing MCP server; verify each tool call
```

Any REST API, for example GitHub:

```python
from corrobo import Corrobo, RestStateSource

vd = Corrobo(mode="monitor", state=RestStateSource(
    "https://api.github.com",
    resources={"issue": {"path": "/repos/OWNER/REPO/issues/{number}", "pk": "number"}},
    headers={"Authorization": f"Bearer {TOKEN}", "Accept": "application/vnd.github+json"}))

vd.check("rename_issue", {"number": 1, "new_title": "Fixed"},
         lambda: github_patch(1, {"title": "Fixed"}),      # your real tool call
         succeeded=lambda r: r.ok,                           # HTTP 2xx = the tool claims success
         reads=lambda a: [("issue", {"number": a["number"]})],
         effect=lambda before, after, a: after["issue"][a["number"]]["title"] == a["new_title"])

print(vd.last().status)        # VERIFIED  |  SAY-DO GAP  |  UNVERIFIED
print(vd.last().explain())     # a human, secrets-free explanation:
#   Action:    rename_issue  (args: new_title, number)
#   Expected:  the declared effect to hold on issue[number] (read independently, before & after)
#   Observed:  real state changed [issue #1: comments, updated_at], but the promised effect did NOT hold
#   Status:    SAY-DO GAP — the real state did not match the expected effect (say-do gap)
```

Your own system is a small class with one method, "read these things"; a conformance kit checks it
returns the right shape (`docs/STATESOURCE.md`).

### Do not want to write the contract? Let it be proposed

Point `mine_tool` at your function with a handful of calls. It snapshots before and after each one,
finds what always changes and how it relates to the arguments, mutation-tests each rule, and returns
**proposals**: things a human accepts or rejects, never contracts that apply themselves.

```python
from corrobo import Corrobo, SQLiteStateSource, mine_tool, render_proposals

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="app.db"))

def upgrade(user_id, plan):
    con.execute("UPDATE users SET plan=? WHERE id=?", (plan, user_id)); con.commit()

props = mine_tool(vd, "upgrade", lambda a: upgrade(**a),
                  [{"user_id": i, "plan": "pro"} for i in range(1, 7)],
                  reads=lambda a: [("users", {"id": a["user_id"]})])
print(render_proposals(props))
# [1] `upgrade` always set `plan` to whatever was passed as `plan` (6 calls) ...   score 18/18 mutations caught
# [2] `upgrade` changed exactly 1 row(s) in all 6 of 6 calls. Should more than 1 be refused?
# [3] `upgrade` only ever touched ['users']. Should touching anything else be refused?

# accept one: an effect proposal is a contract in one line
rule = next(p for p in props if p.kind == "effect").as_effect()
vd.check("upgrade", {"user_id": 7, "plan": "pro"}, lambda: upgrade(7, "pro"),
         reads=lambda a: [("users", {"id": a["user_id"]})], effect=rule)
print(vd.last()["status"])   # VERIFIED
```

It really runs the tool six times, so use a copy of the data. Fewer than five calls proposes nothing,
on purpose: a rule learned from three calls is a guess wearing a badge. Shape proposals (row counts,
collections touched) become invariants through `accept(p)`.

## Beyond one check: an independent trust boundary around the agent

The say-do-gap check is layer 1. What makes Corrobo more than "a library that diffs database state"
is that the agent can be kept out of the loop that judges it:

1. **Independent verification**: read real state; decide VERIFIED / SAY-DO GAP / UNVERIFIED (above).
2. **Cross-system outcomes**: one contract can require an effect across *multiple* systems of record
   (`MultiStateSource` + identity correlation). The whole invariant must hold, or it is a gap.
3. **Independent verifier, out of process**: run the engine as its OWN process with its OWN read-only
   credentials, contracts, and signing key. The agent sends only `{contract_id, args}`. It cannot
   supply the state source, the contract, or the key (`VerifierService` / `VerifierClient`).
4. **Enforcement gate**: an authoritative success is released only on an independent VERIFIED
   (`OutcomeGate`).
5. **Signed success attestations**: on a genuine VERIFIED the verifier mints a short-lived **Ed25519**
   attestation bound to the operation; any consumer validates it with the **public key**, no secret.
   It cannot be forged, modified, replayed, retargeted, or outlived (`OutcomeGate.accept`,
   `verify_attestation`).
6. **Consumption-time re-verification**: optionally re-check the effect at the moment of release to
   shrink the TOCTOU window (`accept(..., reverify=True)`).

Also in the package, each with its own doc: invariants and frame conditions (what must NOT have
changed: blast radius), an ambiguity gate (refuse to act when the target is a guess), confirmation
binding (verify against what a person approved), a retry gate, per-step chain verification, and drift
detection against a baseline.

## Evidence

Every verification appends to a tamper-evident, hash-chained log. It stores **only hashes, the
argument keys, the scoped footprint (column names), and which columns changed**: never raw values,
never secrets, never the connection string.

```python
vd = Corrobo(mode="monitor", state=..., evidence_path="corrobo.jsonl")
vd.evidence_log.verify()     # True: the chain is intact
```

## Current guarantees (validated on real systems)

- VERIFIED is emitted only when the promised effect is actually present, decided by an **independent
  read**, never the tool's response body. Multi-condition effects and idempotent re-application verify
  correctly.
- Real say-do gaps are caught: wrong record, wrong field, partial effect, or a lying success response.
- Honest tool failures (401/403/404/HTTP errors, DB errors, tool-reported failure) become
  **action-failed / UNVERIFIED**, never a say-do gap.
- **Fail-safe under unavailability**: a read 404, a timeout, a dropped connection, or a resource that
  vanished gives **UNVERIFIED, never a false VERIFIED**.
- **Concurrency**: a concurrent change to the **effect-relevant** state gives UNVERIFIED /
  CorroboUncertain; **unrelated** concurrent churn (an extra comment, a bumped `updated_at`, an
  unrelated row or column) does **not** cause a false failure.
- Direct-record reads are treated as strongly consistent; genuinely lagging reads are handled by a
  bounded settle window (`settle_retries`); rate limits honor a bounded `Retry-After`.

## Limitations (honest)

- Corrobo verifies **"did the tool do what this call promised"**, not **"was this call what the user
  asked for"**. An agent that confidently acts on the wrong order id, spoken by the customer and acted
  on faithfully, passes. That needs a check *before* execution (the ambiguity gate and confirmation
  binding are the start of one), not a better check after it.
- **Concurrency detection is verdict-based, not field-traced**: a concurrent write that changes a
  footprint row but does **not** flip the effect's verdict (a loose or inequality effect) is not
  flagged. Exact-value effects, the common case, have no blind spot.
- A read **404 is ambiguous** (truly absent vs. no access), so a genuinely absent target is reported as
  **UNVERIFIED (fail-safe)**, not a definitive say-do gap.
- Corrobo is a **verifier, not a lock manager or a rollback engine**: it detects and, in enforce,
  blocks; it does not undo the tool's transaction.
- Postgres contracts must declare a `reads` footprint (whole-DB reads are refused for production
  safety), and every table read needs a primary key.
- Expiring-token refresh for REST auth is not built in yet (static headers today).

## What ships in the package

`corrobo/` only: the engine, the SQLite / Postgres / REST adapters, the framework adapters, the MCP
proxy, contract mining, invariants, frame conditions, drift detection, the retry and ambiguity gates,
the out-of-process verifier, the enforcement gate and Ed25519 attestations. About 8,000 lines, with
roughly 11,500 lines of tests behind them (87 files: engine tests plus live validation campaigns
against GitHub, Postgres, Jira, MCP, and the real LangChain and OpenAI Agents packages).

## Status

Beta. The engine has had three adversarial review passes, a full CodeRabbit sweep, a public-benchmark
campaign and one live production workload. The verdict logic is stable; what is still moving is
deployment hardening (key distribution, a shared replay store, TLS, multi-instance) and the ergonomics
people ask for. **If you run an agent that writes to a real system, the most useful thing you can do
is run Corrobo in monitor mode beside it for a week and tell the author what it found.**

## Feedback

Bugs, false verdicts, and "it should have caught this" reports are the most valuable thing you can
send. Email **fahadrahiman10@gmail.com** and include the `explain()` output and the evidence entry
(both are secrets-free by construction). Commercial licensing goes to the same address.

## License

PolyForm Noncommercial 1.0.0. **Free for noncommercial use**: personal projects, research, teaching,
evaluation, and noncommercial organizations. **Commercial use needs a separate license**: contact the
author. See `LICENSE` for the full terms.
