Metadata-Version: 2.4
Name: critic-orchestrator
Version: 0.8.0
Summary: MCP server that spawns adversarial reviewers in parallel (Claude CLI, or any OpenAI-compatible / Anthropic API) to verify a claim/fix. Anti-confabulation layer for AI coding agents.
Author-email: Aurelio Capriello <aureliocpr@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/aureliocpr-ctrl/critic-orchestrator
Project-URL: Repository, https://github.com/aureliocpr-ctrl/critic-orchestrator
Project-URL: Issues, https://github.com/aureliocpr-ctrl/critic-orchestrator/issues
Keywords: mcp,claude,adversarial-review,code-review,anti-confabulation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=1.0
Requires-Dist: psutil>=5.9
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Dynamic: license-file

# critic-orchestrator

> An anti-confabulation verification layer for AI coding agents. It spawns
> independent adversarial reviewers, in parallel, to check a claimed
> fix — *before* you trust it. Provider- and agent-agnostic.

**Status**: 406 tests green. Used daily in my own workflow; APIs beyond the
Claude CLI backend are unit-tested and live-smoke-tested (see *Verified* below).

**What it found in itself.** The strongest thing I can say about this tool is
what happened when I pointed it at its own execution surface: four criticals in
one night, each verified before it was cured, three of them found by a
different model than the one before. All four were the same class — a defence
enumerating what someone had already imagined, or inspecting a different
representation than the one it protected. Details in the CHANGELOG, including
the two cases where the *gate itself* reported "clean" over a finding it had
just made.

---

## The problem

When an AI coding assistant tells you *"I fixed the bug, the test passes now"*,
three things are easy to miss:

- Did the test actually **fail before** the fix, or was it written to pass
  after the fact (a confirmation, not a real regression test)?
- Is the function that was patched actually **reached** from a real entry
  point, or is it dead code?
- Does the fix hold at the **boundaries**, or is there an input where the
  bug still bites?

These are exactly the gaps that produce confident-but-wrong output. This tool
turns those three questions into three independent checks.

---

## How it works

`critic-orchestrator` runs **three adversarial reviewers in parallel**, each
starting from a fresh context and prompted to *disprove* the claim, not agree
with it:

| Reviewer | Role | Procedure |
|---|---|---|
| `falsification` | Pin the test | `git stash` → run the test → `git stash pop` → run again. The test must **fail** before the fix and **pass** after. |
| `caller_verification` | Reach reality | Find the call sites of the patched function. At least one must sit on a real entry point (CLI, API, hook), not only tests. |
| `counterexample` | Break the fix | Look for a concrete input/boundary case where the fixed code still misbehaves. |

Each reviewer returns a JSON-schema-constrained verdict; the orchestrator
aggregates them into a consensus: `claim_holds` / `claim_fails` / `split` /
`undecided`.

The reviewers run in an async job model: `start_adversarial_review` returns a
`job_id` in under 100 ms, then you poll — so a long review never trips the
60 s MCP client timeout.

---

## Before the fix: design review (the no-claim gate)

The three reviewers above verify a *claim* against a *diff* — they run
**after** a change. That contract has a measured failure mode: the proposer
writes the claim, and the claim silently defines the review perimeter (the
counterexample reviewer is instructed to discard findings outside the claim).
The review then scores the proposer's prudence, not the artifact — observed
here as four consecutive 3-0 `claim_holds` votes on changes where deterministic
mutation testing later found five theatre-tests.

`start_design_review` is the structural cure. It reviews **modules or a design
doc, before or without a fix**, and it accepts **no claim and no diff** — by MCP
schema, not by convention. The reviewers read the files raw and infer what the
code is *for* from the artifact itself:

| Reviewer | Lens | Human practice it encodes |
|---|---|---|
| `premortem` | "It **has** failed — reconstruct the post-mortem." | Klein's prospective hindsight (measurably out-produces "please critique this"), with an honest empty-handed exit. |
| `perimeter` | "Is this the **right problem**?" Category errors, not tuning. | Murder-board / red-team: the reviewer may attack the frame, not only the build. |
| `detection` | "Can each declared safeguard **fire**, and who notices its **silent death**?" | FMEA-detection × STAMP control-loop audit. |

A grid of historically frequent failure classes (built-never-wired,
guard-that-does-not-guard, test-theatre, silent-failopen, …) is injected as an
**additive** search aid — it extends the reviewers' search, it never limits it.
Output is aggregated by deduped, severity-ranked **findings** (a design review
has no binary claim to hold or fail), with cross-lens corroboration recorded.

On a first live run against a 5.7k-line retrieval core, the three no-claim
reviewers produced 11 findings that survived line-by-line verification with **0
false positives**: **7 genuinely new defects** (including a lock omission on the
exact path a claim-fed review had voted 3-0 clean the day before) and 4 that the
code itself already declared as known limits. A control arm given the proposer's
*true* claim put 3 of 3 findings inside that claim's subject, against 1 of 5 for
the same lens without it — the capture effect this contract removes, in the
predicted direction but **not statistically significant** at n=1 per arm
(Fisher exact p=0.14). Single run, and the reviewers ran on the same model
family as the proposer: this measures the *input contract*, not model
independence.

### Backends: design review needs an *agentic* one

All three design lenses read files (`Read`/`Grep`/`Glob`), so they carry
`requires_execution=True`. Consequence, stated plainly because it is easy to
miss: on a **reasoning-only** API backend (`openai_compat`, `anthropic_api`) a
design review runs **zero of three** reviewers — they are honestly reported as
`skipped: requires an agentic backend` and the report comes back `undecided`,
never fabricated. That is stricter than the post-fix triad, where the
`counterexample` reviewer still runs on any provider (1 of 3).

| Backend | offers | post-fix triad | design review |
|---|---|---|---|
| `claude_cli` / `ghost_cli` (subscription) | read + exec | 3 of 3 | **3 of 3** |
| **`agentic_api`** (Kimi, GLM, DeepSeek, …) | read; **exec per run** where the operator's `CRITIC_ALLOW_EXEC=1` + `CRITIC_EXEC_ROOTS` policy grants it | **3 of 3** with the opt-in (verified live, below); 2 of 3 without | **3 of 3** |
| `openai_compat` / `anthropic_api` | — | 1 of 3 | 0 of 3 |

With the opt-in, `falsification` runs through **pinned execution**: the model
sees ONE tool, `run_falsification_experiment`, that takes **no arguments** and
runs the experiment code already decided — the test at HEAD in an ephemeral
worktree (fix present, expected PASS) and at the pre-fix baseline with only the
test file taken from HEAD (fix absent, expected FAIL). The model's job is the
part that needs a model: telling the pinned assertion's failure from an
unrelated ImportError. Verified live on DeepSeek reviewing this repository's
own editable-install fix: 3/3 reviewers, `falsification` at 0.98 explicitly
distinguishing "genuine assertion error" from "import error or missing
dependency". A denied policy produces a skip whose message names the knobs —
never a reviewer that reasons where it should have observed.

Measured on real endpoints, one design lens over a small module:

| Provider | model | result |
|---|---|---|
| DeepSeek | `deepseek-chat` | pass, ~130 s |
| GLM | `glm-4.6` | pass, ~186 s (needed the wind-down: it first spent every step reading) |
| Kimi | `kimi-k3` | 2/3 before retries (the failure a 429), **3/3 with them** |

Reasoning models are slow here and their findings vary run to run — worth
knowing before wiring one into a gate.

**Retries, not streaming, are what fixed Kimi** — measured by changing one
factor at a time, because the first run changed both and could not
attribute the improvement:

| Kimi k3, 3 runs each | result | wall time |
|---|---|---|
| retries + `CRITIC_STREAM=1` | 3/3 | 315 s · 353 s · 472 s |
| retries, no streaming | 3/3 | 161 s · 502 s · **1112 s** |

So streaming is not what makes it succeed; it compresses the tail (worst
case 472 s vs 1112 s) and keeps the connection from sitting silent long
enough for an intermediary to reset it. Recommended for reasoning models,
not required.

`agentic_api` drives those same OpenAI-compatible endpoints as a real agent loop
with read-only filesystem tools, so the design lenses run on any provider with
tool calling — no coding-agent CLI in the middle, no extra dependency.

**Why the skip exists at all.** Each reviewer declares what it *needs*
(`WorkerSpec.needs`) and each backend declares what it *offers*
(`capabilities`); a reviewer whose needs exceed the offer is skipped, never
answered. `caller_verification` needs `read` (grep is its whole method) and
always runs. `falsification` needs `exec` — its method *is* run-the-test — so
without the operator opt-in a read-only sandbox refuses it. That refusal is
the point: left to run, the model would read the test, *reason* about whether
it would fail pre-fix, and answer `test_falsifies_master: true` having
executed nothing. A conclusion with no observation behind it is exactly the
confabulation this tool exists to catch, so no verdict is better than that
one. The `exec` capability is decided **per run**, never stored on the
backend: it exists only for a `project_dir` inside the operator's allowlisted
roots — the caller does not choose where this server executes; the operator
does.

### Deterministic audit (`audit_detectors.py`)

Some failure classes are bookkeeping, not judgment, and an LLM only adds
confabulation risk. Two grep-based detectors (no model, no network) feed the
design grid:

- **dead env flags** — capability gated behind an env flag defaulting OFF that
  no config, doc, script, or assignment ever sets (the built-never-wired class).
- **deviation register** — declared limitations (`KNOWN LIMIT`, `for now`,
  `FIXME`) collected *with their git-blame age*, so an old unrevisited deroga
  becomes a queue item instead of wallpaper. No silent caps: truncation is
  reported.

Exposed as the MCP tool **`run_repo_audit`** (returns immediately, no job), or
standalone: `python -m critic_orchestrator.audit_detectors <repo> --no-age`.

The detectors report *candidates* for a judge, not verdicts, and their blind
spots are declared: the flag detector models `os.environ.get()` / `os.getenv()`
(a subscript occurrence anywhere counts as a reference, so a blind spot never
reads as evidence of deadness), and it treats an `or`-fallback as the effective
default — a false positive found on the first live run and pinned by a test.

### Product probe (`start_product_probe`)

The design lenses read code; `falsification` runs the *author's* tests. Both
verify what someone already thought to check. The probe treats documentation
and packaging as a **contract** and executes it: shell fences in
README/USAGE/QUICKSTART/docs, `[project.scripts]` entry points (`--help` — the
testable promise is "exists and starts", not its real job with invented
arguments), and `python -m` targets with a `__main__`.

Each promise is then **kept or broken with the real output**. A documented
server keeps its promise by *staying up* through the grace period; exiting on
its own is the failure. Promises are extracted from a worktree at HEAD — an
uncommitted README line is not a promise yet — and executed there: argv only,
no shell, per-promise timeout with process-tree kill.

What is never executed, whatever the docs say: setup (`pip install`, …),
destructive and outbound commands, and anything that would *reintroduce a
shell* (`powershell -c`, `cmd /c`, `bash -c`, `env`, `xargs`, `eval`, …) —
argv-only execution is worthless if one argv element hands the shell back.
Execution requires the same operator opt-in as the falsification wiring
(`CRITIC_ALLOW_EXEC=1` + `CRITIC_EXEC_ROOTS`); without it the tool refuses
immediately with the knobs named. Async: `start_product_probe` returns a
`job_id`; poll and cancel with the existing review tools.

---

## Works with any coding agent

The tool is an **MCP server**, so any MCP-capable coding agent can call it
(Claude Code today; any client that speaks the Model Context Protocol). It is
a callable primitive, not a plugin tied to one host.

## Works with any provider

Each reviewer is run by a **backend**, selected via environment variables.
There is a hard, honest capability split:

| Backend | `CRITIC_BACKEND` | Can run | Notes |
|---|---|---|---|
| Claude CLI (agentic) | `claude_cli` (default) | **all 3 reviewers** | Has real Bash/Grep/Read tools, so it can run the test and grep call sites. Uses your Claude Code subscription — no API key. |
| Ghost CLI (agentic) | `ghost_cli` | **all 3 reviewers** | Windows-only. Each reviewer runs in a fresh **hidden interactive** Claude session (no `claude --print` anywhere), driven via the `clp ai-eye` console transport with a filesystem handshake. Stays on the flat subscription when headless calls become metered. |
| Any OpenAI-compatible API | `openai_compat` | `counterexample` (reasoning) | OpenAI, DeepSeek, Moonshot/Kimi, OpenRouter, Groq, local vLLM/Ollama. One shot, JSON-schema output. |
| Anthropic API | `anthropic_api` | `counterexample` (reasoning) | Native Messages API, structured output. |

**Why the split is honest, not a limitation hidden:** a plain chat-completion
API makes a single call — it cannot run `git stash`, `pytest`, or `grep`. So
the two reviewers that *require* executing commands are reported as
`skipped: requires an agentic backend` — **never faked**. The reasoning
reviewer (`counterexample`) runs on any provider. To run the full triad on a
non-Claude provider you need an *agentic* backend (an agent CLI with tools);
that adapter is a documented, in-progress extension point, and providers differ
in how they expose tool-calling — so it is added per-provider, verified against
the real endpoint, not guessed.

```bash
# Example: reasoning reviewer via DeepSeek
export CRITIC_BACKEND=openai_compat
export CRITIC_BASE_URL=https://api.deepseek.com
export CRITIC_MODEL=deepseek-v4-pro
export CRITIC_API_KEY=...        # your key
# structured output is auto-negotiated (json_schema → json_object → none)

# Example: via Anthropic API
export CRITIC_BACKEND=anthropic_api
export ANTHROPIC_API_KEY=...
export CRITIC_MODEL=claude-sonnet-5

# Example: full triad WITHOUT claude --print (Windows + clp arsenal)
export CRITIC_BACKEND=ghost_cli
export CRITIC_WORKER_MODEL=opus            # model pin for the sisters
export CRITIC_GHOST_MAX_SISTERS=4          # hard cap on hidden sessions
```

Temperature is not sent unless you set `CRITIC_TEMPERATURE` — some reasoning
models reject any value other than their default.

---

## Install

```bash
git clone https://github.com/aureliocpr-ctrl/critic-orchestrator.git
cd critic-orchestrator
python -m pip install -e .[dev]
python -m pytest tests -q        # 93 passed in ~3s
```

Requires Python ≥ 3.12 and `psutil`. The default Claude CLI backend also needs
the `claude` CLI on `$PATH`; the API backends use only the standard library.

## MCP setup

Register in `~/.claude.json` (or your agent's MCP config):

```json
{
  "mcpServers": {
    "critic-orchestrator": {
      "command": "python",
      "args": ["-m", "critic_orchestrator.mcp_server"],
      "env": {
        "PYTHONPATH": "/absolute/path/to/parent/of/critic_orchestrator",
        "PYTHONUTF8": "1"
      }
    }
  }
}
```

Five tools become available:

| Tool | When |
|---|---|
| `start_adversarial_review` | **Default.** Returns a `job_id`; you poll. |
| `poll_adversarial_review` | Check status / fetch the report. |
| `cancel_adversarial_review` | Abort a running review (kills the whole worker process tree). |
| `list_adversarial_reviews` | Inspect running + recent jobs. |
| `force_adversarial_review` | Legacy synchronous path; safe only for sub-60 s reviews. |

See [USAGE.md](./USAGE.md) for the full API contract and operational patterns.

---

## Caveats — read before enabling the falsification reviewer

- The `falsification` reviewer runs `git stash` and `git stash pop` on the
  working tree to test with and without the fix. **Do not run it while you
  have uncommitted edits in flight**, and prefer running it on a clean commit:
  a concurrent edit can collide with the stash. Committing first is the safe
  pattern.
- The falsification reviewer executes commands (test runner) with tool access
  pre-authorized (non-interactive mode has no prompt to confirm). Its tools are
  restricted by an allow-list, but treat it like any tool-enabled agent: run it
  on code you trust.

## Untrusted input handling

The `claim` and `diff_summary` you pass in are wrapped in explicit
data-only envelopes and sanitized (control characters stripped, envelope-escape
sequences neutralized, clipped to 4 KB) before reaching a reviewer. This is
defense-in-depth for the case where those fields originate from an untrusted
source (a commit message, a CI artifact). It is not a substitute for trusting
the caller — it removes the easy injection shapes, not the need for judgment.

---

## A note on prompt design (empirical)

The `counterexample` reviewer once carried a single instruction —
*"bias hard toward finding a counterexample"* — that pushed its false-positive
rate to **100%** in a controlled experiment (it invented problems that didn't
exist). Removing that line and adding an explicit false-positive checklist
brought it to **0%** with no measurable loss in real-bug recall. The raw runs
are in [`experiments/`](./experiments/) and the reasoning is in the
[CHANGELOG](./CHANGELOG.md).

The design principle it left behind: *a confabulated bug is worse than a missed
one.* The tool that checks for confabulation must hold itself to the same
standard — including not shipping "confidence-weighted voting" or "ensembles"
that the data showed didn't help.

---

## Verified

- Test suite: **417 green, 2 skipped** (`python -m pytest tests -q`).
- Claude CLI backend: end-to-end smoke via `smoke_live.py`.
- Ghost CLI backend: **live smoke 2026-07-02** — an execution reviewer ran a
  real `pytest` inside a hidden sister (verdict + verbatim pytest summary via
  the response file, 31.4 s wall), `MainWindowHandle == 0` on every sample
  while alive (true ghost), zero `claude.exe` left after the run (tree-kill).
- OpenAI-compatible backend: unit-tested (mocked transport) and **live
  smoke-tested against two providers with different quirks** — Moonshot Kimi
  `k2.7-code` (accepts `json_schema`, rejects `temperature != 1`) and DeepSeek
  `deepseek-v4-pro` (rejects `json_schema`, auto-falls-back to `json_object`).
  In both, the reasoning reviewer returned a correct verdict.
- Anthropic API backend: unit-tested (mocked transport); live use needs your key.

---

## Releasing

**Clean first, or you ship the tests.** `packages` excludes
`critic_orchestrator.tests`, but a stale `build/` or `*.egg-info/` from an
earlier build re-adds them from its own `SOURCES.txt` — measured: the wheel
came out at 174 KB with all 17 test files in it, carrying the security
fixtures' literal payloads into every user's `site-packages`. After the
clean it is 106 KB with none.

```bash
rm -rf build dist *.egg-info && python -m pip wheel . --no-deps --no-build-isolation --no-cache-dir -w dist
```

Then verify the artifact rather than the intention — unzip it, confirm no
`tests/` entries, and import every module from the extracted directory with
nothing else on the path. The motivating case for this whole tool includes
"a defect found by probing the built wheel that the entire internal suite
had missed".

---

## License

**MIT** © 2026 Aurelio Capriello. See [LICENSE](./LICENSE). Free to use, modify,
and redistribute — attribution appreciated.
