Metadata-Version: 2.5
Name: llm-panel
Version: 0.1.1
Summary: Put one question to several LLMs independently and read every answer in full -- parallel judges, failure classes, cost, and an anonymized rebuttal round
Project-URL: Homepage, https://github.com/musharna/llm-panel
Project-URL: Issues, https://github.com/musharna/llm-panel/issues
Project-URL: Changelog, https://github.com/musharna/llm-panel/blob/main/CHANGELOG.md
Author: Jaret Arnold
License-Expression: MIT
License-File: LICENSE
Keywords: code-review,judge,llm,multi-model,ollama,openrouter,panel
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown

# llm-panel

[![controls](https://github.com/musharna/llm-panel/actions/workflows/controls.yml/badge.svg)](https://github.com/musharna/llm-panel/actions/workflows/controls.yml)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)

Put the same question to several models independently, then read every answer in full.

Judges run in parallel, never see each other's work, and answer from their own reading of
your repo. An optional second round shows each of them the others' findings — anonymised —
and asks them to defend or withdraw. The output is a single self-contained HTML page where
that second round is grouped **by the finding being argued about**, so comparing what five
models said about one line of code doesn't mean holding five documents in your head.

It is not a voting machine. A panel _generates candidate defects_; it does not establish
truth by counting agreements. Every finding still has to be checked against the code — and
the tool's other half, [`recall/`](#what-it-actually-catches), exists to measure what the
panel misses rather than assert what it catches.

```
llm-panel --diff "Which of these changes is most likely to be wrong?"
panel-report --open          # render the newest run and open it
panel-triage --bad           # which runs went wrong, across every run root
```

A real run, with the waiting compressed: three judges asked in parallel (two free-tier,
one on a ChatGPT plan) landing as they finish, the scoreboard from `panel.md`, and
`panel-report` rendering it ([recording](https://github.com/musharna/llm-panel/blob/main/docs/demo.cast)):

![terminal: llm-panel asks three judges in parallel, reports each as it lands, then head -n 12 panel.md shows the scoreboard and panel-report writes the HTML page](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/demo.gif)

The rendered report — the scoreboard counts spend and names who answered; the
citation-overlap tables show where the panel's attention landed (three judges reviewing a
[cline](https://github.com/cline/cline) PR, converging on one line of `TerminalProcess.ts`):

![panel report: scoreboard, bench, and citation-overlap tables](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/report.png)

**Contents:** [What's here](#whats-here) · [Install](#install) ·
[Configure your roster](#configure-your-roster) · [Using it](#using-it) ·
[What it actually catches](#what-it-actually-catches) ·
[On real PRs](#on-real-prs-aacr-bench) · [Tests](#tests) ·
[Known limitations](#known-limitations)

## What's here

```mermaid
%%{init: {"theme": "neutral", "flowchart": {"wrappingWidth": 320}}}%%
flowchart TB
  Q["question  (+ --diff, or stdin)"] --> LP["llm-panel"]
  LP --> A["judge A"]
  LP -- "in parallel<br/>blind to each other<br/>reading your repo" --> B["judge B"]
  LP --> C["judge C"]
  A & B & C --> RUN["run directory<br/>one .md and one .prompt.md per judge,<br/>panel.md, run.json"]
  A & B & C -.-> RB
  RB["round 2, only with --rebut<br/>each judge sees the others' findings,<br/>anonymised as Reviewer A / B / C,<br/>and defends or withdraws"] -.-> RUN
  style RB stroke-dasharray: 6 4
  RUN --> REP["panel-report<br/>one self-contained HTML page,<br/>rebuttals grouped by finding"]
  RUN --> TRI["panel-triage<br/>the runs that failed"]
```

| tool                   | what it does                                                                           |
| ---------------------- | -------------------------------------------------------------------------------------- |
| `llm-panel`            | asks the judges, in parallel, and writes the run to disk                               |
| `panel-report`         | renders a run as one self-contained HTML page, grouped by claim                        |
| `panel-triage`         | finds the runs that _failed_, which a listing shows as ordinary rows                   |
| `recall/panel-recall`  | measures what the panel **misses**, against a corpus of planted defects                |
| `recall/aacr-upstream` | runs the panel over AACR-Bench PRs and hands the findings to **upstream's** evaluator  |
| `recall/aacr-score`    | invokes that evaluator, and refuses to report a number from a judge that isn't running |
| `claimlib.py`          | the one measurement boundary: reviews → span-grounded observations                     |
| `*-controls`           | the regression suites — 810 controls, every one tied to a defect that shipped          |

## Install

Pure Python 3.11+ standard library. No dependencies, no build step. Each tool is one
readable file, so either install route runs identical code:

```sh
# as a package (entry points: llm-panel, panel-report, panel-triage)
uv tool install llm-panel        # or: pipx install llm-panel

# or as the files themselves
git clone https://github.com/musharna/llm-panel ~/llm-panel
ln -s ~/llm-panel/{llm-panel,panel-report,panel-triage} ~/.local/bin/
```

3.11 is a hard floor (the link renderer uses atomic groups, added to `re` in 3.11);
`panel-report` says so at startup rather than failing part-way through a render.

Judges reach models through command-line tools you install separately — none are bundled,
and you need at most one to start:

| tool       | who it is                                    | billing                                                                       |
| ---------- | -------------------------------------------- | ----------------------------------------------------------------------------- |
| `codex`    | OpenAI's CLI                                 | a ChatGPT plan, not metered API                                               |
| `opencode` | multi-provider CLI most judges route through | your OpenRouter / HuggingFace keys                                            |
| `claude`   | Anthropic's CLI                              | a claude.ai subscription (setting `ANTHROPIC_API_KEY` switches it to metered) |
| `ollama`   | local models                                 | free, and no tool loop — see the caveat below                                 |

If none are present the panel still runs, fails loudly, exits 4, and tells you what to
install. A missing tool is one judge's problem, never the whole panel's.

### The read-only agent for `opencode` judges

`opencode` judges run as an agent named `panelist` that can read the repo but not write
to it, and `llm-panel` refuses to start an opencode judge until that agent is defined and
verified read-only (exit 9) — opencode's default `build` agent will happily edit the tree
it is reviewing. The definition ships as [`opencode.jsonc`](opencode.jsonc): merge its
`agent.panelist` block into `~/.config/opencode/opencode.jsonc`, or keep the file at the
root of a repo you review with it — opencode reads project-local config too.

## Configure your roster

**The built-in judge list is a default, not a fixture — it names the author's accounts.**
Yours will be different. Point the roster at models you actually have:

Copy [`roster.example.json`](roster.example.json) to
`~/.config/llm-panel/roster.json` (`$XDG_CONFIG_HOME` honoured; `$LLM_PANEL_CONFIG` wins).
It is strict JSON — no comments, no trailing commas — and a malformed config is **fatal**
and names the offending key, because quietly falling back to the built-in roster would run
a panel you didn't ask for, and bill you for it:

```json
{
  "default": ["codex", "nemotron", "glm", "kimi", "or-deepseek", "or-grok"],
  "judges": {
    "my-gpt": {
      "transport": "opencode",
      "model": "openrouter/openai/gpt-5.6",
      "family": "OpenAI"
    },
    "big-pickle": null
  }
}
```

`null` drops a shipped judge. `default` is the panel run when `--judges` is absent.
`llm-panel --list` shows the roster offline and marks config-defined judges.
`llm-panel --check` actually pings each one. `llm-panel --help-config` prints this schema.

### Picking judges

**"One per vendor" is _not_ the answer.** It is tempting to
treat vendor labels as a proxy for independent opinions. The evidence says they aren't:
[Kohli 2026](https://arxiv.org/html/2605.29800) measured cross-family judge correlation at
φ̄=0.389 against same-family 0.437 — barely different — with the three _most_ correlated pairs
being cross-family, and found that restricting to one judge per family made effective
independence **worse** (n_eff 1.93 vs 2.18). Family is display metadata here, not policy.

The six-judge set above did score 6/6 against this corpus where a two-vendor panel scored
4/6, but **treat that as debugging evidence, not as a result**: the roster was repaired
_because_ of what happened on those very fixtures, so the comparison is in-sample, and the
six defects live in only two files (effective n≈2, 95% CI 61–100%).

What to actually do: pick judges by what they find on _your_ code, and use `panel-recall` to
measure it. The quantity worth maximising is each judge's **marginal rescue rate** — how
often it catches something every other judge on the roster missed — not how many logos are
represented.

Two practical constraints: wall-clock is the **slowest** judge, not the sum, so one slow
model sets the pace for every run; and `claude-*` are deliberately absent from the default,
because when Claude wrote the code under review a Claude judge shares the author's blind
spots. Add it explicitly when that isn't the case — it is strong.

## Using it

- `--diff` attaches the working-tree diff, so nobody has to describe the change —
  including you, who would describe it favourably.
- `--rebut` adds the anonymised second round. Worth it whenever a finding would trigger
  real work: the first run of it killed three confident findings that were simply wrong.
  To be precise about the report's grouping of that round: it keys on the rebuttal letter
  each finding is given (A1, B2 …), so it collects the _discussion_ around one judge's
  finding. It is **not** semantic clustering — two judges independently raising the same
  underlying defect stay two findings, and without `--rebut` there is no grouping at all.
- `--judges a,b,c` overrides the default panel. `codex~2` runs the same model a second
  time as a **full, separate judge** — its own file, its own letter, its own row.
  Collapsing repeats would hide exactly the disagreement that makes them worth running.
- `--thread NAME` keeps a persistent conversation per judge. For design questions, not review.
- Long questions go via stdin: `llm-panel - <<'ASK' … ASK`.

Judges reading through `codex`/`opencode`/`claude` can **read your repo**. `ollama` judges
answer from the prompt alone with no tool loop, so they cannot verify a claim against code.
Treat their findings accordingly.

The rebuttal round as rendered — every position each judge took on each finding, grouped
by the finding under dispute, disagreements marked CONTESTED. This run: four free-tier
judges asked to review llm-panel's own failure-classification code; one failed and is
reported as `harness`, the other three upheld 7 findings, rejected 4, and missed 6:

![rebuttal round: positions grouped by the finding being argued about](https://raw.githubusercontent.com/musharna/llm-panel/main/docs/rebuttal.png)

## What it actually catches

`recall/panel-recall` is the part most tools like this don't have: a corpus of defects
planted in real code, each one **proven to misbehave by execution**, so "the panel missed
it" is a measurement rather than an impression.

> **At least one of four independent passes (codex ×2 + claude-opus ×2) matched 25 of 27
> known targets in this controlled, single-file Python corpus.** That is a keyword-matched
> lower bound on an easy corpus — not an estimate of real-world code-review capability.
> 95% CI 76.6–97.9%, and that is before accounting for defects clustering within fixtures.

**Read that next to a real-world number.** [CR-Bench](https://arxiv.org/html/2603.11078v1)
(Nutanix, 2026) builds review tasks from _real_ bugs `git blame`d out of merged PRs in
django, sympy, astropy and scikit-learn, and reports GPT-5.2 + Reflexion at **32.8% recall
and 5.1% precision**. The gap between that and 25/27 is the corpus, not the panel: hand-
planted single-mechanism defects in ~40-line files are far easier than real defects in
mature codebases, and the two numbers are not even the same estimand — different agents,
different context, different definitions of a hit.

So this corpus is a **development instrument**, good for controlled A/Bs where ground truth
must be known and iteration must be cheap (the abstention experiment below is exactly that).
It is not evidence of absolute capability, and no number from it should be quoted as one.

Three results worth knowing before you trust any of the output:

- **Recall was limited by the roster, not by the models.** The two defects that panel
  never found — a `.get(k, default)` that doesn't apply to an explicit `null`, and a
  corrupt cache file silently becoming empty — are both found by a **six-vendor** panel
  (OpenAI / NVIDIA / Zhipu / Moonshot / DeepSeek / xAI): 4/6 → **6/6** on those two
  fixtures. The best two judges there, at 4/6 each, beat codex at 2/6 — and both were
  broken or out of credit until the roster was repaired. If your panel is missing things,
  check who is actually answering before concluding the models can't see it.
- **Running the same model twice recovered nothing.** First passes 25/27, with repeats
  25/27. The repeat-passes idea is well supported in the literature and did not reproduce
  here. An earlier grader bug reported +1 and it was an artifact. Adding a _different
  vendor_ did what adding a second pass of the same one could not.
- **Letting judges say "nothing is wrong here" is a precision/recall trade, not a free
  win either way.** One sentence of abstention licence is the whole difference.

  |             | findings/fixture | false positives       |
  | ----------- | ---------------- | --------------------- |
  | licence on  | 0.42             | 0 / 6 judges          |
  | licence off | 2.17             | 2, from 1 of 5 judges |

  Findings-per-fixture is measured on fixtures that _do_ contain defects, where the extra
  findings were verified **true** — so the licence suppresses real findings (one judge went
  3.00 → 0.00 on files with genuine defects). False positives are measured on
  `p01-exhaustive-codec`, the one fixture with **proven** absence rather than verified
  scope — which is what makes a false-positive rate computable at all. There, the same
  judge on the same code abstained with the licence and produced two demonstrably false
  findings without it (it claimed int and str subclasses were rejected; `encode(MyInt(1))`
  returns `'A'`).

  So: the licence costs true findings and prevents false ones. Which you want depends on
  whether chasing a false lead costs you more than missing a real defect. Caveat worth
  stating: one proven fixture, eleven reviews.

## On real PRs (AACR-Bench)

The planted corpus above is a development instrument; the real-world numbers come from
running the panel over [AACR-Bench](https://github.com/alibaba/aacr-bench) PRs and
scoring the findings with **upstream's own evaluator** — a real LLM judge doing
path → line → semantic matching, so the numbers are theirs, not a self-graded matcher's.
On 18 PRs at full roster (extractor-3 re-measurements, 2026-08-28):

| `--prompt-style`   | semantic recall | precision | findings read per validated hit |
| ------------------ | --------------- | --------- | ------------------------------- |
| `defect` (default) | 12.2%           | 16.5%     | 6.1                             |
| `broad`            | 26.0%           | 13.2%     | 7.6                             |
| `volume`           | 25.2%           | 7.9%      | 12.6                            |

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/musharna/llm-panel/main/docs/bench-dark.png">
  <img alt="recall against precision for the three prompt styles; error bars are the ±2 pp re-run noise floor" src="https://raw.githubusercontent.com/musharna/llm-panel/main/docs/bench-light.png" width="660">
</picture>

`broad` — asking for what a careful maintainer would actually raise — doubles the
default's recall (McNemar on paired references, p = 0.0005). But the `volume` control
shows what that class of gain is made of: it is the `defect` prompt plus one
exhaustiveness clause, reaches the same recall (p = 1.0 vs broad), and pays for it with
half of broad's precision. On a 35-PR replication the ordering holds on both transports
while every arm's precision falls (broad ~9.7%, volume ~5.5–6.1%, ~16–18 findings read
per hit). A declared cost cut over all of it settled the product default: **it stays
`defect`**; the only candidate for a future default change is `broad`
(`recall/benchmarks/cost-cut/README.md`).

What keeps these numbers honest:

- **The variance floor is measured.** Re-running the same judge on the same 35 PRs moves
  up to ±3 human-reference matches of 150, with an evaluator replicate at exactly zero —
  so effects under ~5–7 pp of recall are re-run noise at this n, which every subgroup
  claim so far was (`recall/benchmarks/results-human-2arm-orgpt/perjudge35/`).
- **Three earlier readings were withdrawn on re-measurement**: a DEFECT/IMPROVEMENT
  split (my classifier was circular), "broad finds different hits" (pre-registered
  replication on 35 fresh PRs, p = 0.40), and a transport/harness effect (its 13-PR
  foothold did not survive a re-run; a same-transport re-run of another judge moved as
  much). The audit trail is in the benchmark READMEs; nothing above rests on a withdrawn
  claim.
- **Diff-in-prompt review is the measured condition** — each panel runs in an empty
  directory with the diff in the prompt. The paired repo-checkout arm moves recall
  12.2% → 15.4% (p = 0.48) while _losing_ 7 of the diff arm's matches and gaining 11:
  repo access changes what judges attend to more than it strictly adds
  (`recall/benchmarks/results-checkout-3judge/README.md`).
- **Location agreement overstates semantic agreement ~2x** (22.8% of references had a
  finding at the right file and line; 12.2% had one a judge called the same concern) —
  which is why scoring is delegated upstream instead of done by a local matcher.
- **A degraded roster costs about half the recall** (6.5% vs 12.2% with one judge's
  quota spent and a 300s timeout, same extractor). Check who actually answered before
  reading any number.
- **The panel does not discriminate accepted from rejected reviewer comments**
  (12.2% vs 11.1%, Fisher p = 1.0).
- **Unlocated findings are withheld from upstream, not handed over empty** — upstream's
  filters treat a missing path or line as match-everything, and passing them through
  inflated line matches from 20 to 50 on the first scoring run.

Full run ledgers: `recall/benchmarks/results-human-2arm/README.md` and
`results-human-2arm-orgpt/README.md`; data licensing in
`recall/benchmarks/PROVENANCE.md`.

## Tests

```sh
./claimlib-controls              #  83
./llm-panel-controls             # 279
./panel-report-controls          # 310
./panel-triage-controls          #  15
./recall/aacr-upstream-controls  #  96
./recall/aacr-recut-controls     #  27
cd recall && ./panel-recall selftest && python3 validate_corpus.py
```

CI runs all six suites on every push (Python 3.11 and 3.13).

Every control corresponds to a defect that **shipped**, and each asserts the fixed
behaviour _and_ — where the pre-fix input is representable — that the broken version would
have failed on it. An assertion that passes on both the broken and the fixed code tells you
nothing.

## Known limitations

- **The judge roster's shipped defaults will not work for you** until you configure it.
- **Judges can read the working tree.** `--diff` sends untracked file contents to remote
  APIs. Don't point it at a repo holding secrets you haven't gitignored.
- **A panel is not a jury.** Independent models generate candidates; verification against
  code, tests, and execution is still yours to do.
- **Recall is measured on a 27-defect Python corpus.** That number does not transfer to
  other languages or to defect classes the corpus doesn't contain.
- **The headline recall numbers are prompt- and condition-specific.** They move with
  `--prompt-style`, roster health, and diff-vs-checkout context — see
  [On real PRs](#on-real-prs-aacr-bench) before quoting any of them.
- **Every other fixture has verified _scope_, not proven absence.** Their known unplanted
  defects are recorded in each `truth.json` and re-checked by execution in
  `validate_corpus.py`, so a judge that finds one is not scored as wrong. Anything not yet
  recorded still depresses the recall floor by making a true finding look like noise.

## License

MIT — see [LICENSE](LICENSE). The MIT grant covers the code in this repository; the
benchmark data under `recall/benchmarks/` contains third-party material (PR diffs and
review-comment text) that stays under its upstream terms — see
[recall/benchmarks/PROVENANCE.md](recall/benchmarks/PROVENANCE.md).
