Metadata-Version: 2.4
Name: refereed
Version: 0.1.0
Summary: Run a panel of seven independent AI referees over a research paper, then synthesise their reports into one recommendation.
Author: Abhishek Shivakumar
License: MIT
Project-URL: Homepage, https://github.com/godofecht/refereed
Project-URL: Documentation, https://github.com/godofecht/refereed/blob/main/README.md
Project-URL: Issues, https://github.com/godofecht/refereed/issues
Keywords: peer-review,research,papers,claude,review
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: ruff; extra == "dev"

# refereed

Run a panel of seven independent AI referees over a research paper, then
synthesise their reports into one recommendation.

```bash
pip install "refereed[anthropic]"
refereed review paper.md
```

```
My paper
==========================================================
  ✓  Correctness reviewer             7/10  minor_revisions
        major: Lemma 2 assumes independence that Section 3 does not establish
  ✦  Novelty reviewer                 6/10  major_revisions
  ⚙  Methodology reviewer             7/10  minor_revisions
  ❖  Citations reviewer               8/10  accept
  ↻  Reproducibility reviewer         5/10  major_revisions
        major: No code or data released; results cannot be checked
  ✎  Writing & clarity reviewer       8/10  accept
  ⚔  Adversarial (Reviewer 2)         5/10  major_revisions

  weighted score: 6.53/10
  recommendation: major_revisions

  Revisions requested:
    1. State the independence assumption in Lemma 2 explicitly.
    2. Release the experiment code, or specify conditions precisely enough to rerun.
```

## Why seven reviewers

A single "review this paper" prompt produces a diffuse answer that hedges
across every dimension at once. Seven narrow reviewers each have to commit to a
position, and the disagreements between them carry information a single pass
throws away. The synthesiser reads all seven and reports where they converged
and where they split.

| Reviewer | Remit | Weight |
|---|---|---|
| Correctness | Claims, internal logic, the dependency graph of the argument | 1.0 |
| Novelty | What is genuinely new against the surrounding literature | 0.9 |
| Methodology | Baselines, controls, ablations, definitions, honest limitations | 1.0 |
| Citations | Whether the bibliography does real work; pairs with `citeverify` | 0.7 |
| Reproducibility | What an independent researcher would need to verify the claims | 0.9 |
| Writing | Structure, hedging balance, jargon, clarity for the intended reader | 0.6 |
| Adversarial | The strongest hostile reading, stated concretely | 1.0 |

Weights reflect what is fixable. Correctness, methodology and the adversarial
reading bear on whether the work is right. Prose can be rewritten, so writing
carries least.

The **weighted score is computed locally** from the reviewer scores.
The synthesiser does not supply it. It is reproducible from the JSON output and cannot drift
with the model's mood. The synthesiser's own score is kept alongside it.

## Usage

### CLI

```bash
refereed review paper.md                          # full panel
refereed review paper.md --json review.json       # machine-readable output
refereed review paper.md --only correctness,adversarial
refereed review paper.md --effort medium          # cheaper bulk pass
refereed review paper.md --context cites.json     # feed in citeverify output
refereed review paper.md --strict minor_revisions # exit 1 if below that bar
refereed reviewers                                # list the panel, no API call
```

### Python

```python
import asyncio
from refereed import review_paper, anthropic_complete

review = asyncio.run(review_paper(
    open("paper.md").read(),
    anthropic_complete(effort="high"),
    title="My paper",
))

print(review.weighted_score)
for r in review.reports:
    if r.ok:
        print(r.name, r.data["score"], r.data["recommendation"])
```

## Failure behaviour

A panel is useful even when part of it fails, so failures are recorded rather
than raised:

- A reviewer that errors or returns unparseable output is marked failed; the
  other six still report, and the failure is visible in the output.
- The weighted score is computed from the reviewers that succeeded.
- A failed synthesis leaves the seven reports intact.

The raw text of an unparseable response is kept so you can see what happened.

## Bring your own backend

Model access goes through a single injectable callable. Nothing else in the
package touches the network:

```python
async def my_complete(system: str, user: str) -> str:
    return await my_gateway.complete(system=system, user=user)

review = await review_paper(paper, my_complete)
```

Point it at an internal gateway, a different provider, or a recorded fixture.
The entire test suite runs through this seam, so it is a supported path.

## Model

Defaults to Claude Opus 4.8 with adaptive thinking, so the model decides how
much to reason per reviewer instead of working to a fixed token budget.
`--effort` controls depth and therefore most of the cost: `high` by default
because reviewing is intelligence-sensitive, `medium` for cheap bulk passes.

The core package is **stdlib-only**. The Anthropic SDK is an extra, needed only
for live reviews, so the panel definitions, prompt construction and parsing can
be imported and tested without it.

## Limits, and what this is not

**This is a reading aid.** It does not replace referees, and
its output must never be presented as though it had been through peer review.

- It reads what you give it. It cannot check a claim against an experiment it
  cannot run, or verify data it cannot see.
- Novelty assessment is bounded by the model's knowledge. It will miss recent
  or obscure prior art. Treat a novelty claim as a prompt to go and
  search.
- Scores are ordinal at best. Use the concerns and the revision list; treat the
  number as a sort key.
- Runs cost money and are non-deterministic. Two runs on the same paper will
  differ in wording and may differ in score.

Use it to find what you would have missed on your own reread, before a human
referee finds it.

## Provenance

Developed while reviewing a corpus of roughly 105 papers, and paired with
[`citeverify`](https://github.com/godofecht/citeverify) for the citations
reviewer. Extracted from tooling that had to work at that scale.

## Development

```bash
pip install -e ".[dev]"
pytest        # 27 tests, all offline. No API key required.
```

## Licence

MIT.
