Metadata-Version: 2.4
Name: refereed
Version: 0.2.0
Summary: Multi-perspective review of research papers. Open panel runs on your own model key; a calibrated panel is available as a hosted service.
Author: Abhishek Shivakumar
License: MIT
Project-URL: Homepage, https://github.com/godofecht/refereed
Project-URL: Pricing, https://quilio.dev/pricing
Project-URL: Issues, https://github.com/godofecht/refereed/issues
Keywords: peer-review,research,papers,manuscript,claude,llm
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# refereed

Multi-perspective review of research papers. Several reviewers read the paper
independently, then a synthesiser combines their reports into one
recommendation.

There are two panels. Both run through the same engine, so code written
against one works against the other.

| | Open panel | Hosted panel |
|---|---|---|
| Reviewers | 3 (claims, method, clarity) | 7 narrow reviewers |
| Weights | Equal | Per-reviewer, calibrated |
| Calibration | None | Tuned against a corpus of ~105 papers |
| Runs on | Your machine | Quilio infrastructure |
| Model key | Yours | Included |
| Account | None | Quilio API key |
| Licence | MIT | Commercial |

## Install

```bash
pip install "refereed[anthropic]"
```

Python 3.9 or later. The engine and the open panel have no third-party
dependencies; the `anthropic` extra is only needed to run the open panel
against Claude.

## Open panel

Runs locally against your own model key. No account, no rate limit.

```bash
export ANTHROPIC_API_KEY=...
refereed review paper.md
refereed reviewers            # list the panel, no API call
```

```python
import asyncio
from refereed import anthropic_complete, review_paper

review = asyncio.run(review_paper(
    open("paper.md").read(),
    anthropic_complete(effort="high"),
    title="Paper title",
))

review.weighted_score
[r for r in review.reports if r.ok]
review.meta["revision_list"]
```

## Hosted panel

Seven narrow reviewers with calibrated weights and a separate synthesiser.
Requires a Quilio API key: https://quilio.dev/pricing

```bash
export QUILIO_API_KEY=qk_...
refereed review paper.md --hosted
```

```python
from refereed import review

r = review(open("paper.md").read(), api_key="qk_...")
r.weighted_score
r.recommendation
r.major_concerns          # each tagged with the reviewer that raised it
```

`QuotaExceeded` is raised when the period's quota is spent. `ReviewError`
covers transport and API failures.

## Scoring

`weighted_score` is the weight-weighted mean of reviewer scores over reviewers
that returned parseable output:

```
sum(weight_i * score_i) / sum(weight_i)
```

It is computed from the returned JSON and is reproducible from it. In the open
panel the weights are equal, so this reduces to a plain mean.

## Failure semantics

- A reviewer that raises, or returns output with no JSON object, is recorded
  with `ok=False` and an error string. The other reviewers still run.
- `weighted_score` is computed over successful reviewers only.
- Unparseable output is kept in `ReviewerReport.raw`, truncated to 2000
  characters.
- The synthesiser runs only if at least one reviewer succeeded. A failure there
  sets `meta_error` and leaves the reports intact.
- `extract_json` accepts a bare object, a fenced block, or an object embedded
  in prose. It rejects a top-level array.

## Bring your own backend

Model access goes through one callable, `async (system, user) -> str`. Nothing
else in the open panel performs I/O.

```python
async def complete(system: str, user: str) -> str:
    return await gateway.complete(system=system, user=user)

review = await review_paper(paper, complete)
```

The hosted client takes an injectable `transport` for the same reason.

## Limits

- Assessment is bounded by the model's knowledge. It does not perform a
  literature search, so treat any novelty claim as a prompt to go and check.
- It reads the text you supply. It cannot run experiments or inspect data it
  has not been given.
- Output is non-deterministic. Two runs on the same paper differ in wording and
  may differ in score.
- Scores are ordinal. The concerns and the revision list carry the detail.
- Output is model-generated. It is a reading aid before submission and carries
  no editorial standing.

## Related

- [citeverify](https://pypi.org/project/citeverify/): checks whether the
  references in a bibliography exist, against OpenAlex, Crossref and arXiv.
- [corpus-lens](https://pypi.org/project/corpus-lens/): keywords, rhetorical
  fingerprints, similarity and embeddings across a whole corpus.

`citeverify` output can be passed to either panel with `--context`.

## Development

```bash
pip install -e ".[dev]"
pytest
```

40 tests, all offline. No model key or Quilio key required; the model backend
and the HTTP transport are both injected.

## Licence

MIT for this package. The hosted panel is a commercial service.
