Metadata-Version: 2.5
Name: pergamon-bench
Version: 0.8.0
Summary: Pergamon Bench: the OpenCode agent wrapper for Harbor, with the grader and job configurations
Project-URL: Dataset, https://github.com/pergamon-ai/pergamon-bench
Project-URL: Source, https://github.com/pergamon-ai/pergamon-bench-eval
Author: Pergamon
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: benchmark,evaluation,harbor,law,llm
Requires-Python: >=3.12
Requires-Dist: harbor-rewardkit<0.3,>=0.2.1
Requires-Dist: harbor<0.24,>=0.23.0
Requires-Dist: httpx>=0.27
Requires-Dist: litellm>=1.92.0
Requires-Dist: pydantic>=2.12
Requires-Dist: pyyaml>=6.0
Requires-Dist: tenacity>=9.1
Provides-Extra: dev
Requires-Dist: pbench-search; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Description-Content-Type: text/markdown

# Pergamon Bench — eval

The code behind [Pergamon Bench](https://github.com/pergamon-ai/pergamon-bench), a benchmark of
legal reasoning and research tasks on continental-European law, run with
[Harbor](https://github.com/laude-institute/harbor).

**To run the benchmark, start with the
[Pergamon Bench README](https://github.com/pergamon-ai/pergamon-bench#readme).** It covers
installing, running, fitting the run to your machine, results and `pergamon-bench validate`.

This repository holds:

- `pergamon-bench` (`src/pergamon_bench/`): the agent wrapper Harbor loads (OpenCode with the
  official settings), the job file `pergamon-bench init` writes, and the `pergamon-bench`
  command;
- `pbench-grader` (`packages/pbench-grader/`): the grading;
- `pbench-search` (`packages/pbench-search/`): the agent's web search tool.

Documentation for contributors and the curious, in [docs/](docs/):

- [architecture.md](docs/architecture.md): how the pieces fit together, and what "official" means in code;
- [grading.md](docs/grading.md): how answers are graded, and every grading setting;
- [grading-versions.md](docs/grading-versions.md): what each grading version pins, and why it changed;
- [methodology.md](docs/methodology.md): the grader card;
- [failures-and-retries.md](docs/failures-and-retries.md): which failures are retried instead of graded;
- [web-search.md](docs/web-search.md): the web tools and search providers;
- [images.md](docs/images.md): the sandbox and grading images;
- [private-set-process.md](docs/private-set-process.md): evaluating on the private set;
- [development.md](docs/development.md): tests, lint and releases.

`CHANGELOG.md` lists every release; changes that can move scores are marked **[score-impact]**.

## Releasing to PyPI

The `pergamon-bench` package (one wheel: `pergamon_bench` and `pbench_grader`) is published by
the `release` workflow on every `v*` tag, through PyPI trusted publishing (no token). Bump
`version` in `pyproject.toml` and `packages/pbench-grader/pyproject.toml` to the same value,
update `CHANGELOG.md`, push, then:

```bash
gh release create v0.8.0 --title "v0.8.0" --notes "See CHANGELOG.md"
gh run watch
```

The workflow refuses a tag that does not match `pyproject.toml`'s version. The images
(`workspace-v*`, `verifier-v*` tags) are released separately: see [images.md](docs/images.md).
