Metadata-Version: 2.4
Name: casebench
Version: 0.1.1
Summary: Measure the legal accuracy of an AI system against court decisions and lawyer-written references.
Keywords: legal,evaluation,llm,benchmark,citations
Author: Casebench maintainers
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Legal Industry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Dist: click>=8.5.0
Requires-Dist: httpx2>=2.13.1
Requires-Dist: jsonpath-ng>=1.10.0
Requires-Dist: mcp>=2.3.0
Requires-Dist: opentelemetry-proto>=1.45.1
Requires-Dist: packaging>=26.3
Requires-Dist: pydantic>=2.14.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: questionary>=2.1.1
Requires-Dist: rich>=15.0.0
Requires-Dist: scikit-learn>=1.9.1
Requires-Dist: scipy>=1.17.1
Requires-Dist: starlette>=1.7.0
Requires-Dist: uvicorn>=0.54.0
Requires-Python: >=3.11
Project-URL: Repository, https://github.com/Frontier-Legal/casebench
Project-URL: Issues, https://github.com/Frontier-Legal/casebench/issues
Project-URL: Changelog, https://github.com/Frontier-Legal/casebench/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# Casebench

Casebench measures the legal accuracy of an AI system against real court decisions and lawyer-written references, in any jurisdiction.

**Status:** early development. Casebench is on [PyPI](https://pypi.org/project/casebench/). It follows [SemVer](https://semver.org/), and while the version is 0.x any minor release may break the command line, the file formats or the Python interface. The changelog names every break.

## What it will do

- Build evaluation cases from court decisions a team already holds: the ruling, the statutes applied and the precedents relied on, each fact pointing at a passage in the decision.
- Take lawyer-written cases from a spreadsheet, for question answering, research, drafting, editing and extraction tasks.
- Check every citation in every answer against the corpus: does the authority exist, and does it say what the answer claims?
- Show where in a multi-step system the right authority was lost: retrieved, kept, read, cited, characterised.
- Report per task, per metric and per reference source, with the failing example beside every rate. There is no single accuracy number.

## Set it up

Paste this into the coding agent you use in your project:

> Set up Casebench for this project. Run `uvx casebench guide` and follow it.

The guide has the coding agent read your project and write two things: `casebench.yaml` and an adapter in `.casebench/` that starts your agent the way your production does. Nothing else in your project changes, and no key's value is read or written. It ends when `casebench check` passes, and it asks you one question: which judge model to pay for. Without a coding agent, `casebench init` asks the same things in the terminal.

## Jurisdictions

Casebench works in any country with no country-specific code. Models read the decisions and the answers, and your corpus resolves every citation, whatever its format. A data provider that serves several countries needs one connector. [PLUGINS.md](PLUGINS.md) explains how to add a data provider or a model API.

## Principles

- Nothing leaves your machines except calls to the system under test and the judge you configure. There is no telemetry.
- The grader never asks the system it grades.
- Every result reruns on your own corpus with the open-source code. A shared server, hosted or self-hosted, stores and shows results and never computes a score.
- Frontier Legal, which maintains Casebench, publishes no rankings of products, and neither does the project.

## Documentation

- [docs/design/](docs/design/) describes how Casebench works.
- [docs/decisions/](docs/decisions/) records why it is built that way.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). The rules for changing the code are in [AGENTS.md](AGENTS.md). Everyone taking part follows the [code of conduct](CODE_OF_CONDUCT.md). Report security issues as [SECURITY.md](SECURITY.md) describes.

## Licence

Apache-2.0, except the `ee/` directory, which holds the multi-user team server under its own source-available licence. [LICENSE](LICENSE) states the boundary.
