Metadata-Version: 2.4
Name: routeval
Version: 0.0.1
Summary: Evals for AI pipelines that route: cost per correct decision, deflection quality, lucky-hit rate
Project-URL: Homepage, https://nicopurnomo.me/11-Quorum
Author: Nico Purnomo
License: MIT
License-File: LICENSE
Keywords: agents,cascade,cost,evals,llm,routing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: pretty
Requires-Dist: rich>=13; extra == 'pretty'
Description-Content-Type: text/markdown

# routeval

**Evals for AI pipelines that route.**

For pipelines with more than one path — a model cascade, an abstain, a human
handoff, a cache — routeval replays your golden set through your *real*
pipeline and tells you whether the routing earned its money: **cost per
correct decision**, deflection quality, and the **lucky hits** your accuracy
number is hiding.

> Why not promptfoo or DeepEval? They score outputs. They can't tell you that
> 40 of your 184 wins took the wrong path to get there.

## Status

Pre-v0.1 — planning complete, build in progress. The
[launch essay](content/launch-essay.md) is the spec; the
[run.json schema](docs/run-schema.md) is the API.

## The contract (all of it)

You write one function. It returns an answer, the route it took, and what it
cost (or model + tokens, and routeval prices it). You point the harness at a
JSONL golden set of `{input, expected}` — optionally with `expected_route`.
You get a terminal report, a versioned `run.json`, and CI exit codes.

Metrics are tiered by what you've labelled:

| Tier | Needs | You get |
|---|---|---|
| 1 | expected answers only | per-route accuracy/cost/latency, cost-per-correct-decision, deflection rate, error accounting |
| 2 | + `expected_route` labels | routing 2×2, lucky-hit rate, escalation precision/recall |
| 3 | (v0.2) nothing — spends budget | counterfactual routing: force-run sampled items down the other path, measure what it buys |

Errors are never swallowed: accuracy is computed over *attempted* items, and
failures are listed. (If your pipeline crashed on 10% of the golden set,
that's a result, not an inconvenience.)

## Hello world (zero API keys)

```
pip install routeval
routeval init      # scaffolds a toy two-route pipeline + 20-item golden set
routeval run       # full routing-economics report, no keys required
```

## What this is not

Not a framework. Not a judge. Not a dashboard. Not hosted. It composes with
pytest, promptfoo, and whatever judge you already trust.

MIT. Extracted from [Quorum](https://nicopurnomo.me/11-Quorum), where these
metrics killed a threshold-tuning project in one afternoon (22% lucky-hit
rate, a flat cost-quality frontier, and a 0.44-accurate escalation stage that
no threshold could fix).
