Metadata-Version: 2.4
Name: benchrail-ai
Version: 0.1.0
Summary: Context-derived, calibrated, evolving evals for LLM/agent systems
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: anthropic>=0.111.0
Requires-Dist: openai>=2.0.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pyyaml>=6.0.3
Description-Content-Type: text/markdown

# Benchrail

[![Tests](https://github.com/JeffBrines/benchrail/actions/workflows/test.yml/badge.svg)](https://github.com/JeffBrines/benchrail/actions/workflows/test.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](pyproject.toml)

**Know whether your AI agent actually got better.**

Benchrail is a framework for building evals for LLM and agent systems. You change
a prompt, a skill, a tool, a model — and today you have no trustworthy signal for
whether the system improved against intended output. Eyeballing a few runs is
anecdote, and non-determinism means "it looked better" might be luck. Benchrail
gives you a domain-true standard to measure against.

Benchrail gives AI teams two things they need during change: a benchmark that
defines quality and a release rail that prevents regressions from slipping
through.

## Research grounding

Benchrail is opinionated, but not "trust me bro" opinionated. The design follows
several findings from AI evaluation research:

- **Use multiple, context-specific metrics.** Stanford CRFM's
  [HELM](https://arxiv.org/abs/2211.09110) argues for evaluating language models
  across scenarios and metrics beyond accuracy, including calibration,
  robustness, fairness, bias, toxicity, and efficiency. Benchrail applies that
  idea at the product level: each AI system gets its own benchmark and quality
  dimensions.
- **Do not rely on public leaderboard scores as product truth.**
  [The Leaderboard Illusion](https://arxiv.org/abs/2504.20879) documents how
  public model rankings can be distorted by selective disclosure, asymmetric
  data access, and overfitting to leaderboard dynamics. Benchrail is built around
  private, customer-owned evals tied to the actual system under test.
- **LLM judges are useful, but only with calibration.**
  [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)
  shows strong LLM judges can approximate human preferences, while also noting
  biases such as position, verbosity, and self-enhancement. Benchrail therefore
  treats judges as candidates until they clear human-label calibration.
- **Evaluation belongs in the lifecycle, not just the launch checklist.**
  The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
  frames trustworthy AI work through governance, mapping, measurement, and
  management. Benchrail mirrors that lifecycle with benchmark history,
  calibration freshness, regression checks, and failure-driven evolution.
- **Nondeterministic systems need repeated measurement.**
  OpenAI's [HumanEval paper](https://arxiv.org/abs/2107.03374) helped popularize
  sampling-aware evaluation for code generation. Benchrail carries that lesson
  into product evals with multi-sample change evaluation and explicit
  `inconclusive` states.

## The idea

Most eval tools make you hand-author every metric, then leave the eval set to rot
as your product and business evolve. Benchrail instead treats the eval suite as a
**calibrated, evolving grading system derived from your business context**:

```
declare context → derive the right rubric → compile + calibrate judges → run → evolve from discovered failures
```

- **Domain-derived rubrics.** Declare your business context (what the agent does,
  what "good" means, what failure modes matter) and Benchrail reasons out the
  *correct set of grading criteria* for it. The rubric a D2C company needs is not
  the rubric a logistics company needs — and a generic "accuracy" metric misses
  both.
- **Calibrated judges.** Each rubric compiles into LLM judges that are validated
  against a human gold set (agreement metrics, not vibes). A judge that can't beat
  the agreement bar doesn't get to gate anything.
- **Evolves from real failures.** The derived rubric is a *prior* — a hypothesis
  about what matters. Discovered production failures are the *posterior* that
  corrects it. Write evaluators for failures you discover, not ones you imagine.
- **Judge-drift aware.** Calibration isn't one-and-done; Benchrail watches for
  judges drifting out of agreement over time.

## Status

Early, but the wedge is now runnable. The first slice proves domain derivation
end to end: a declared Context Spec → an LLM-reasoned Rubric (with a per-criterion
`why` traced to the spec) → a measured check that two domains genuinely diverge.

The product spine is now a custom benchmark lifecycle: define intended output,
create cases, run a baseline, compare changes, and evolve the benchmark from
real failures.

Use `uv run benchrail help` for the command list, or
`uv run benchrail help evaluate-change` for command-specific options.

## Install

Benchrail is in public-alpha shape. The PyPI distribution name is
`benchrail-ai`; the CLI command and Python import package are `benchrail`.

From a clone:

```bash
git clone https://github.com/JeffBrines/benchrail.git
cd benchrail
uv sync
uv run benchrail help
```

Once published to PyPI:

```bash
uvx --from benchrail-ai benchrail help
```

## Public Alpha Expectations

- APIs and artifact schemas may change before a stable release.
- Benchmark artifacts can contain sensitive prompts, policies, customer inputs,
  retrieved context, and model outputs; do not commit secrets or unredacted PII.
- LLM judges should not gate releases until they pass human-label calibration.
- Public leaderboard scores are not a substitute for product-specific evals.

## Quick Example

Run the checked-in support-agent example:

```bash
uv run benchrail run examples/support-agent/benchmark.yaml \
  --outputs examples/support-agent/baseline.outputs.yaml \
  --system-version baseline \
  --out examples/support-agent/baseline.run.yaml

uv run benchrail run examples/support-agent/benchmark.yaml \
  --outputs examples/support-agent/candidate.outputs.yaml \
  --system-version candidate \
  --out examples/support-agent/candidate.run.yaml

uv run benchrail compare \
  examples/support-agent/baseline.run.yaml \
  examples/support-agent/candidate.run.yaml
```

Expected result: `improved`.

### Create a benchmark

```bash
uv sync
uv run benchrail init "Support Agent" \
  --intended-output "Resolve customer issues accurately using company policy." \
  --good "Uses the current policy source" \
  --bad "Invents refund promises" \
  --user "Customers" \
  --risk "Wrong billing guidance" \
  --calibration-max-age-days 30 \
  --calibration-min-labels 4
```

This creates `benchmarks/support-agent/benchmark.yaml`, including the benchmark's
release-gate calibration policy. Commands such as `inspect`, `next`, and
`evaluate-change` use that policy by default; CLI policy flags override it for a
single run.

Add a real case:

```bash
uv run benchrail add-case benchmarks/support-agent/benchmark.yaml \
  --name "Refund outside window" \
  --input "Can I get a refund after 60 days?" \
  --expected-behavior "Use the refund policy and do not invent exceptions." \
  --tag policy \
  --source production-failure
```

This benchmark artifact is the durable optimization target. Scorers, baseline
runs, comparisons, and failure-driven evolution build on it.

Derive candidate scorers from the benchmark:

```bash
export OPENAI_API_KEY=sk-...
uv run benchrail derive-scorers benchmarks/support-agent/benchmark.yaml \
  --provider openai
```

Generated scorers are saved into the benchmark file as human-reviewable proposals.
They are not release gates until they have been reviewed and, for LLM judges,
calibrated against human labels.

Create a run from saved system outputs:

```yaml
# outputs.prompt-v1.yaml
outputs:
  - case_id: refund-outside-window
    output: "Refunds are only available within 30 days."
```

Or collect outputs by running your system once per case:

```bash
uv run benchrail collect benchmarks/support-agent/benchmark.yaml \
  --command "python agent.py --input {input}" \
  --out outputs.prompt-v1.yaml
```

Command placeholders include `{case_id}`, `{input}`, `{context}`, and
`{expected_behavior}`. Benchrail shell-quotes placeholder values, so do not wrap
placeholders in additional quotes.

Or import outputs from an AI product/integration log:

```jsonl
{"case_id":"refund-outside-window","output":"Refunds are only available within 30 days.","trace_id":"trace-123","provider":"openai","model":"gpt-small","latency_ms":321.5,"total_tokens":162,"cost":0.0042,"feedback":"thumbs_down","outcome":"ticket_reopened"}
```

```bash
uv run benchrail import-outputs benchmarks/support-agent/benchmark.yaml \
  app-outputs.jsonl \
  --out outputs.prompt-v1.yaml
```

`import-outputs` accepts JSONL, CSV, YAML, or JSON. By default it expects
`case_id` and `output` fields, validates case IDs against the benchmark, and
writes the same `outputs.yaml` format used by `benchrail run`. It also preserves
optional production metadata such as `source_id`, `trace_id`, `timestamp`,
`environment`, `provider`, `model`, `latency_ms`, `input_tokens`,
`output_tokens`, `total_tokens`, `cost`, `feedback`, `feedback_score`,
`outcome`, and a scalar `metadata` mapping. Use `--case-id-field` and
`--output-field` when your integration log uses different field names.

Run LLM judge scorers automatically:

```bash
export OPENAI_API_KEY=sk-...
uv run benchrail run benchmarks/support-agent/benchmark.yaml \
  --outputs outputs.prompt-v1.yaml \
  --system-version prompt-v1 \
  --judge-provider openai \
  --out runs/support-agent/prompt-v1.yaml
```

You can also provide manual scorer results in the outputs file. Manual results
override judge execution for that scorer:

```yaml
outputs:
  - case_id: refund-outside-window
    output: "Refunds are only available within 30 days."
    scorer_results:
      - scorer_id: policy-correctness
        status: pass
        rationale: "Matches the refund policy."
```

Unreviewed LLM judges are advisory by default. Their results are stored on run
artifacts, but hard-gate summaries only count LLM judges whose scorer
`trust_status` has been promoted to `trusted` after human review/calibration.

Calibrate judge scorers against human gold labels:

```yaml
# labels.support-agent.yaml
labels:
  - case_id: refund-outside-window
    scorer_id: policy-correctness
    expected_status: pass
    notes: "Human reviewer confirmed the answer follows the refund policy."
```

```bash
uv run benchrail calibrate benchmarks/support-agent/benchmark.yaml \
  --run runs/support-agent/gold-run.yaml \
  --labels labels.support-agent.yaml \
  --out calibrations/support-agent/gold-run.yaml \
  --dry-run
```

Remove `--dry-run` to update each LLM judge scorer's `trust_status` to
`trusted`, `warning`, or `blocked` based on sample size, TPR, TNR, and Cohen's
kappa thresholds. Calibration reports include case-level `failures` showing
false positives, false negatives, and missing judge predictions for human review.
Saved reports are durable artifacts under `calibrations/<benchmark-id>/`.

Inspect calibration history and latest drift:

```bash
uv run benchrail calibration-history benchmarks/support-agent/benchmark.yaml
```

Inspect release-gate readiness with a freshness policy:

```bash
uv run benchrail inspect benchmarks/support-agent/benchmark.yaml \
  --calibration-max-age-days 30 \
  --calibration-min-labels 4
```

An LLM hard gate is only "ready" when it is trusted, has calibration provenance,
has enough human labels, and its calibration is fresh under the policy.

Compare a candidate run against a baseline:

```bash
uv run benchrail compare \
  runs/support-agent/prompt-v1.yaml \
  runs/support-agent/prompt-v2.yaml
```

Or let Benchrail run the full agent-native change loop:

```bash
uv run benchrail evaluate-change benchmarks/support-agent/benchmark.yaml \
  --baseline-command "python agent.py --input {input}" \
  --candidate-command "python agent.py --input {input}" \
  --judge-provider openai \
  --artifact-dir evaluations/support-agent/current-change \
  --baseline-model gpt-premium \
  --candidate-model gpt-small \
  --baseline-cost 1.00 \
  --candidate-cost 0.25 \
  --samples 3
```

This writes baseline/candidate outputs, run artifacts, and an
`evaluation.yaml` decision artifact with `pass`, `fail`, `mixed`, or
`inconclusive`. By default, `evaluate-change` requires LLM hard gates to be
release-ready under the benchmark's calibration policy. Use
`--allow-unready-gates` only for exploratory runs. For nondeterministic agents,
`--samples` repeats the baseline/candidate comparison and returns
`inconclusive` when sample decisions disagree.

If you provide model and cost metadata, the evaluation artifact also records
cost movement and a recommendation such as `safe_to_downgrade`,
`do_not_downgrade`, `needs_more_samples`, or
`quality_improved_but_cost_increased`. Costs are user-supplied estimates for now;
use any consistent unit across baseline and candidate.

Evolve the benchmark from a real failure:

```yaml
# failures/refund-60-days.yaml
input: "Can I get a refund after 60 days?"
bad_output: "Sure, I can refund that."
expected_behavior: "Use the 30-day refund policy and do not invent exceptions."
source: production-failure
tags: [policy, regression]
rationale: "The agent promised a refund outside policy."
name: "Refund outside window"
```

```bash
uv run benchrail --json evolve benchmarks/support-agent/benchmark.yaml \
  --from-failure failures/refund-60-days.yaml \
  --dry-run
```

Remove `--dry-run` to add the case and append an evolution history event.

Import an existing eval definition:

```yaml
# external-eval.yaml
name: Support Refund Eval
intended_output: Answer refund questions using policy.
provider: json
external_id: support/refund-policy
cases:
  - id: refund-60-days
    name: Refund after 60 days
    input: Can I get a refund after 60 days?
    expected_behavior: Say refunds are only available within 30 days.
    source: imported
```

```bash
uv run benchrail import-json external-eval.yaml \
  --out benchmarks/support-refund/benchmark.yaml
```

Discover available and planned import/attach routes:

```bash
uv run benchrail --json discover "refund policy support"
```

Attach a run executed outside Benchrail:

```bash
uv run benchrail attach-run benchmarks/support-refund/benchmark.yaml \
  --provider inspect \
  --external-id run-123 \
  --system-version prompt-v1 \
  --summary external-run-summary.yaml \
  --out runs/support-refund/inspect-run-123.yaml
```

### Run the milestone demo

The demo derives a rubric for the *same* agent task ("monthly revenue forecast")
from two different Context Specs — D2C ecommerce vs freight logistics — and asserts
the rubrics diverge.

```bash
uv sync
export ANTHROPIC_API_KEY=sk-ant-...
uv run python scripts/derive_demo.py --provider anthropic
```

Or run the same derivation through OpenAI/GPT:

```bash
export OPENAI_API_KEY=sk-...
uv run python scripts/derive_demo.py --provider openai
```

It writes `out/rubric.d2c.yaml` and `out/rubric.logistics.yaml`, prints a
divergence report, and exits non-zero if the rubrics *don't* diverge (which would
mean the derivation isn't domain-sensitive). Offline tests: `uv run pytest`.

Provider defaults can also be set with environment variables:

```bash
export BENCHRAIL_LLM_PROVIDER=openai
export BENCHRAIL_LLM_MODEL=gpt-5.5
uv run python scripts/derive_demo.py
```

### What's wired

| Piece | Module |
| --- | --- |
| Context Spec schema + loader | `src/benchrail/context_spec.py` |
| Benchmark schema + loader | `src/benchrail/benchmark.py` |
| Benchmark CLI | `src/benchrail/cli.py` |
| Agent inspection + next action | `src/benchrail/agent.py` |
| Failure-driven evolution | `src/benchrail/evolve.py` |
| Agent-native evaluation loop | `src/benchrail/evaluation.py` |
| External benchmark import | `src/benchrail/importer.py` |
| Source discovery | `src/benchrail/discovery.py` |
| Scorer derivation | `src/benchrail/scorer.py` |
| LLM judge execution | `src/benchrail/judge.py` |
| Judge calibration | `src/benchrail/calibration.py` |
| Run + comparison artifacts | `src/benchrail/run.py` |
| Rubric / Criterion schema | `src/benchrail/rubric.py` |
| Failure-mode-enumeration prompt | `src/benchrail/prompt.py` |
| Provider-neutral LLM derivation | `src/benchrail/derive.py`, `src/benchrail/llm.py` |
| Divergence measurement | `src/benchrail/divergence.py` |

Inspect runner integration is the next major slice — see below.

### Design docs

- [`DESIGN.md`](DESIGN.md) — thesis, positioning, build-on decision, trust model.
- [`BRANDING.md`](BRANDING.md) — Benchrail naming decision and rename migration plan.
- [`CUTOVER.md`](CUTOVER.md) — external repository, package, domain, and compatibility checklist.
- [`CHANGELOG.md`](CHANGELOG.md) — release history and public-alpha contents.
- [`RELEASE.md`](RELEASE.md) — release validation, publishing, and announcement checklist.
- [`ROADMAP.md`](ROADMAP.md) — execution plan from provider-neutral derivation to calibrated runs.
- [`docs/agent-harness.md`](docs/agent-harness.md) — how build agents should operate Benchrail.
- [`docs/derivation-v0.md`](docs/derivation-v0.md) — the Context Spec schema, the
  derivation method, and a worked D2C-vs-logistics example showing how the rubrics
  diverge.

Benchrail is intended to run on top of [Inspect AI](https://inspect.aisi.org.uk/)
rather than reimplementing an eval runner.

Security note: benchmark artifacts can contain production inputs, policies,
customer data, and model outputs. Treat them as sensitive repo artifacts, review
imported eval definitions before running judges over them, and avoid committing
PII or secrets.

## License

[MIT](LICENSE).
