Metadata-Version: 2.5
Name: oloproof
Version: 0.1.0a16
Summary: Know if your AI system is good enough: evaluation, intervals and decisions for AI systems.
Project-URL: Homepage, https://oloproof.com
Project-URL: Documentation, https://oloproof.com/docs
Project-URL: Docs for coding agents, https://oloproof.com/llms.txt
Project-URL: Changelog, https://oloproof.com/changelog
Project-URL: Second opinion, https://oloproof.com/second-opinion
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Requires-Python: >=3.11
Requires-Dist: cryptography>=43
Requires-Dist: httpx<1,>=0.27
Requires-Dist: jsonschema>=4.22
Requires-Dist: numpy>=2.0
Requires-Dist: pydantic>=2.12
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.14
Requires-Dist: typer>=0.15.4
Provides-Extra: collect
Requires-Dist: opentelemetry-proto>=1.34; extra == 'collect'
Provides-Extra: dev
Requires-Dist: cryptography>=43; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.34; extra == 'dev'
Requires-Dist: opentelemetry-proto>=1.34; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.34; extra == 'dev'
Requires-Dist: pillow>=10.1; extra == 'dev'
Requires-Dist: psycopg[binary]>=3.2; extra == 'dev'
Requires-Dist: pytest-xdist>=3.6; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: scikit-learn>=1.6.1; extra == 'dev'
Provides-Extra: media
Requires-Dist: pillow>=10.1; extra == 'media'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: worker
Requires-Dist: cryptography>=43; extra == 'worker'
Description-Content-Type: text/markdown

# Oloproof
Evidence for every AI decision.

Your eval score went up. Is it real? And can you trust the judge that measured it?
Oloproof tells you if it's real, and if the judge can be trusted.
Works on your own runs or on scores from the tool you already use. It puts an interval behind
every decision and decides: `PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` or `MANUAL_REVIEW`.

Start with the scores you already have, one file before the change and one after:

```bash
pip install oloproof
oloproof scores import before.csv after.csv
```

Or use the hosted [second opinion](https://oloproof.com/second-opinion), with nothing to install.
The [second opinion guide](https://oloproof.com/docs/second-opinion) shows the file each tool
exports. Coding agents can read the docs as text at
[oloproof.com/llms.txt](https://oloproof.com/llms.txt).

It works in three layers. What happened: every case, score, latency and cost, traced to the run,
the judge and the data that produced it. How sure you can be: an interval with the right
denominator wherever one is admitted, and judges checked against people before they are allowed to count.
What happens next: pass, fail, or not enough evidence yet, with how many more cases would settle
it and which part broke.

There are two ways in. Evaluate a system: hold one system to what you require of it, with no
baseline needed. Compare a change: compare a change with what is live, on the same cases, and
know whether the difference is real. Both run locally with no account, from the terminal or in
CI, and a hosted workspace shows the same records in the browser once you push them. Once the
system is live, people review a sample a hosted workspace draws from real traffic, in the browser,
while the content stays on your side. The engine, CLI and SDK are free, Apache-2.0 licensed.

## Quick Start

Your first evaluation, with no baseline: one version of one application, held to a bar you
write down. Five steps take you from nothing to a release decision on your own machine, with no
account and no model calls (installing needs the network once), then a sixth points it at your own application. Every
output below is from running these exact commands in an empty directory. The terms (case,
evaluator, metric, interval, gate) are defined in the [core concepts guide](https://oloproof.com/docs/concepts).

### 1. Install

```bash
python3 -m venv .venv
. .venv/bin/activate
pip install oloproof
```

Python 3.11 or later. The command-line tool, the engine and the SDK all run locally. The
optional extra `oloproof[collect]` receives OpenTelemetry traces with `oloproof collect`.

### 2. Scaffold a project

```bash
oloproof init oloproof-demo
cd oloproof-demo
```

`init` writes a small support bot to evaluate (`app.py`, declared with `@system`), 30 cases in
`data/example.jsonl`, the suite in `oloproof.yaml` (the dataset, the system and an evaluator that
checks each answer's label) and the release policy in `release.yaml` (the confidence level and a
rule that the label is right at least 80% of the time).

It then prints what that suite and bar can decide, before anything has run:

```text
What this suite can decide, if every case is measured:
  label-floor (exact_label at least 0.8, 30 cases):
    29 or 30 pass: PASS (established); 19 to 28: INSUFFICIENT_EVIDENCE (not
    established yet); 18 or fewer: FAIL (ruled out). With 50 cases, PASS from
    46; with 100, from 89.
```

Each count is the gate's own interval at that count, so 28 passing cases out of 30 cannot pass
an 80% bar however good the system is. The same lines sit as a comment above the rule in
`release.yaml`, and `oloproof plan` prints them again for the suite as it is now.

### 3. Run the suite

```bash
oloproof run
```

The output (run ids differ on your machine):

```text
Run run_01M4FC9Q6KT0C7GD33DV6K9628 [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)
Decisions
  Rule         Metric       State  Reasons
  label-floor  exact_label  PASS   lower_bound_meets_minimum
Metrics
  Metric       Estimate  Interval         N
  exact_label  100.0%    [88.4%, 100.0%]  30 / 30 observed, 0 missing, 0 excluded
Reach: inputs like these: the population this suite stands for, as far as it is a fair sample.
Cache: execution 0 hit/30 miss; judgment 0 hit/30 miss

Share it: `oloproof login --url https://oloproof.com`, then `oloproof push run_01M4FC9Q6KT0C7GD33DV6K9628`. Nothing leaves this machine until you push.
```

How to read it:

- `Decisions` has one row per rule in `release.yaml`. `label-floor` asks that the `exact_label`
  pass rate be at least 80%. Its state is `PASS` because the interval's lower bound, 88.4%, is
  above 80%; the reason code says exactly that.
- `Metrics` shows the estimate (30 of 30 right, 100%) and its 95% interval. The interval is what
  decides, not the estimate: with 30 cases, a system that is truly right 89% of the time could
  still score 30 of 30.
- `N` is the denominator: how many cases were observed, missing or excluded. In `30 / 30
  observed`, the first number is the cases that passed (the estimate's numerator) and the second
  is the cases observed, so `15 / 20 observed` would mean 15 passed out of 20. A case that could
  not be measured is counted, never silently dropped.
- In a terminal narrower than the table, its columns are cut short (`INSUFFICIENT_EVIDE…`).
  Widen the window, or prefix the command with `COLUMNS=200`, to see every column whole.
- `Gate: ALLOW (exit 0)` is the release action: nothing the policy blocks on occurred.
- The last line is printed only until this terminal is logged in, and never in CI. The run
  stays on your machine; sharing it is optional, and the end of this page shows how.

Run it again and the system is not called: `Cache: execution 30 hit/0 miss`, because neither the
system nor any case's input changed.

### 4. Inspect a case

```bash
oloproof inspect latest --failures
oloproof inspect latest --case refund_00
```

`latest` is the newest run and `latest~1` the one before it. Any command that takes a run id also
accepts a unique start or end of one, and `oloproof runs` lists your runs with their answers.

Here every case passed, so the first prints `0 of 30 cases failed, errored or did not finish`.
The second shows one case whole: its input, its expected label, the output the system returned
(`reused from cache` on a second run) and each evaluator's verdict (`exact_label: passed`). On
your own application, `--failures` is where you start.

### 5. Gate on the decision

`oloproof run` exits on its gate, so a CI step that runs it needs no further wiring. To see it
block, raise the rule's `min` in `release.yaml` from `0.80` to `0.95` and run again:

```bash
oloproof run
echo "exit $?"
```

```text
Gate: BLOCK (exit 3)
  label-floor  exact_label  INSUFFICIENT_EVIDENCE  interval_overlaps_threshold
exit 3
```

The estimate is still 100%, but the interval, [88.4%, 100.0%], straddles 95%: 30 cases cannot
show a rate that high with 95% confidence. That is `INSUFFICIENT_EVIDENCE`, not a failure, and the
policy's `block_on` blocks on it. To ask how many cases would settle it:

```bash
oloproof plan latest
```

```text
label-floor: a suite of about 72 cases (42 more) would decide it, if the next cases go like these; at that size it is settled about half the time
```

`oloproof gate latest --policy release.yaml` re-decides a stored run against a changed policy
without running anything again. The [gating guide](https://oloproof.com/docs/gating) lists every exit code.

### 6. Your own application

The example shows the tool works; your application is what you came to evaluate. No baseline is
needed: a first evaluation asks whether your application meets your requirements, and comparing
versions comes later.

Point `system` in `oloproof.yaml` at your application: a Python callable that takes a case's
`input` and returns a dictionary (`callable: app:answer`, as the example does), or an HTTP endpoint
that does the same over JSON (`http: {url: http://localhost:8000/answer}`). Replace
`data/example.jsonl` with your own cases, one JSON object per line with an `id`, the `input` your
application takes and the `expected` answer or label. Fifty real cases are about the floor for an
interval narrow enough to decide anything, and they are a better start than a hundred invented
ones. Keep an evaluator that can check your
`expected` field (`exact_match` on a label, `contains`, `regex`, `json_schema`, or an LLM judge
with a rubric), and write what acceptable means as a rule in `release.yaml`: the metric and the
rate it must clear, at the confidence level you set.

A rule's `metric` is an evaluator's `criterion`, so the two are written together. The scaffold's
evaluator is named `exact_label`: it compares the `label` your application returns with the
`label` under each case's `expected`. A rule on it names the same criterion:

```yaml
# oloproof.yaml
evaluators:
  - type: exact_match
    criterion: exact_label
    field: label
```

```yaml
# release.yaml
rules:
  - id: label-correct
    metric: exact_label
    min: 0.90
```

Rename the criterion if you like, in both places at once; a rule naming a metric no evaluator
produces stops the run with a configuration error that lists the metrics there are.

Then `oloproof run`. `PASS` means the whole interval cleared your bar on your cases; `FAIL` means
it cannot; `INSUFFICIENT_EVIDENCE` means the run could not settle it, and the decision's reason
says why. Where the reason is too few cases, `oloproof plan latest` estimates how many more
would settle it, for the methods that support sizing; where it is something else, such as
executions that failed or a judge not yet validated, the reason names what to fix instead. Each
is an answer about your application on the cases you gave it, and nothing more.

### Where to go next

- Choose the guide for your kind of system (classifier, text generation, HTTP service, RAG,
  agent, predictive model, conversation) from the [start-here page](https://oloproof.com/docs/start-here).
- Change the system and compare the new version with this run: the [compare guide](https://oloproof.com/docs/compare).
- Add human review of a judge, CI and a hosted workspace: the [judges](https://oloproof.com/docs/judges), [CI](https://oloproof.com/docs/ci)
  and [gating](https://oloproof.com/docs/gating) guides.

### Evaluators

The `type:` values `oloproof.yaml` accepts:

| | |
|---|---|
| deterministic | `exact_match`, `contains`, `regex`, `json_schema`; `canary_absent`: fails an output that carries a generated attack's canary |
| LLM judge | `rubric_judge`, `groundedness_judge`, `citation_support_judge`; `probability_judge`: a yes/no, choice or score question answered by one token's probabilities; `cascade`: a probability judge first, a second judge only where it is unsure |
| model | `model_classifier`: a trained classifier or NLI model on a TEI server, scored against a threshold |
| retrieval | `hit_rate`, `recall`, `mrr`, `ndcg`, `citation_validity` |
| agent | `agent_max_steps`, `agent_tool_called`, `agent_no_tool_loop`, `agent_tool_sequence`, `agent_no_undeclared_tool`, `agent_constraints_satisfied` |
| multi-agent | `agent_route`, `agent_tool_permissions`, `agent_max_handoffs` |
| predictive | `predictive_correct`, `predictive_precision`, `predictive_recall`, `predictive_ranking`, `predictive_brier`, `predictive_log_loss`, `predictive_absolute_error` |
| reinforcement learning | `episode_return`, `episode_success`, `episode_cost_within`, `off_policy_value`, `off_policy_improvement`, `sequential_off_policy_value`, `sequential_off_policy_improvement` |
| media | `media_check`: format, dimensions, duration and size measured from the recorded bytes; `image_matches_reference`: a perceptual-hash match with a reference image |
| conversation | `conversation_completed`, `conversation_judge`; `every_turn`: a condition every answer must meet; `goal_reached`, `turns_to_goal`: whether a simulated user's goal was met, and within how many of its messages |

A judge names its `provider` (`anthropic`, `openai` or `openai_compatible`), its `model`, and its
rubric as `rubric_text` or `rubric_file`. One on a provider that speaks the OpenAI API but is not
OpenAI (Groq, Together, a local server) also needs `base_url` and `api_key_env`:

```yaml
evaluators:
  - type: rubric_judge
    criterion: answer_correct
    provider: openai_compatible
    model: openai/gpt-oss-20b
    base_url: https://api.groq.com/openai/v1
    api_key_env: GROQ_API_KEY
    rubric_text: "PASS if the answer conveys the same fact as the reference."
```

Naming a field an evaluator does not take is answered with the ones it does.

See a run in the browser by pushing it to a hosted workspace. Sign-up at
[oloproof.com](https://oloproof.com) is open; `login` opens your browser on a short code to approve there, and
nothing leaves your machine until you push:

```bash
oloproof login --url https://oloproof.com
oloproof push latest
```

`push` ends with `Open it:` and the run's address in the workbench, to open or send to a
teammate, and remembers it, so a later `--summary markdown` links the pull request to the same
page ([CI guide](https://oloproof.com/docs/ci)).


## Engine Rules

The evidence chain is:

```text
scenarios -> execution -> artifacts -> evaluators -> judgments
           -> statistical inference -> diagnostics/interventions
           -> decisions -> evidence/provenance
           -> LLM explanation/hypothesis/next experiment
```

Decision states are exactly `PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` and
`MANUAL_REVIEW`. Execution states such as `RUN_ERROR` and `CANCELLED` are not
decision states. Release actions are separate: `ALLOW`, `WARN` or `BLOCK`.

For minimum-threshold interval rules, `PASS` requires the lower bound to meet the
threshold and `FAIL` requires the upper bound to be below it. Otherwise the result
is `INSUFFICIENT_EVIDENCE`. Unsupported declared structures fail closed.

## Current Scope

Built:

- Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and
  every record is content-addressed.
- Deterministic, custom evaluators in the Python SDK (`@evaluator`), LLM-judge, retrieval, agent-trajectory and predictive
  evaluators, and an evaluator registry that measures a judge against human labels on the answers
  people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a
  judge may gate. People label from a file (`oloproof labels export` and `import`), the terminal
  (`oloproof review`) or, in a hosted workspace, a review queue in the browser; `oloproof evaluators
  try` measures a draft judge before it is adopted. Probability judges can be calibrated against
  labels, and a cascade sends only uncertain cases to a stronger judge.
- Judges measured without labels: `oloproof evaluators probe` reports how often a judge's verdict
  moves when it is asked again or shown padding, and `oloproof prefer` asks a pairwise judge which
  of two runs answered better, in both orders, with the judge's position bias beside the result.
- Systems of several agents: each trajectory step names the agent that took it, a hand-off is a
  step, and deterministic criteria judge the route against a declared one, tool calls against a
  closed permission map, and the number of hand-offs against a bound. No failure is attributed to
  an agent.
- Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and
  cluster-aware analysis where declared.
- Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
- Declared slices with multiplicity control; a slice rule gates only inside a declared family with
  a correction and a minimum support.
- Intervention-driven diagnosis of RAG failures (`oloproof diagnose`) and sample-size planning
  for undecided rules (`oloproof plan`).
- Streaming progress with live provisional estimates, bounded concurrency for systems and
  judges (`concurrency:`), and cache reuse across execution, judgment, analysis and gate
  recomputation.
- CI gating through exit codes, sign-off, usage and cost reporting.
- Early stopping (`early_stopping: true` in a policy): a run, or a comparison, stops after a
  seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst
  value, so a stop is valid whenever it happens, and the run reports the work it avoided.
- Prediction-powered inference (`ppi_judge_rate@1`): a judge's pass rate corrected by a random,
  blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted
  workspace draws the sample itself and verifies the interval with its own engine.
- Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute,
  an Owner pins the evaluation and the release policy, and the workspace gate decides each system
  version on its first run of that evaluation, counting independent labellers by account.
- The workbench, as a hosted workspace that `oloproof push` sends runs to. Sign-up for a
  hosted workspace is open.
- Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores
  or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the
  reviewer's browser from `oloproof collect`, never through Oloproof. Scores and preferences are
  stored and enter no statistic yet.
- Production traces received locally: `oloproof collect` accepts OpenTelemetry (GenAI and
  OpenInference conventions) and pushes them through the project's egress policy. A hosted
  deployment can also take spans directly, opt-in, redacted at ingest by a dedicated key's
  policy.
- Production evaluation: workspace-drawn traffic samples over collector commitments, online
  scoring and anytime-valid monitoring, and turning a trace into a test case.

Not built yet: the rest of the statistical roadmap: conformal risk control, hierarchical
dependence, causal intervention analysis, IRT and a method for score and preference labels.

## License

Oloproof is licensed in two parts. The engine, CLI, SDK and runner, which are this package, and
everything else a developer installs are under the Apache License 2.0 (`LICENSE`). The hosted
control plane and workbench are not part of this package and are under the Elastic License 2.0.
`NOTICE` says which covers what.
