Metadata-Version: 2.5
Name: oloproof
Version: 0.1.0a4
Summary: Developer-first AI systems evaluation, diagnostics, and decision evidence.
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Requires-Python: >=3.11
Requires-Dist: cryptography>=43
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.22
Requires-Dist: numpy>=2.0
Requires-Dist: pydantic>=2.12
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.14
Requires-Dist: typer>=0.15.4
Provides-Extra: collect
Requires-Dist: opentelemetry-proto>=1.34; extra == 'collect'
Provides-Extra: dev
Requires-Dist: cryptography>=43; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.34; extra == 'dev'
Requires-Dist: opentelemetry-proto>=1.34; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.34; extra == 'dev'
Requires-Dist: psycopg[binary]>=3.2; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: scikit-learn>=1.6.1; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: worker
Requires-Dist: cryptography>=43; extra == 'worker'
Description-Content-Type: text/markdown

# Oloproof

Oloproof is a developer-first AI systems evaluation, diagnostics and decision
platform. It runs an AI system on a versioned suite of cases, measures it with
deterministic, custom, judge, retrieval, agent and predictive evaluators, puts an
interval with the right denominator on every number, compares a candidate with a
baseline, diagnoses failures by intervening on them, and decides a release as
`PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` or `MANUAL_REVIEW` from declared rules. The
same canonical objects are read in the terminal and in the Next.js workbench.

Oloproof is not a toy eval harness. The intended product direction is credible
beside DeepEval/Confident AI, Braintrust, LangSmith, Arize/Phoenix, Galileo,
Langfuse and Opik, while differentiating through statistical rigor, evaluator
reliability, intervention-driven diagnosis and decision-grade evidence.

## Quick Start

Four steps from nothing to a release decision on your own machine, with no account and no
network, and a fifth that points it at your own application.

### 1. Install

```bash
python3 -m venv .venv
. .venv/bin/activate
pip install oloproof
```

Python 3.11 or later. The command-line tool, the engine and the SDK all run locally. Optional
extras: `oloproof[collect]` to receive OpenTelemetry traces with `oloproof collect`, and
`oloproof[postgres,worker]` for running the hosted store and worker yourself.

### 2. Scaffold a project

```bash
oloproof init oloproof-demo
cd oloproof-demo
```

`init` writes a small support bot to evaluate (`app.py`, declared with `@system`), 30 cases in
`data/example.jsonl`, the suite in `oloproof.yaml` (the dataset, the system and an evaluator that
checks each answer's label) and the release policy in `release.yaml` (the confidence level and a
rule that the label is right at least 80% of the time).

### 3. Run the suite

```bash
oloproof run
```

The run prints each rule's decision, `PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` or `MANUAL_REVIEW`,
and each metric with its interval and how many cases stand behind it. A rule passes only when the
whole interval clears its threshold, not the estimate. Run it again and the system is not called:
its outputs for an unchanged dataset come from the cache.

### 4. Gate on the decision

```bash
oloproof run
echo "exit $?"
```

`oloproof run` exits on its gate: `0` when the policy lets the release go, and a non-zero code
naming why when it blocks, so a CI step that runs it needs no further wiring. To see it block,
raise the rule's `min` in `release.yaml` to `0.95` and run again: 30 cases cannot show a rate that
high with 95% confidence, so the rule is `INSUFFICIENT_EVIDENCE` and the run exits `3`. The gating
guide lists every code.

### 5. Your own application

The example shows the tool works; your application is what you came to evaluate. No baseline is
needed: a first evaluation asks whether your application meets your requirements, and comparing
versions comes later.

Point `system` in `oloproof.yaml` at your application: a Python callable that takes a case's
`input` and returns a dictionary (`callable: app:answer`, as the example does), or an HTTP endpoint
that does the same over JSON (`http: {url: http://localhost:8000/answer}`). Replace
`data/example.jsonl` with a handful of your own cases, one JSON object per line with an `id`, the
`input` your application takes and the `expected` answer or label; ten real cases are a better
start than a hundred invented ones, and more can come later. Keep an evaluator that can check your
`expected` field (`exact_match` on a label, `contains`, `regex`, `json_schema`, or an LLM judge
with a rubric), and write what acceptable means as a rule in `release.yaml`: the metric and the
rate it must clear, at the confidence level you set.

A rule's `metric` is an evaluator's `criterion`, so the two are written together. The scaffold's
evaluator is named `exact_label`: it compares the `label` your application returns with the
`label` under each case's `expected`. A rule on it names the same criterion:

```yaml
# oloproof.yaml
evaluators:
  - type: exact_match
    criterion: exact_label
    field: label
```

```yaml
# release.yaml
rules:
  - id: label-correct
    metric: exact_label
    min: 0.90
```

Rename the criterion if you like, in both places at once; a rule naming a metric no evaluator
produces stops the run with a configuration error that lists the metrics there are.

Then `oloproof run`. `PASS` means the whole interval cleared your bar on your cases; `FAIL` means
it cannot; `INSUFFICIENT_EVIDENCE` means the run could not settle it, and the decision's reason
says why. Where the reason is too few cases, `oloproof plan RUN_ID --run` estimates how many more
would settle it, for the methods that support sizing; where it is something else, such as
executions that failed or a judge not yet validated, the reason names what to fix instead. Each
is an answer about your application on the cases you gave it, and nothing more.

### Evaluators

The `type:` values `oloproof.yaml` accepts:

| | |
|---|---|
| deterministic | `exact_match`, `contains`, `regex`, `json_schema` |
| LLM judge | `rubric_judge`, `groundedness_judge`, `citation_support_judge`; `probability_judge`: a yes/no, choice or score question answered by one token's probabilities; `cascade`: a probability judge first, a second judge only where it is unsure |
| model | `model_classifier`: a trained classifier or NLI model on a TEI server, scored against a threshold |
| retrieval | `hit_rate`, `recall`, `mrr`, `ndcg`, `citation_validity` |
| agent | `agent_max_steps`, `agent_tool_called`, `agent_no_tool_loop`, `agent_tool_sequence`, `agent_no_undeclared_tool`, `agent_constraints_satisfied` |
| multi-agent | `agent_route`, `agent_tool_permissions`, `agent_max_handoffs` |
| predictive | `predictive_correct`, `predictive_precision`, `predictive_recall`, `predictive_ranking`, `predictive_brier`, `predictive_log_loss`, `predictive_absolute_error` |

A judge names its `provider` (`anthropic`, `openai` or `openai_compatible`), its `model`, and its
rubric as `rubric_text` or `rubric_file`. One on a provider that speaks the OpenAI API but is not
OpenAI (Groq, Together, a local server) also needs `base_url` and `api_key_env`:

```yaml
evaluators:
  - type: rubric_judge
    criterion: answer_correct
    provider: openai_compatible
    model: openai/gpt-oss-20b
    base_url: https://api.groq.com/openai/v1
    api_key_env: GROQ_API_KEY
    rubric_text: "PASS if the answer conveys the same fact as the reference."
```

Naming a field an evaluator does not take is answered with the ones it does.

Re-decide an existing run without rerunning the system or evaluators:

```bash
oloproof gate RUN_ID --policy release.yaml
```

Inspect failures and one case:

```bash
oloproof inspect RUN_ID --failures
oloproof inspect RUN_ID --case refund_00
```

Export a bundle and open it in the web workbench:

```bash
oloproof export RUN_ID
OLOPROOF_BUNDLE_DIR="$PWD/.oloproof/bundles" pnpm --dir /path/to/Oloproof --filter @oloproof/web dev
```

The bundle path must be absolute, because pnpm starts the app from `apps/web`. The workbench opens
on its workspace index; a lone `OLOPROOF_BUNDLE_DIR` appears there as the `local/bundles` project.
To hold several projects on one machine, point `OLOPROOF_WORKBENCH_DIR` at a directory containing a
`workbench.json` instead (`docs/API_CONTRACTS.md`).


## Install For Development

From the repository root:

```bash
python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
pnpm install --frozen-lockfile
make check
pnpm --filter @oloproof/web check
```

The public Python surface is:

```python
from oloproof import evaluate, system, current_case
from oloproof.evaluators import ExactMatch, Contains, JsonSchema, Regex, RubricJudge, evaluator
```

Everything under `oloproof_core` is engine internals.

## Examples

Five runnable projects, none of which needs provider credentials. Each README explains what the
run shows and why it decides what it decides.

| Example | Shows |
|---|---|
| `examples/support_bot/` | the first run: a label accuracy that is too thin to decide, inspecting failures, re-deciding with `gate`, adding an LLM judge |
| `examples/support_rag/` | RAG evaluation: retrieval and generation evaluators, declared slices, `expected.gold_context`, and `oloproof diagnose` intervening on failed cases |
| `examples/support_agent/` | agent and tool evaluation: an agent that runs its own loop and records a trajectory with `current_case().agent_trajectory(...)`, a tool reference in each case's `expected.tools` |
| `examples/triage_agents/` | several agents in one trajectory: each step names its agent, hand-offs, a declared route in each case's `expected.route`, a closed tool-permission map and a hand-off bound |
| `examples/churn_model/` | predictive classification: the `predictive:` block, three denominators, calibration and ranking |

```bash
cd examples/support_bot
../../.venv/bin/python -m oloproof_cli run
```

The run exits 3. Its keyword-rule bot labels 13 of 18 questions correctly,
which is too few cases to show label accuracy of at least 0.70, so the release
is blocked with `INSUFFICIENT_EVIDENCE`.

## Benchmark

The Slice A analysis benchmark stores runs of synthetic precomputed evidence and
times what `oloproof gate` does with them: loading, denominators, cold and
cached analysis, the clustered bootstrap and decisions. It excludes system and
model latency.

```bash
.venv/bin/python benchmarks/analysis_bench.py --sizes 1000,10000,100000
```

Recorded results are in `benchmarks/README.md`.

## Engine Rules

The evidence chain is:

```text
scenarios -> execution -> artifacts -> evaluators -> judgments
           -> statistical inference -> diagnostics/interventions
           -> decisions -> evidence/provenance
           -> LLM explanation/hypothesis/next experiment
```

Decision states are exactly `PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` and
`MANUAL_REVIEW`. Execution states such as `RUN_ERROR` and `CANCELLED` are not
decision states. Release actions are separate: `ALLOW`, `WARN` or `BLOCK`.

For minimum-threshold interval rules, `PASS` requires the lower bound to meet the
threshold and `FAIL` requires the upper bound to be below it. Otherwise the result
is `INSUFFICIENT_EVIDENCE`. Unsupported declared structures fail closed.

## Current Scope

Built:

- Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and
  every record is content-addressed.
- Deterministic, custom (`@evaluator`), LLM-judge, retrieval, agent-trajectory and predictive
  evaluators, and an evaluator registry that measures a judge against human labels on the answers
  people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a
  judge may gate. People label from a file (`oloproof labels export` and `import`), the terminal
  (`oloproof review`) or, in a hosted workspace, a review queue in the browser; `oloproof evaluators
  try` measures a draft judge before it is adopted. Probability judges can be calibrated against
  labels, and a cascade sends only uncertain cases to a stronger judge.
- Judges measured without labels: `oloproof evaluators probe` reports how often a judge's verdict
  moves when it is asked again or shown padding, and `oloproof prefer` asks a pairwise judge which
  of two runs answered better, in both orders, with the judge's position bias beside the result.
- Systems of several agents: each trajectory step names the agent that took it, a hand-off is a
  step, and deterministic criteria judge the route against a declared one, tool calls against a
  closed permission map, and the number of hand-offs against a bound. No failure is attributed to
  an agent.
- Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and
  cluster-aware analysis where declared.
- Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
- Declared slices with multiplicity control; a slice rule gates only inside a declared family with
  a correction and a minimum support.
- Intervention-driven diagnosis of RAG failures (`oloproof diagnose`) and sample-size planning
  for undecided rules (`oloproof plan`).
- Streaming progress with live provisional estimates, bounded concurrency for systems and
  judges (`concurrency:`), and cache reuse across execution, judgment, analysis and gate
  recomputation.
- CI gating through exit codes, sign-off, usage and cost reporting.
- Early stopping (`early_stopping: true` in a policy): a run, or a comparison, stops after a
  seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst
  value, so a stop is valid whenever it happens, and the run reports the work it avoided.
- Prediction-powered inference (`ppi_judge_rate@1`): a judge's pass rate corrected by a random,
  blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted
  workspace draws the sample itself and verifies the interval with its own engine.
- Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute,
  an Owner pins the evaluation and the release policy, and the workspace gate decides each system
  version on its first run of that evaluation, counting independent labellers by account.
- The Next.js workbench, locally over exported bundles or as a hosted workspace that
  `oloproof push` sends runs to. The hosted workspace is invitation-only.
- Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores
  or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the
  reviewer's browser from `oloproof collect`, never through Oloproof. Scores and preferences are
  stored and enter no statistic yet.
- Production traces received locally: `oloproof collect` accepts OpenTelemetry (GenAI and
  OpenInference conventions) and pushes them through the project's egress policy. A hosted
  deployment can also take spans directly, opt-in, redacted at ingest by a dedicated key's policy
  (`docs/plans/SLICE_S.md` S06).
- Production evaluation: workspace-drawn traffic samples over collector commitments, online
  scoring and anytime-valid monitoring, and turning a trace into a test case
  (`docs/plans/SLICE_S.md` §7).

Not built yet: the rest of the statistical roadmap in Engine §24: conformal risk control,
hierarchical dependence, causal intervention analysis, IRT and a method for score and preference
labels. The launch checklist (`docs/roadmap/INITIAL_COMPETITIVE_RELEASE.md`) records what is
left before the first invited user: an independent review of the whole product, and watching
that user run the golden path.

## Project Instructions

Coding agents should start with:

- `AGENTS.md` - shared instructions and source-of-truth map.
- `AGENT_WORKFLOWS.md` - Claude/Codex division of labor and task prompts.
- `DOCUMENTATION_STATUS.md` - canonical synchronized doc versions and status.
- `docs/engine/ENGINE_SPECIFICATION.md` - statistical and decision contract.
- `docs/plans/` - the implementation plans, one per slice, and their accepted decisions.

The top-level local repository folder is `Oloproof/` with a capital O. Python
packages, npm package names, CLI commands and environment variables remain
lowercase or uppercase as documented: `oloproof`, `oloproof_core`,
`@oloproof/...`, `OLOPROOF_*`.

## License

Oloproof is licensed in two parts. The engine, CLI, SDK and runner (`python/`), the UI packages
(`packages/`) and everything else a developer installs are under the
[Apache License 2.0](LICENSE). The hosted control plane and workbench (`apps/web/`) are under the
[Elastic License 2.0](apps/web/LICENSE): you may use, modify and self-host it, and you may not offer
it to others as a managed service. [NOTICE](NOTICE) says which covers what.
