Metadata-Version: 2.4
Name: orchestrator-eval-harness
Version: 0.2.0
Summary: Contract-driven evaluation of software-development orchestrator models
Author: Dark Light
Keywords: agents,evaluation,orchestration,llm,openai-compatible
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Orchestrator Evaluation Harness

Installable, contract-driven comparison of two software-development
orchestrator models. It supports OpenRouter and other OpenAI-compatible Chat
Completions endpoints, including different endpoints for the two candidates
and the coding worker.

The package contains the v2 contract gateway, deterministic Oracle arm, live
worker arm with replay cassettes, six validated task/scenario pairs, budget
ledger, paired statistics, Markdown/JSON reporting, and neutral evidence-bundle
export. It contains no historical results.

## Install

Build locally:

```bash
python3 -m build
python3 -m pip install dist/orchestrator_eval_harness-0.2.0-py3-none-any.whl
```

Or install the source checkout:

```bash
python3 -m pip install .
```

Create an editable configuration:

```bash
orchestrator-eval --output experiment.json init-config
```

## Configure endpoints and models

Credentials are never stored in JSON. `api_key_env` names the environment
variable that contains the bearer token. Set it to `null` for an endpoint that
does not require authentication.

```json
{
  "experiment_id": "candidate-a-vs-candidate-b",
  "endpoints": {
    "openrouter": {
      "url": "https://openrouter.ai/api/v1/chat/completions",
      "api_key_env": "OPENROUTER_API_KEY",
      "provider_routing": true,
      "include_usage": true,
      "headers": {
        "HTTP-Referer": "https://localhost/orchestrator-eval",
        "X-Title": "Orchestrator Evaluation Harness"
      }
    },
    "local": {
      "url": "http://127.0.0.1:1234/v1/chat/completions",
      "api_key_env": null,
      "provider_routing": false,
      "include_usage": false,
      "headers": {}
    }
  },
  "orchestrators": {
    "candidate_a": {
      "endpoint": "openrouter",
      "model": "vendor/model-a",
      "input_price_per_token": 0.0000001,
      "output_price_per_token": 0.0000004,
      "reasoning": {"enabled": true}
    },
    "candidate_b": {
      "endpoint": "local",
      "model": "model-b",
      "input_price_per_token": 0,
      "output_price_per_token": 0,
      "reasoning": {}
    }
  },
  "worker": {
    "endpoint": "local",
    "model": "coding-worker",
    "input_price_per_token": 0,
    "output_price_per_token": 0,
    "reasoning": {}
  }
}
```

Keep the remaining limits and statistical parameters from the generated
template. Exactly two orchestrators are required. Static authorization,
cookie, and API-key headers are rejected; secrets must come from environment
variables.

`provider_routing: true` enables OpenRouter provider discovery and pins each
model to its discovered provider for scored calls. Generic endpoints should
normally use `false`. `include_usage: true` sends OpenRouter's explicit usage
request field; leave it `false` for generic OpenAI-compatible servers.

## Run

Offline checks require no credentials:

```bash
orchestrator-eval --config experiment.json validate
orchestrator-eval --config experiment.json power
orchestrator-eval --config experiment.json --results-dir /tmp/eval-dry dry-run
```

Live evaluation:

```bash
export OPENROUTER_API_KEY="..."
orchestrator-eval --config experiment.json run
```

By default, results are written to
`./results/<experiment_id>/`. Use `--results-dir` to choose another location.
Stages are resumable:

```bash
orchestrator-eval --config experiment.json smoke
orchestrator-eval --config experiment.json screen
orchestrator-eval --config experiment.json e2e
orchestrator-eval --config experiment.json report
```

Custom task/scenario sets can be supplied together:

```bash
orchestrator-eval \
  --config experiment.json \
  --tasks-dir ./my-tasks \
  --scenarios-dir ./my-scenarios \
  validate
```

Both directories must contain matching unique `id` and `class` fields.

## Analyze an existing run

`report` recomputes the paired statistics and writes `report.json` plus
`report.md`. It does not call a model:

```bash
orchestrator-eval \
  --config experiment.json \
  --results-dir ./results/candidate-a-vs-candidate-b \
  report
```

Export the raw run rows, screening rows, configuration, report, ledger,
provider pins, cassette statistics, and integrity hashes into one neutral JSON
file:

```bash
orchestrator-eval \
  --config experiment.json \
  --results-dir ./results/candidate-a-vs-candidate-b \
  bundle
```

## What the evaluation measures

The primary `orchestration` arm uses a deterministic Oracle worker, isolating
delegation, ownership, verification, correction, and finalization behavior.
The secondary `system` arm uses the configured coding worker behind a cassette
store. Provider failures are recorded as `infra_error` and excluded from model
success rates.

Every manager tool call is locally schema-validated. Delegated paths must be
authorized and disjoint, acceptance checks must name manifest checks, worker
test claims are checked, and finalization requires fresh harness-issued
verification evidence.

The decision rule requires the configured minimum number of complete pairs, a
paired bootstrap interval excluding zero, the configured practical effect
size, and exact McNemar significance.

## Verify a distribution

```bash
python3 scripts/verify_distribution.py dist/*.whl dist/*.tar.gz
```

The verifier rejects archives containing result directories, workspaces,
cassettes, generated reports/bundles, caches, or secret-looking API keys.
