Metadata-Version: 2.4
Name: orchestrator-uplift-harness
Version: 0.1.1
Summary: Harness-agnostic evaluation of single-agent and orchestrated coding systems
Author-email: Dark Light <darklight@noreply.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,benchmark,codex,evaluation,opencode,orchestration,swe-bench
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.14
Description-Content-Type: text/markdown

# Orchestrator Uplift Harness

[![PyPI](https://img.shields.io/pypi/v/orchestrator-uplift-harness)](https://pypi.org/project/orchestrator-uplift-harness/)
[![Python](https://img.shields.io/pypi/pyversions/orchestrator-uplift-harness)](https://pypi.org/project/orchestrator-uplift-harness/)

`ouh` compares a coding manager working alone with the same manager using
model-bound subagents. Evaluation logic is independent of Codex and OpenCode;
harness-specific behavior lives behind adapters.

The first comparison targets OpenCode with Gemma 4 31B as the manager and
`mistralai/devstral-2512` as the optional coding worker. The checked-in
condition keeps the canonical model ID
`openrouter/mistralai/devstral-2512`. Because current OpenCode model discovery
does not expose that entry, the adapter creates an isolated OpenRouter-compatible
provider alias for transport and records both identities in every trial.

## Quick start

Install the published CLI and copy its immutable input suite into a writable
working directory:

```bash
uv tool install orchestrator-uplift-harness
ouh init --destination ./orchestrator-eval
cd orchestrator-eval
ouh validate --root .
```

For development from a source checkout:

```bash
uv sync --python 3.14
uv run ouh validate --root .
uv run ouh preflight \
  --comparison comparisons/opencode_gemma4_devstral2.toml
uv run ouh run \
  --comparison comparisons/opencode_gemma4_devstral2.toml \
  --suite suites/spawn_smoke.toml \
  --repeats 1 \
  --run-id opencode-spawn-proof
uv run ouh report --run-id opencode-spawn-proof
```

## Coding-Harness auswählen

Codex und OpenCode sind als Adapter verfügbar, aber sie laufen nicht
gleichzeitig. Der Pfad hinter `--comparison` wählt genau einen Harness. Die
Comparison verweist auf zwei Condition-Dateien – Single-Agent-Baseline und
Orchestrator-Kandidat – und beide müssen denselben `harness`-Wert besitzen.

OpenCode mit Gemma 4 allein gegen Gemma 4 plus Devstral:

```bash
ouh preflight --comparison comparisons/opencode_gemma4_devstral2.toml
ouh run \
  --comparison comparisons/opencode_gemma4_devstral2.toml \
  --suite suites/smoke.toml \
  --repeats 3
```

Codex Sol allein gegen Sol mit den konfigurierten Workern:

```bash
ouh preflight --comparison comparisons/codex_sol_workers.toml
ouh run \
  --comparison comparisons/codex_sol_workers.toml \
  --suite suites/smoke.toml \
  --repeats 3
```

Die indirekte Auswahlkette lautet:

```text
--comparison
    -> baseline/candidate Condition-Dateien
    -> harness = "opencode" oder harness = "codex"
    -> ausschließlich dieser Adapter führt beide Trial-Seiten aus
```

Die CLI nennt den aktiven Harness in der Preflight- und Run-Ausgabe sowie in
`run.json`. Ein Vergleich, der Codex und OpenCode mischt, wird bei der
Konfigurationsvalidierung abgelehnt. Alternative Binary-Pfade können über
`OPENCODE_BIN`, `CODEX_BIN` und `CODEX_HOME` gesetzt werden.

Use `suites/smoke.toml` and multiple repeats for an uplift experiment. The
one-case `spawn_smoke.toml` suite proves the wiring only; it cannot establish a
quality advantage.

The native smoke suite now spans eight task shapes, including concurrency,
graph/cycle debugging, backward-compatible feature work, and security
hardening. For real-repository evidence, `benchmarks/` contains pinned,
deterministically stratified SWE-bench Verified and Multilingual selections.
The package ships selection metadata and digests, not upstream repositories,
test patches, gold patches, model outputs, or run artifacts.

## Included inputs

- Eight deterministic native coding tasks and hidden workspace graders.
- Codex and OpenCode single-agent/orchestrated conditions.
- OpenCode Gemma 4 31B plus `mistralai/devstral-2512` worker condition.
- SWE-bench Verified selections with 24 and 100 pinned instance IDs.
- SWE-bench Multilingual selections with 18 and 42 pinned instance IDs.
- The checkout and official Docker-evaluator bridge for real SWE-bench issues.

All benchmark families remain separate in reports. Function synthesis,
repository issue resolution, and terminal work are not collapsed into one
opaque score.

## Evaluation contract

- Baseline and candidate receive the same task and fresh fixture state.
- The baseline technically denies child creation.
- Delegation-required cases must produce a child session; negative controls
  must not delegate.
- Quality comes only from an external deterministic workspace grader.
- Runtime status, quality, tokens, cost, latency, and cleanup are stored as
  separate observations.
- Every observed child records its agent, canonical model, transport model,
  parent session, and exact ID.
- OpenCode state is isolated per trial. Cleanup deletes only observed session
  IDs and verifies that the trial database contains none of them afterward.
- Large or truncated OpenCode exports fall back to the same isolated session
  database for identity and usage; failure of both sources blocks the trial.

Generated runs live under `artifacts/<run-id>/` and are intentionally ignored
by Git because they may contain large transient workspaces. The final report is
available as JSON, Markdown, and static HTML inside that directory.

## Requirements

- Python 3.14 or newer.
- Git for isolated benchmark workspaces.
- OpenCode and/or the Codex app binary for their respective adapters.
- Docker plus a separately installed, pinned SWE-bench evaluator only when
  running external SWE-bench instances.

Provider credentials remain in the native harness credential stores. They are
not copied into the generated working directory or distribution archives.

## License

MIT. Upstream benchmark repositories and datasets retain their own licenses.

## Development

```bash
uv sync --python 3.14
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy src
```

See the versioned design and implementation plan under `docs/superpowers/`.
