Metadata-Version: 2.4
Name: aehf
Version: 0.1.0
Summary: agent evaluation harness framework
Project-URL: Homepage, https://github.com/salasya2/aehf
Project-URL: Repository, https://github.com/salasya2/aehf
Author-email: Sai Teja Alasyam <salasya2@asu.edu>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: anthropic
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv
Requires-Dist: pyyaml
Requires-Dist: typer
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Description-Content-Type: text/markdown

 ![demo](docs/demo.gif)
# aehf — agent evaluation harness framework


A framework-agnostic harness for evaluating tool-using LLM agents: run a suite
of cases, judge the transcripts, and get **statistically honest** pass rates and
regression checks — not a single flaky number.

The point of aehf is not just to score agents, but to **validate the judge doing
the scoring first**, then use that trusted judge to make claims you can defend.

## Headline finding

On 90 hand-labeled agent transcripts:

| Judge          | Raw agreement | Cohen's kappa |
|----------------|---------------|---------------|
| AssertionJudge | 97%           | **0.000**     |
| LLMJudge (v1)  | 97%           | **0.894**     |

Both judges agree with the human ~97% of the time, yet one is worthless and one
is excellent. Raw agreement doesn't correct for chance given the base rates;
**Cohen's kappa does.** The assertion judge passes essentially everything, so its
97% is meaningless (kappa 0); the calibrated LLM judge reaches "almost perfect"
agreement. This gap is the whole reason the harness measures kappa before
trusting any downstream number. (Details and caveats: `docs/calibration.md`.)

## Architecture — everything is a protocol

The eval core is decoupled from any provider by three Python `Protocol`s. The
runner, judges, and stats know only the interfaces:

```mermaid
                  flowchart TD
    %% Main Flow
    EvalCase([EvalCase YAML]) --> Runner

    subgraph ProtocolBlock [Agent & Evaluation Lifecycle]
        Runner[runner] --> Run[Agent.run case -> Transcript]
        Adapter[AnthropicAdapter, FakeAgent, ...] -.->|Configures| Run
        
        Run --> Tool[ToolProvider.execute]
        ToolMode[mock / record / replay] -.->|Modes| Tool
        
        Tool --> Judge[Judge.score case, transcript]
        JudgeType[AssertionJudge, LLMJudge] -.->|Types| Judge
        
        Judge --> Verdict([Verdict])
    end

    Verdict --> Stats[stats]
    Verdict --> Calibration[calibration]
    Verdict --> Regression[regression]

    %% Labels & Details
    Stats --> StatsDetails[Wilson, McNemar, n=k]
    Calibration --> CalDetails[kappa vs human]
    Regression --> RegDetails[store, diff, CI gate]

    %% Right-side Protocol Tags
    ProtocolBlock -.-> AgentProto[Agent protocol<br><i>bring your own: LangChain, raw, ...</i>]
    Tool -.-> ToolProto[ToolProvider protocol]
    Judge -.-> JudgeProto[Judge protocol]

    %% Styling
    style EvalCase fill:#f9f9f9,stroke:#333,stroke-width:2px
    style Verdict fill:#f9f9f9,stroke:#333,stroke-width:2px
    style ProtocolBlock fill:#fff,stroke:#333,stroke-dasharray: 5 5
```

The agent under test is a black box behind `Agent`; aehf evaluates whatever
implements `run(case) -> Transcript`. The shipped `AnthropicAdapter` is a
reference implementation, not the framework.

## What's inside

- **Runner** — async, budget/timeout-enforced, captures agent crashes as failed
  cases (never harness crashes), bounded concurrency.
- **Tools** — mock fixtures, plus record/replay for deterministic offline reruns.
- **Judges** — programmatic assertions and a versioned LLM judge with structured
  (forced-tool) verdicts; calibrated against human labels with Cohen's kappa.
- **Stats** — n-sample execution, per-case Wilson confidence intervals, flakiness
  flags, and McNemar's exact paired test for model/prompt comparison.
- **Regression** — results store keyed by (git SHA, model, judge version),
  `aehf diff`, a markdown scorecard, and a PR-gate GitHub Action.

## Install

```bash
pip install -e ".[dev]"        # editable, with dev tooling
```

Set `ANTHROPIC_API_KEY` (a `.env` file is loaded automatically) for anything that
runs a live model. The test suite runs **green without a key** — CI is free and
offline.

## Quickstart

```bash
# run a suite with mock tools, assertion judge (no API cost for the judge)
aehf run examples/calibration_suite.yaml anthropic mock --judgechoice assertion

# n-sampled run: per-case pass rate + Wilson CI + flakiness
aehf run examples/calibration_suite.yaml anthropic mock --n-samples 5

# calibrate a judge against human labels -> Cohen's kappa + disagreements
aehf calibrate labels/labels_filled.jsonl llm --prompt-version v1

# compare two saved runs with McNemar's test
aehf compare runA.json runB.json

# regression diff between two stored runs (CI gate)
aehf diff <base-sha> <head-sha> --store .aehf
```

Full command list: `run`, `calibrate`, `export-labels`, `label`, `compare`,
`diff`.

## Two findings, in one narrative

1. **Judge calibration** — assertions kappa=0.00 -> LLM judge kappa=0.894 (n=90).
   The instrument is validated.
2. **Model comparison** — using that validated judge, Haiku and Sonnet were
   **statistically indistinguishable** on the suite (15/18 vs 14/18, McNemar
   p=1.0). The apparent edge is noise; a single-run comparison would have misled.
   (`docs/calibration.md`.)

Both come with honest caveats stated in the docs (single annotator; a saturated
suite that can't discriminate the two models). Reporting the null result honestly
is the point — see `docs/decision.md` for every design choice and rejected
alternative.

## Scope (v0.1)

- Ships an Anthropic reference adapter and judge. The `Agent`/`Judge`/`ToolProvider`
  protocols make other providers drop-in; a second provider is the planned
  demonstration of the framework-agnostic claim.
- The CI eval gate compares **committed** baseline runs (golden-file style),
  because replay pins tool results but not the model. See `docs/eval-gate.md`.
- Tested: 115 tests, `mypy --strict` clean, ruff clean.
