Metadata-Version: 2.4
Name: evalpilot
Version: 0.1.1
Summary: A general framework for online-driving, recording, and scoring AI agents against task suites.
Author-email: ailearneryang <ailearneryang@gmail.com>
License: MIT
Keywords: agent,evaluation,eval,llm,benchmark,testing,cli
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Requires-Dist: PyYAML>=6.0
Requires-Dist: jinja2>=3.1
Requires-Dist: click>=8.1
Requires-Dist: anthropic>=0.40
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: webui
Requires-Dist: playwright>=1.40; extra == "webui"
Dynamic: license-file

# evalpilot

A general framework for evaluating agents. It online-drives any agent through a
task suite, records its trajectory, scores it with four pluggable scorers
(objective checkpoints, LLM-as-judge, trajectory/process, efficiency), and emits
a JSON + static HTML report.

## Install

Published on PyPI as `evalpilot`; the CLI command is `evalpilot`.

Run with no install (recommended):

    uvx evalpilot run

Install as a tool:

    uv tool install evalpilot      # or: pipx install evalpilot

For browser-UI (`webui`) agents, install the extra and the browser once:

    uv tool install "evalpilot[webui]"
    playwright install chromium

From source (development):

    python -m venv .venv
    .venv/bin/pip install -e ".[dev]"

## Quick start

    .venv/bin/evalpilot run --suite suites/example.yaml --agent configs/function-echo.yaml --run-id demo --no-llm-judge
    open runs/demo/report.html

Or run it interactively — omit `--suite`/`--agent` and a wizard walks you through
defining the agent and tasks (no YAML needed):

    evalpilot run

Browse all runs at once:

    open runs/index.html

## Concepts

- **Task / TaskSuite** — declarative units of evaluation (prompt, setup,
  checkpoints, rubric, budget).
- **AgentAdapter** — narrow interface that drives the agent under test. Built-in:
  `function`, `cli`, `http`, `webui`. Add a new agent by writing one subclass.
- **Trajectory** — the recorded run (steps, output, usage, final state). Sole
  data source for all scorers.
- **Scorers** — checkpoint, llm_judge, trajectory, efficiency. Read-only,
  pluggable, weighted. Checkpoints include file/command checks plus
  `output_contains` / `output_matches` for grading text answers objectively.
- **Runner / Reporter** — orchestrate and report.

## Adding an agent

Write a YAML config selecting an adapter type. For a CLI agent:

    type: cli
    name: my-agent
    command: my-agent --prompt {prompt} --json-stream

For an in-process Python callable:

    type: function
    name: my-agent
    entrypoint: my_module:run

For an OpenAI-style HTTP endpoint:

    type: http
    name: my-agent
    url: http://localhost:8000/v1/chat/completions

For a browser-based (web UI) agent:

    type: webui
    name: my-chat-agent
    url: https://my-agent.example.com/chat
    input_selector: "textarea"
    output_selector: ".message.assistant:last-child"

The `webui` adapter needs Playwright (an optional extra):

    pip install -e ".[webui]"
    playwright install chromium

It types the prompt into `input_selector`, waits until the reply text in
`output_selector` stops changing (default 1.5s), then scrapes it. Reuse a
logged-in session with `storage_state: ./auth.json`.

## Scoring weights

Defaults: checkpoint=3, llm_judge=2, trajectory=1, efficiency=0. Override per
suite (`default_weights`) or per task (`weights`). Each scorer's sub-score and
its reason are always preserved in the report, not just the aggregate.

## Commands

    evalpilot run --suite SUITE.yaml --agent AGENT.yaml [--run-id ID] [--out DIR] [--no-llm-judge]
    evalpilot run                            # interactive wizard (builds agent + tasks)
    evalpilot report --run runs/ID           # re-render report.html from result.json
    evalpilot compare runs/a runs/b          # combined leaderboard across runs
    evalpilot index [--out runs]             # overview page listing all runs -> runs/index.html

## Testing

    .venv/bin/python -m pytest

All scorer/adapter tests inject fakes (fake CLI scripts, fake HTTP post, fake
LLM completion), so the suite runs fully offline with no network or API key.
