Metadata-Version: 2.5
Name: ageval-cli
Version: 0.7.1
Summary: ageval — agent evaluation runtime (Attempt pipeline + environment kinds)
Project-URL: Homepage, https://github.com/ZJU-REAL/AgEval
Project-URL: Repository, https://github.com/ZJU-REAL/AgEval
Project-URL: Issues, https://github.com/ZJU-REAL/AgEval/issues
Author: ZJU-REAL
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: agent-client-protocol==0.12.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: typer>=0.16.0
Provides-Extra: all
Requires-Dist: daytona>=0.21.0; extra == 'all'
Requires-Dist: deepseek-harness-sdk==0.1.0rc6; extra == 'all'
Requires-Dist: e2b>=2.25.0; extra == 'all'
Requires-Dist: httpx[socks]>=0.28.1; extra == 'all'
Requires-Dist: mini-swe-agent>=2.0; extra == 'all'
Requires-Dist: nooa>=0.0.7; extra == 'all'
Requires-Dist: socksio>=1.0.0; extra == 'all'
Provides-Extra: daytona
Requires-Dist: daytona>=0.21.0; extra == 'daytona'
Requires-Dist: socksio>=1.0.0; extra == 'daytona'
Provides-Extra: dev
Requires-Dist: pyright>=1.1.400; extra == 'dev'
Requires-Dist: pytest>=8.3.0; extra == 'dev'
Requires-Dist: ruff>=0.12.0; extra == 'dev'
Provides-Extra: dsh
Requires-Dist: deepseek-harness-sdk==0.1.0rc6; extra == 'dsh'
Provides-Extra: e2b
Requires-Dist: e2b>=2.25.0; extra == 'e2b'
Provides-Extra: miniswe
Requires-Dist: mini-swe-agent>=2.0; extra == 'miniswe'
Requires-Dist: socksio>=1.0.0; extra == 'miniswe'
Provides-Extra: nooa
Requires-Dist: httpx[socks]>=0.28.1; extra == 'nooa'
Requires-Dist: nooa>=0.0.7; extra == 'nooa'
Requires-Dist: socksio>=1.0.0; extra == 'nooa'
Provides-Extra: registry
Requires-Dist: boto3>=1.35.0; extra == 'registry'
Requires-Dist: psycopg[binary]>=3.2.0; extra == 'registry'
Requires-Dist: starlette>=0.41.0; extra == 'registry'
Requires-Dist: uvicorn>=0.32.0; extra == 'registry'
Description-Content-Type: text/markdown

# ageval

**agent eval** — lock a dataset, bind environment and agent, run the same task.

<p align="center">
  <a href="README.md">English</a> · <a href="README.zh-CN.md">中文</a>
  &nbsp;&nbsp;
  <a href="https://github.com/ZJU-REAL/AgEval/releases"><img alt="Release" src="https://img.shields.io/github/v/release/ZJU-REAL/AgEval?display_name=tag&sort=semver"></a>
  <a href="https://www.python.org/downloads/"><img alt="Python" src="https://img.shields.io/badge/python-3.12%2B-blue"></a>
</p>

> [!IMPORTANT]
> Most agent evaluation still sits at the **model**: same prompts, same tool contract, different weights or APIs. That split is failing. A shippable agent is a Model plus a Harness: the same weights on a different coding-agent runtime, tool policy, or isolation change behavior and cost. Harness has to be a first-class eval axis, locked with the Model and the environment.

**ageval** decouples the runtime of a bench run. Environment and Agent combine through plugins, so one `run.py` runs under each binding.

<p align="center">
  <img src="docs/assets/why-ageval.jpg" alt="N environments × M harnesses: each combination would need its own scaffold; ageval composes environment and harness through plugins, so one eval runs anywhere." width="800">
</p>

## Contents

- [What it is](#what-it-is)
- [How it works](#how-it-works)
- [Features](#features)
- [Getting started](#getting-started)
- [Architecture](#architecture)
- [Project structure](#project-structure)
- [Docs](#docs)

## What it is

ageval treats **Harness** as a first-class evaluation axis.

- **The unit of delivery is a dataset.** A dataset holds tasks; each task owns the loop (`run.py`), the score (`evaluator.py`), and gold. Environment and Agent bind on the job, not in the task.
- **An Attempt is visible.** Order is environment → run → evaluate → record; cleanup always runs.
- **Environment is injected by name.** The environment winner **exports** the service name `environment`. An agent backend **injects** that name and **requires** the capabilities it needs (`attach_stdio` or `exec`). Missing capability fails at lock. Calls stay on the Protocol. Default is the host and local docker; cloud sandboxes [e2b](https://e2b.dev) and [daytona](https://www.daytona.io) are optional.
- **Coding agents enter through plugins.** Default is [ACP](https://agentclientprotocol.com) ([pi](https://pi.dev), [Codex](https://github.com/openai/codex), [Claude Code](https://github.com/anthropics/claude-code), [OpenCode](https://github.com/sst/opencode)); heterogeneous harnesses such as [nooa](https://github.com/NVIDIA-NeMo/labs-OO-Agents), [dsh](https://github.com/deepseek-ai/deepseek-harness), and [miniswe](https://github.com/SWE-agent/mini-swe-agent) join the same way. Open slots extend the harness set and run on the same Attempt path and leaderboard.

## How it works

```text
            you ── lock / run ──►  ┌─────────────────┐
                                   │     Attempt     │  lock dataset · digest
                                   │   ageval core   │  environment → run
                                   │                 │  evaluate → record
                                   │                 │  finally cleanup
                                   └────────┬────────┘
                                            │ opens one environment
              ┌─────────────────────────────┼─────────────────────────────┐
              ▼                             ▼                             ▼
        ┌───────────┐                 ┌───────────┐                 ┌───────────┐
        │   local   │                 │  docker   │                 │ e2b/ssh/daytona │
        └─────┬─────┘                 └─────┬─────┘                 └─────┬─────┘
              └──────── Protocol: upload · exec · attach_stdio ───────────┘
                                            │
                                            ▼
                                   ┌─────────────────┐
                                   │     run.py      │  task loop · Agent invoke
                                   │  ACP / plugin   │  plugin inlet
                                   └────────┬────────┘
                                            ▼
                                   ┌─────────────────┐
                                   │  evaluator.py   │  only source of PASS
                                   └─────────────────┘
```

1. **Lock the experiment.** Dataset, Environment, and Agent are composed into a digest. Secrets remain locators; they are not stored as plaintext in the lock.
2. **Open an environment.** Host, docker, a cloud sandbox, or a remote. Insufficient capability or missing credentials fail before the environment opens.
3. **Execute the task.** `run.py` owns the loop, tools, and Agent invocation. Changing the environment or the Agent does not require rewriting the task.
4. **Score independently.** Gold enters the environment at this point. `evaluator.py` binds PASS / FAIL / ERROR. Cleanup always runs.

## Features

**Evaluation**

- **Lock one experiment.** Dataset, Harness, and environment lock together into a reproducible digest.
- **One task, a suite, a matrix, or repeats.** Run a single task, a full dataset, a parameter matrix on one task, or multiple independent Attempts of the same job (pass@k).
- **Scoring is separate from the Agent.** Gold does not enter the Agent view. PASS comes only from `evaluator.py` (deterministic script by default; optional `Agent.session` as LLM-as-judge). Trajectories are for inspection.
- **Limits are enforced before invocation.** Wall time, memory, processes, and invocation ceilings are set by the runtime before invoke.

**Composition**

- **One `run.py` under each binding.** Environment and Agent combine through plugins. Default is [ACP](https://agentclientprotocol.com); heterogeneous harnesses such as [nooa](https://github.com/NVIDIA-NeMo/labs-OO-Agents), [dsh](https://github.com/deepseek-ai/deepseek-harness), and [miniswe](https://github.com/SWE-agent/mini-swe-agent) join through the same plugin path. Open slots extend or replace a harness and participate in the same Attempt path and leaderboard.
- **Agent packages.** A harness (`ageval.agent/1`: executor, entry, overlays). Shipped mechanism cards bind with `--agent pi` (no install). Custom overlay packs still `ageval agent install` then `--agent org/name@version`. `binding.model` is the default; `--model` overrides this run.
- **Multiple roles and sessions.** The task owns dialog, tools, and handoff; the runtime supplies the environment and the Agent inlet.
- **Validate before invoke.** Capabilities and credentials are checked before the Agent is called; absence fails and invoke does not start.

**Environment**

- **Host, container, cloud sandbox, remote — one Protocol.** local, docker, [e2b](https://e2b.dev), ssh, [daytona](https://www.daytona.io): `upload` / `exec` / `attach_stdio`.
- **Visibility is isolated.** The Agent sees only the projected workspace; gold and host credentials do not enter the task default environment.
- **Official Attempt image.** Docker installs ACP entries at build time; they are not installed at invoke.

**Results**

- **Local Viewer.** Inspect trajectory, environment, and score along Jobs → Tasks → Attempt.
- **Sealed trajectory.** Export a copy without modifying the score.
- **Hub.** Publish datasets, plugins, and Agent packages; upload suites. Organizations manage members, visibility, and versions. The public Leaderboard lists complete, release-bound suites only. Operators can `docker compose -f services/registry/docker-compose.yml up -d` (Postgres, object store, Registry, Hub) and pull `ghcr.io/zju-real/ageval-hub` / `ageval-registry` from a release tag.

**Authoring**

- **The task owns only that task.** Loop, tools, scoring, and gold; orchestration does not belong in the task.
- **SDK is optional.** Sessions, tools, terminals. It does not decide PASS and does not hold host credentials.

## Getting started

Requires [uv](https://docs.astral.sh/uv/) and CPython **3.12+**. A live coding-agent run also requires a host ACP entry and credentials. `ageval lock` does not.

```bash
git clone https://github.com/ZJU-REAL/AgEval.git
cd ageval
uv sync --frozen --all-packages
uv run ageval -V
```

```bash
uv run ageval tasks examples/datasets/minimal-demo
uv run ageval lock examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg \
  --profiles examples/datasets/minimal-demo/profiles.e2b-acp.yaml --probe
uv run ageval executors -v
uv run ageval view examples/datasets/minimal-demo --no-browser
```

Default `examples/datasets/minimal-demo` profiles use `environment: docker`. Bind a shipped harness with `--agent pi` (no install). Optional `--model` overrides this run. Custom overlay packs still use `ageval agent install` then `--agent org/name@version`.

In-repo examples: [`examples/README.md`](examples/README.md) — `minimal-demo`, a five-task `tau3-airline` cut, and catalog Agents.

## Architecture

```text
  ageval.yaml + task.yaml + profiles.yaml
                 │
                 ▼
           ageval lock                 digest · extension_bindings
                 │
                 ▼
           ageval run  ── Attempt ── environment → run → evaluate → record
                 │                      finally cleanup
                 ▼
        .ageval/runs/<id>/             lock.json · result.json · trajectory.jsonl
                 │
      ┌──────────┼──────────┐
      ▼                     ▼
 ageval view          Registry / Hub
 local Jobs           publish · upload-suite · Leaderboard
```

- lock is the normative gate: unknown format fails once. Plugins change the binding, not the five Attempt phases.
- Attempt owns identity, deadlines, cleanup, and the score.
- Local Viewer reads files; Hub talks to Registry.

## Project structure

Simplified from [`ARCHITECTURE.md`](ARCHITECTURE.md). Generated trees (`.ageval/`, `.venv/`) are not source.

```text
ageval/
├── src/ageval/
│   ├── cli/                         # argv, help, exit code
│   ├── application/
│   │   ├── composition.py           # sole production wiring; CLI imports build_* here
│   │   ├── lock.py                  # load_and_lock
│   │   ├── run.py                   # mint identity → run_attempt
│   │   ├── campaign.py / suite/     # matrix · suite · Always-k
│   │   └── agent_ops/ / plugin_ops / registry_ops/
│   ├── attempt/                     # visible pipeline
│   │   ├── __init__.py              # run_attempt
│   │   └── phases/                  # environment → run → evaluate → record · cleanup
│   ├── config/                      # dataset + task.yaml + profiles
│   ├── environments/protocol.py     # EnvironmentProvider · caps; no vendor SDK
│   ├── plugins/
│   │   ├── slots.py                 # exclusive / chain
│   │   └── contrib/                 # acp · local · docker · e2b · daytona · ssh
│   ├── runtime/                     # identity, parent Agent Service, task_worker
│   ├── evaluation/                  # barrier + bind PASS
│   └── evidence/                    # trajectory.jsonl layout
├── src/ageval_sdk/                  # ageval_sdk for run.py (no PASS, no host credentials)
├── plugins/                         # external ageval.plugin/1 (nooa, dsh, miniswe, …)
├── examples/
│   ├── datasets/
│   │   ├── minimal-demo/            # terminal-jsonl-agg · tau2-dialog-min · multiagent-env-min
│   │   └── tau3-airline/            # airline-00 … airline-04
│   └── agents/                      # ageval.agent/1
├── apps/viewer                      # ageval view SPA
├── apps/hub                         # Hub SPA
├── services/registry/               # package + results HTTP
├── docker/attempt/                  # official image; ACP entries baked in
├── docs/                            # mechanism design
└── website/                         # product docs
```

## Docs

- Usage: [`website/`](website/)
- Design: [`docs/`](docs/README.md)
- Examples: [`examples/README.md`](examples/README.md)
- [`AGENTS.md`](AGENTS.md)
- [`ARCHITECTURE.md`](ARCHITECTURE.md)
