Metadata-Version: 2.4
Name: agentverdict
Version: 0.1.0
Summary: Benchmark and compare AI agents with deterministic, reproducible tests.
Author: AgentVerdict Contributors
License-Expression: MIT
Keywords: ai,agents,benchmarking,evaluation,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Requires-Dist: PyYAML>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6.0; extra == "dev"
Dynamic: license-file

# AgentVerdict

**Benchmark and compare AI agents with reproducible tests.**

AgentVerdict is an open-source Python CLI for testing agent behavior across
deterministic benchmarks, repeated runs, and side-by-side comparisons.

```bash
pip install agentverdict
agentverdict test my_agent.py
```

## Quickstart

AgentVerdict currently loads agents through a small Python contract. Create
`my_agent.py`:

```python
def run_agent(task: str) -> str:
    return f"Received: {task}"
```

Run it against the bundled benchmark:

```bash
agentverdict test my_agent.py
agentverdict test my_agent.py --runs 5
```

For a representative passing run, use the included example:

```bash
agentverdict test examples/benchmark_agent.py --runs 5
```

```text
AgentVerdict

Agent: benchmark_agent
Benchmark: General Agent Benchmark
Runs per task: 5

Tasks:                  10
Runs per task:           5
Total attempts:          50
Passed attempts:         50
Failed attempts:         0
Execution errors:        0
Overall reliability:     100.0%
Average consistency:     100.0%
Error rate:              0.0%
Average latency:          0.000s
```

The exact latency depends on your machine and the agent being tested.

## Why AgentVerdict?

An AI agent can work in a demo and still behave differently across repeated
runs. AgentVerdict measures the same agent against the same deterministic
tasks so you can inspect:

- correctness
- reliability across attempts
- literal output consistency
- execution errors
- local execution latency

## Compare agents

Run two or more agents against the same benchmark:

```bash
agentverdict compare \
    examples/benchmark_agent.py \
    examples/simple_agent.py \
    --runs 5
```

Each agent receives the same benchmark, task order, evaluator semantics, and
runs per task. The comparison presents the individual metrics rather than
inventing an arbitrary weighted overall score, so tradeoffs remain visible.

## Benchmark format

Benchmarks are versioned YAML files. A minimal definition looks like this:

```yaml
version: "1"
name: Example Benchmark

tasks:
  - id: capital-france
    name: Capital of France
    category: factual
    prompt: "Name the capital of France."
    evaluation:
      type: contains
      expected: "Paris"
```

Current deterministic evaluators are `exact_match`, `contains`, and `regex`.
Use `agentverdict benchmark-info` to inspect a bundled benchmark or YAML file.
The [result schema documentation](docs/result-schema.md) describes the
structured output format.

## Metrics

**Reliability** is passed attempts divided by total attempts.

**Output consistency** is, for each task, the frequency of its most common
valid literal output, averaged across tasks. It is case-sensitive,
whitespace-sensitive, deterministic, and not semantic similarity. Execution
errors are excluded because they have no output.

**Error rate** is execution errors divided by total attempts.

**Latency** is measured locally for each agent execution. It reflects the
machine, environment, and any services contacted by the agent.

### PASS, FAIL, and ERROR

- **PASS** — the agent returned a valid output and satisfied the evaluator.
- **FAIL** — the agent executed successfully but did not satisfy the evaluator.
- **ERROR** — the agent failed to produce a valid output.

## Reports

Generate reports locally while testing or comparing agents:

```bash
agentverdict test agent.py --runs 5 --report html
agentverdict compare agent_a.py agent_b.py --report all
```

`--report` accepts `none`, `json`, `html`, or `all`. HTML reports are
standalone and require no server. JSON reports are useful for tooling and CI.
Reports are generated locally; AgentVerdict does not upload them.

## CLI reference

| Command | Purpose |
| --- | --- |
| `agentverdict run` | Run a Python-file agent once. |
| `agentverdict test` | Test one agent against a benchmark. |
| `agentverdict compare` | Compare two or more agents on one benchmark. |
| `agentverdict benchmarks` | List bundled benchmarks. |
| `agentverdict benchmark-info` | Inspect a benchmark definition. |
| `agentverdict --version` | Show the installed version. |

For detailed options, run `agentverdict COMMAND --help`.

## CI usage

`agentverdict test agent.py` can run directly in CI.

- `test`: exit `0` when every attempt passes, `1` when the benchmark completes
  with failures or errors, and `2` for a configuration or application error.
- `compare`: exit `0` when comparison completes and `2` for a configuration or
  application error.

## Installation and requirements

AgentVerdict requires Python 3.11 or newer.

For the public release:

```bash
pip install agentverdict
```

For local development:

```bash
python -m pip install -e ".[dev]"
```

## Architecture

```text
Agent
  ↓
AgentAdapter
  ↓
Benchmark Runner
  ↓
Evaluator
  ↓
Structured Results
  ├── Terminal
  ├── JSON
  └── HTML
```

## Safety and privacy

Agent files are executable Python code. AgentVerdict v0.1.0 does not sandbox
them, so only run agent files you trust. See [SECURITY.md](SECURITY.md).

AgentVerdict has no telemetry and does not upload reports. Benchmarks and their
outputs remain local unless the agent you supply itself contacts an external
service.

## Roadmap

See [ROADMAP.md](ROADMAP.md) for potential post-0.1.0 directions.

## Contributing

Contributions are welcome, especially benchmark packs, evaluator types,
framework adapters, reporting improvements, and documentation. Read
[CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.

## License

AgentVerdict is licensed under the [MIT License](LICENSE).
