Metadata-Version: 2.4
Name: benchspan
Version: 0.1.1
Summary: Run agentic benchmarks end-to-end
Project-URL: Homepage, https://benchspan.com
Project-URL: Documentation, https://docs.benchspan.com
Author-email: Known Quantity Labs <founders@benchspan.com>
License: Proprietary
License-File: NOTICE
Keywords: agent,benchmark,evaluation,llm,testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: requests>=2.28.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.9.0
Provides-Extra: local
Requires-Dist: docker>=7.0.0; extra == 'local'
Requires-Dist: pyarrow>=14.0.0; extra == 'local'
Description-Content-Type: text/markdown

# Benchspan

Run agentic benchmarks end-to-end. One CLI to evaluate any AI agent against 40+ benchmarks with parallel cloud execution, reproducible environments, and a team dashboard.

## Install

```bash
pip install benchspan
```

Requires Python 3.11+.

## Quick start

```bash
# Login to your Benchspan account
benchspan login

# Scaffold a new agent
benchspan init ./my-agent

# Edit my-agent/runner.sh to invoke your agent, then run:
benchspan run --benchmark swebench --agent ./my-agent --instances 10
```

Your agent runs in parallel Docker containers on Benchspan's infrastructure. Results stream to your terminal and appear on the [dashboard](https://benchspan.com).

## Built-in agents

Benchspan ships with pre-configured agents you can use immediately:

```bash
# List available agents
benchspan agents

# Run with Claude Code
benchspan run --benchmark swebench --agent claude-code --instances 10

# Run with OpenHands
benchspan run --benchmark swebench --agent openhands --instances 10
```

Set the required API keys on your [dashboard](https://www.benchspan.com) under Profile > Secrets.

## Writing an agent

An agent is a directory with a `runner.sh`. Benchspan injects it into a benchmark container and runs it:

```bash
#!/bin/bash
# Available environment variables:
#   $PROBLEM_STATEMENT  — what to solve (natural language)
#   $WORKING_DIR        — where to do your work
#   $OUTPUT_DIR         — where to write outputs
#   $INSTANCE_ID        — unique instance identifier

cd "$WORKING_DIR"

# Invoke your agent here
```

If your agent can read a problem statement and write code, it works on Benchspan. See the [full guide](https://docs.benchspan.com/onboard-your-agent/runner-sh) for patterns and best practices.

## Benchmarks

40+ benchmarks available out of the box:

```bash
# List all benchmarks
benchspan benchmarks list

# List instances for a benchmark
benchspan benchmarks instances swebench

# Run with a subset
benchspan run --benchmark swebench.django --agent ./my-agent
```

Includes SWE-bench, HumanEval, Terminal-Bench, BFCL, SimpleQA, and [many more](https://docs.benchspan.com/benchmarks/overview).

## Managing runs

```bash
# List your runs
benchspan runs list

# List all team runs
benchspan runs list --all

# Show run details
benchspan runs show <run_id>

# Compare two runs side-by-side
benchspan runs compare <run_a> <run_b>

# Watch a running benchmark
benchspan runs watch <run_id>

# Cancel a run
benchspan runs cancel <run_id>

# Retry failed instances
benchspan retry <run_id> --agent ./my-agent
```

## Key features

- **Parallel execution** — hundreds of instances run simultaneously, not sequentially
- **Resume on failure** — retry only failed instances, don't re-run the whole suite
- **Reproducible** — every instance runs in an identical Docker container
- **Team dashboard** — all results in one place at [benchspan.com](https://benchspan.com)
- **Any agent** — if it runs via bash, it works on Benchspan
- **Private benchmarks** — bring your own proprietary benchmarks

## Documentation

Full documentation at [docs.benchspan.com](https://docs.benchspan.com), including:

- [Quickstart](https://docs.benchspan.com/getting-started/quickstart) — up and running in 5 minutes
- [Writing runner.sh](https://docs.benchspan.com/onboard-your-agent/runner-sh) — agent integration patterns
- [Healthcheck testing](https://docs.benchspan.com/onboard-your-agent/healthcheck) — validate your agent before running real benchmarks
- [CLI reference](https://docs.benchspan.com/cli-reference/run) — all commands and options
- [Troubleshooting](https://docs.benchspan.com/onboard-your-agent/common-issues) — common issues and fixes

## Links

- [Website](https://benchspan.com)
- [Documentation](https://docs.benchspan.com)
- [Book a demo](https://cal.com/team/benchspan/20-minute-meeting)
- [Email](mailto:founders@benchspan.com)
