Metadata-Version: 2.5
Name: inspect-rogue-arena
Version: 0.4.0
Summary: Inspect sandbox provider for running AI model evals on Rogue Arena cyber ranges
Project-URL: Homepage, https://github.com/rogue-labs-cyber/inspect-rogue-arena
Project-URL: Repository, https://github.com/rogue-labs-cyber/inspect-rogue-arena
Project-URL: Issues, https://github.com/rogue-labs-cyber/inspect-rogue-arena/issues
Project-URL: Rogue Arena, https://arena.roguelabs.io
Author-email: Rogue Labs <info@roguelabs.io>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: evals,inspect,inspect_ai,rogue-arena,sandbox,security
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Requires-Dist: anyio>=4.0
Requires-Dist: httpx>=0.27
Requires-Dist: inspect-ai<0.3.274,>=0.3.260
Requires-Dist: pydantic>=2.0
Provides-Extra: all-providers
Requires-Dist: anthropic>=0.40; extra == 'all-providers'
Requires-Dist: google-genai>=1.0; extra == 'all-providers'
Requires-Dist: openai>=1.0; extra == 'all-providers'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: google
Requires-Dist: google-genai>=1.0; extra == 'google'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: openrouter
Requires-Dist: openai>=1.0; extra == 'openrouter'
Description-Content-Type: text/markdown

# inspect-rogue-arena

[Inspect](https://inspect.aisi.org.uk/) sandbox provider for running AI model
evals on [Rogue Arena](https://arena.roguelabs.io) cyber ranges. You write an
ordinary Inspect task; each sample claims a model-eval copy, runs on its own
range, and Rogue Arena scores the eval's hops when the trial closes.

- The model runs **on your machine**, through your own model-provider key (for
  example Anthropic, Google, OpenAI or OpenRouter).
- The target range runs **in Rogue Arena**, reached with a personal token.
- Scoring is done **by Rogue Arena** (it checks each hop); the package never
  computes a score or submits a key.

Support for running Rogue Arena evals from [Harbor](https://harborframework.com)
is in development.

## Quickstart

1. **Get access.** Running evals needs a Rogue Arena account on a team with
   model-eval access. Access is not self-service: to get access to our library
   of ready-built cyber-range eval scenarios and model-eval runs, email
   <info@roguelabs.io>. Once you're added, create your token below. You will
   also need an API key for a model provider (for example Anthropic, Google,
   OpenAI or OpenRouter).

2. **Install** (Python 3.10+), with the extra for your provider:

   ```bash
   pip install "inspect-rogue-arena[<provider>]"
   # <provider> is one of: anthropic, google, openai, openrouter
   # or [all-providers] for all of them
   ```

   The extra pulls in that provider's client library.

3. **Create your token.** Sign in at <https://arena.roguelabs.io> and go to
   **Team Hub → Model Testing → Tokens** (`/team-hub/model-testing/tokens`).
   Click **Create token**, give it a **Token label** (e.g. `laptop` or
   `ci-runner`), choose **Expires in** (7, 30, 90 or 365 days), and submit. The
   token (`ra_eval_…`) is shown **once**: "Copy it now. You won't see it
   again." It runs any published eval you can deploy, as you.

4. **Set your environment:**

   ```bash
   export ROGUE_ARENA_BASE_URL=https://arena.roguelabs.io
   export ROGUE_ARENA_TOKEN=ra_eval_…     # the token you just copied
   # plus your model-provider key, e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY,
   # GOOGLE_API_KEY or OPENROUTER_API_KEY
   ```

   Keep the token in the environment: it is never a config field, so it never
   lands in an Inspect log, and the CLI redacts it from errors.

5. **Check the token and list evals:**

   ```bash
   rogue-arena-inspect whoami   # your user/org, token label and last 4
   rogue-arena-inspect evals    # the evals this token can run
   ```

6. **Run** the built-in example eval against a model of your choice (replace the
   id with one from `evals`):

   ```bash
   rogue-arena-inspect launch --eval <environmentId> --model <provider>/<model>
   ```

   Pick a model by its Inspect provider id; see Inspect's
   [model providers](https://inspect.aisi.org.uk/providers.html) for the ids and
   the keys each needs. For a dry run with no model spend, use Inspect's
   built-in offline mock, `mockllm/model`.

   The agent's shell tool runs long commands to completion (a background exec
   the sandbox polls), and the model can ask for a longer timeout per command,
   so a slow scan, build or download is not cut off.

7. **See results.** The run prints a log location; open it with Inspect's
   viewer:

   ```bash
   inspect view
   ```

   Your runs also appear in the web app under **Team Hub → Model Testing →
   Runs** (`/team-hub/model-testing/runs`), with the per-hop score. From the
   CLI, `rogue-arena-inspect runs` lists your runs and
   `rogue-arena-inspect run <id>` shows one run's status, score and per-stage
   result.

8. **Revoke** a token any time from the same **Tokens** page: click **Revoke**
   on its row. If you lose a token, revoke it and create a new one.

## Troubleshooting

- **`401` / "set a current token".** The token is wrong, expired or revoked.
  Create a fresh one on the Tokens page and re-export `ROGUE_ARENA_TOKEN`.
- **`evals` lists nothing.** Your team has no model-eval access or no eval
  shared with it yet. Access is not self-service: email <info@roguelabs.io> to
  get access to our library of ready-built cyber-range eval scenarios and
  model-eval runs, and to have an eval shared with your team.
- **A provider error (`401`/`403`, a model/region policy block).** The model
  runs through *your* provider credentials, so this comes from the provider,
  not from Rogue Arena. Check the provider API key and that the model is
  permitted for your account and region. `launch` prints a clean line pointing
  here instead of a raw provider stack trace.
- **A trial won't start (`TrialReadyTimeout`, or a claim refused with `409`).**
  No warm copy and no free deployment/concurrency slot right now. `evals` shows
  each eval's launch readiness (`ready now (N warm)` vs `on demand, ~X min`);
  wait for a build, lower `concurrency`, or retry when a slot frees. A `402`
  funding denial means the eval could not be funded for your organization.

## What the client sends to Rogue Arena

The model's conversation (prompts, reasoning, replies) stays on your machine;
what it does on the range (commands and their output, file transfers, team chat
posts) passes through Rogue Arena. In detail:

- **Run and task identity, on each claim.** The eval you chose (its environment
  id, and the eval's name), your Inspect task name, a run id the client
  generates, and Inspect's sample id; plus Inspect's epoch, sample uuid, eval id
  and eval-set id when Inspect sets them.
- **Commands and their output.** Every shell command the agent runs on the
  range goes to Rogue Arena to run on the VM, and its stdout/stderr come back
  through Rogue Arena (which returns at most the last 10 MiB of a command's
  output). Files you upload or download pass through Rogue Arena as their raw
  bytes.
- **Team chat and activity (team runs only).** Chat messages members post, and
  the activity feed. The feed stores only a summary of each command (the
  command, its exit code and about 400 characters of output); the command's
  full output still passes through Rogue Arena when it runs.
- **Not sent.** Your model-provider key never goes to Rogue Arena, and neither
  does the model's conversation. Your Rogue Arena token is sent only as the
  bearer credential.

---

The rest of this document is reference material.

## Authenticate (detail)

```bash
export ROGUE_ARENA_BASE_URL=https://arena.roguelabs.io
export ROGUE_ARENA_TOKEN=ra_eval_…
```

`ROGUE_ARENA_TOKEN` is your personal model-eval token (`ra_eval_…`), sent as the
bearer on every call. Keep it in the environment: it is not a sandbox config
field, so it is never serialized into an Inspect log, and the CLI never prints
it (it is redacted from error messages).

Supported Inspect versions: `>=0.3.260,<0.3.274` (pip resolves one
automatically). 0.3.260 is the first with the sandbox errors the provider
raises.

## Check the token and pick an eval

```bash
rogue-arena-inspect whoami   # userMainId, organizationId, tokenLabel, tokenLast4
rogue-arena-inspect evals    # the evals this token can run, with launch visibility
```

`evals` prints one block per eval with its name, version, stage (hop) count,
scoring mode, and **launch visibility**:

```
example-ad-chain  |  v3 · 4 stages · sequential · ready now (2 warm)
    id: 11111111-1111-4111-8111-111111111111
example-web-app  |  v1 · 2 stages · independent · on demand, ~7 min
    id: 22222222-2222-4222-8222-222222222222
```

`ready now (N warm)` means warm copies are claimable now; `on demand, ~X min`
means a claim builds a fresh copy; `unknown` means readiness was not reported
(the command still succeeds). `evals --json` prints the raw listing; `whoami`
prints JSON. `--base-url` and `--token` override the environment, but prefer the
environment variable for the token (a flag is visible in the process list).

## Runs

```bash
rogue-arena-inspect runs             # this token's recent runs (jobs)
rogue-arena-inspect run <id>         # one run's status, score and per-stage result
rogue-arena-inspect run <id> --watch # live-tail the run's events until it settles
rogue-arena-inspect run <id> --chat  # print the run's team chatroom transcript
```

`run <id>` takes a run (job) id (or a trial id as a fallback) and prints the
run's status and, per trial, the server score and each stage (`#<step> <name>:
pass/fail`). Each trial's terminal reason reads as the Rogue Arena enum plus a
short gloss, e.g. `reason=timeout (ran out of wall clock)`.

- `--watch` polls the run and its trial events on an interval (`--interval`,
  default 3 s), printing new exec/hop/lifecycle events as they arrive until the
  job settles, then prints the final render. A team run is one trial, so it
  follows that trial like any other.
- `--chat` prints the run's team chatroom, one line per message.

`runs`, `run`, and `run --chat` take `--json` for the raw objects.

## Launch

```bash
rogue-arena-inspect launch --eval <id> --model <provider>/<model>  # built-in example task
rogue-arena-inspect launch --eval <id> --task my_task.py --model … # your own task
rogue-arena-inspect launch --team team.yaml --eval <id>            # a multi-agent team
rogue-arena-inspect launch --eval <id> --model … --max-runtime 1800 # per-trial wall clock (s)
```

`launch` is a thin wrapper around `inspect eval`. It resolves the built-in
example task (shipped inside the package), sets `ROGUE_ARENA_EVAL`, keeps the
token in the environment (never on the child argv), and runs `inspect eval`.
Anything after `--` is passed straight through to `inspect eval` (e.g.
`-- --limit 1`). `--max-runtime SECONDS` sets the per-trial wall clock (60 to
86400).

### Provider errors

The model runs on your machine through your own provider credentials, so a
provider refusal (a model/region policy block, a bad or missing API key) comes
from the provider, not from Rogue Arena. When `launch`'s `inspect eval` child
exits on one of these, `launch` prints a clean, actionable line instead of a raw
provider stack trace; the child's own output is still shown. Check your provider
API key, that the model is allowed for your account and region, and that the
matching provider extra is installed. Inspect's
[model providers](https://inspect.aisi.org.uk/providers.html) page lists each
provider's model ids and keys.

## Write your own task

The example task files ship inside the package, under
`inspect_rogue_arena/examples/` (`eval_example.py`, `team_example.py`,
`team.yaml`, and the `eval-example.eval.yaml` run config).
`rogue-arena-inspect launch` runs them without a checkout. To customize one,
find the folder, copy the file out, and pass it with `--task`:

```bash
python -c "import importlib.resources as r; print(r.files('inspect_rogue_arena') / 'examples')"
```

Or write your own task. Use the scorer and give the sandbox a config:

```python
from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.util import SandboxEnvironmentSpec

from inspect_rogue_arena import RogueArenaSandboxConfig, rogue_arena_hops, rogue_bash


@task
def my_eval() -> Task:
    return Task(
        dataset=[Sample(input="…your objective…", target="scored by Rogue Arena")],
        solver=react(tools=[rogue_bash()]),
        scorer=rogue_arena_hops(),
        sandbox=SandboxEnvironmentSpec(
            "rogue-arena", RogueArenaSandboxConfig(eval="<environmentId>")
        ),
    )
```

`rogue_bash()` is the shell tool the examples use: it states its default timeout
to the model and runs long commands as a background exec, so a slow command runs
to completion and a transient network failure on a poll is retried rather than
ending the run.

## Teams (multi-agent)

A **team** run puts several agents on **one range**. Each member has its own
model, system prompt and shell channel into the one VM; they share a chatroom
(`post_message` / `read_messages`), a scoreboard (`post_finding` /
`read_scoreboard`) and, in `shared_sight`, an activity feed. A team is one
Inspect sample, so it claims exactly one trial and drives one VM.

Describe the team in a YAML/JSON file (the packaged `team.yaml` is a template;
copy it out to edit) and launch it:

```bash
rogue-arena-inspect launch --team team.yaml --eval <environmentId>
```

The team file sets `turn_taking` (`coordinator`, where a lead agent delegates
via Inspect's `handoff`, or `round_robin`), `visibility`, optional
`tool_timeout_seconds`, `rounds` / `turns_per_member`, and the `members` (each
with `model`, `name`, `role`, `prompt`).

**Command timeouts.** A member's `bash` runs every command in background mode:
it runs to completion and returns its full output, for at most
`tool_timeout_seconds` (a top-level team field, default 600), and the tool
description tells the model so. The model may pass its own `timeout` (seconds)
per command. A command that outlives its timeout is stopped, with everything it
started, and ends with Inspect's timeout error.

### Visibility: `shared_sight` vs `communicate_only`

`visibility` decides whether members can read each other's actions. In **both**
modes the activity feed is recorded and sent to Rogue Arena (the run owner can
see it on the run page). The feed stores only a summary of each command (the
command, its exit code and about 400 characters of output); the command's full
output still passes through Rogue Arena when it runs.

- `shared_sight` (the default) gives each member a `read_activity` tool to read
  the feed, so teammates can avoid colliding.
- `communicate_only` withholds `read_activity`, and Rogue Arena also refuses a
  member's read of the feed during the run, so members coordinate through the
  chatroom alone. The feed is still recorded for the run owner.

Either way, members share the one VM, chatroom and scoreboard. If Rogue Arena
does not serve the activity routes, a `shared_sight` team prints one line and
runs chat-only rather than letting a `read_activity` call end the run.

**Team scoring.** A team run is one trial; its score is the team's. Rogue Arena
scores it when the trial closes, like any other trial, and a key-file hop counts
a key any member surfaced.

## Sandbox config

| Key | Environment variable | Default | Meaning |
|---|---|---|---|
| `eval` | `ROGUE_ARENA_EVAL` | none (required) | The eval to run: an `environmentId`, or a `name` that matches exactly one eval. An id match wins; an unknown or ambiguous selector is an error. |
| `max_trials` | `ROGUE_ARENA_MAX_TRIALS` | `10000` | The job's `n`: the most trials the job admits. Set it to at least samples × epochs plus retries. |
| `concurrency` | `ROGUE_ARENA_CONCURRENCY` | `2` | The job's `concurrency`: the most trials open at once. |
| `max_runtime_seconds` | `ROGUE_ARENA_MAX_RUNTIME_SECONDS` | `3600` (60 to 86400) | Each trial's wall clock, counted from its claim, provisioning included. |
| `max_wait_seconds` | `ROGUE_ARENA_MAX_WAIT_SECONDS` | `900` | How long a claimed trial may take to become `ready`. |
| `heartbeat_interval_seconds` | `ROGUE_ARENA_HEARTBEAT_INTERVAL_SECONDS` | `300` | Lease heartbeat cadence (must be under 900). |
| `sync_exec_ceiling_seconds` | `ROGUE_ARENA_SYNC_EXEC_CEILING_SECONDS` | `50` (at most 55) | The longest command timeout that runs as one synchronous request. |
| `poll_interval_seconds` | `ROGUE_ARENA_POLL_INTERVAL_SECONDS` | `2` (at most 60) | Cadence of the ready, close and background-exec polls. |
| `team_visibility` | (from the team file) | `shared_sight` | A team run's activity-feed visibility: `shared_sight` or `communicate_only` (see Teams). |

Explicit Inspect config wins over environment variables, which win over the
defaults. The config is typed, validated and frozen, and rejects unknown keys.

## Lifecycle

There is **one job per sandbox config per eval run**, and **one trial per
sample**:

1. **`task_init`** resolves `eval` and creates the job with `n`, `concurrency`,
   the wall clock and an idempotency key (a transport failure retries once with
   the same key, so it never creates a second job). Tasks in the same run that
   share a sandbox config share that job.
2. **`sample_init`** waits for a free concurrency slot, then claims a trial with
   its own idempotency key: a transport failure or a 502/503/504 resends the
   claim once and Rogue Arena answers the trial it already opened, never a
   second one. A claim can return a trial still `provisioning`; the provider
   polls it to `ready` within `max_wait_seconds`, and cancels it if anything
   fails.
3. **Heartbeat.** A background task heartbeats the trial every
   `heartbeat_interval_seconds` until the wall clock ends; every exec also
   heartbeats first. Only a 409 (the trial is no longer live) stops the loop.
4. **Tool calls** exec, upload and download on the trial, each bounded by the
   trial's remaining wall clock.
5. **Scoring.** `rogue_arena_hops()` closes the trial; Rogue Arena scores every
   hop during the close and returns the scored trial.
6. **`sample_cleanup`** closes a trial that is still open (`cancelled` when the
   sample was interrupted). A close is sent at most once; if it times out, the
   provider reads the trial back until it is `closed` (Rogue Arena finishes the
   close server-side).
7. **`task_cleanup`** closes any leftover trials as `cancelled`, then cancels
   the job.

### Claim errors

Each of these fails the sample and releases its concurrency slot:

| Error | When |
|---|---|
| `EvalClaimDenied` with HTTP 402 (`is_funding` is true) | Funding denial, with the decision (`verdict`, `reason_code`, `error_message`, raw `decision`). |
| `EvalClaimDenied` with HTTP 409 | A deploy denial, with the same decision fields. |
| `RogueArenaApiError` with HTTP 409 | A state refusal: the job is not open, out of trials, already at `concurrency`, or Rogue Arena is not accepting new trials for your organization right now. |
| `TrialReadyTimeout` | The trial was not `ready` within `max_wait_seconds`. The provider cancels it. |
| `TrialClosedBeforeReady` | Rogue Arena closed the trial while it was provisioning. |
| `httpx.TransportError`, or `RogueArenaApiError` with HTTP 502/503/504 | The claim and its one resend both went unanswered. The provider cancels any trial that was opened, then raises. |

### Concurrency

A semaphore per sandbox config, sized to `concurrency`, holds each claim until
an earlier trial closes, so the provider never sends a claim that would be
refused (409). Waiting for a slot is ordinary queueing with no time limit.

The provider also implements Inspect's `default_concurrency()` hook, which sets
Inspect's default `--max-sandboxes` to `ROGUE_ARENA_CONCURRENCY` (default 2).
When a task config sets `concurrency`, pass the same value as `--max-sandboxes`
so Inspect's queue matches the job's limit.

## Commands

`exec` is **synchronous by default**. A command whose timeout is at most
`sync_exec_ceiling_seconds` (default 50, at most 55) runs as one request; a
command with no timeout gets the ceiling as its timeout.

A longer timeout runs in **background mode**: the provider launches the command,
then polls it (first after 0.25 s, the wait doubling up to
`poll_interval_seconds`) until it finishes, for up to the command's timeout plus
5 minutes (the server ceiling is 4 hours). Rogue Arena stops a background
command and everything it started at its timeout and when its trial ends.

| Result | What the provider does |
|---|---|
| The command exits | `ExecResult`, with `success` when the exit code is 0. |
| The command hits its own timeout | `TimeoutError` with the partial output (Inspect's convention). |
| The trial is not live (409), is closed, or is past `max_runtime_seconds` | `SandboxUnavailableError`: a tool error, and the sample continues. |
| A background exec was lost (no exit status recorded) | `SandboxUnavailableError`. |
| A gateway timeout or transport failure during a sync exec or a background launch | `ExecOutcomeUnknown`: the command may still be running, so it is not retried, and the sample errors. |

A background poll that hits a transient failure retries with exponential backoff
(capped at 30 s) until the deadline. Rogue Arena returns at most the last 10 MiB of a
command's output. `cwd`, `env` and `input` (stdin) pass through; `user`,
`timeout_retry` and `concurrency` are ignored.

## Files

`write_file` and `read_file` move raw bytes, base64-encoded in JSON. Text is
UTF-8. Absolute agent paths only, so a relative path resolves against `/`. A
`read_file` larger than Inspect's read limit raises `OutputLimitExceededError`.
An upload is capped at 10 MiB.

## Scoring: `rogue_arena_hops()`

The scorer closes the trial and reads the **server score** the close returns. It
never computes a hop locally, never runs a command, and never submits a key.

Key-file hops are proven server-side: Rogue Arena counts a key-file hop only
when the planted key appears in the agent's exec output (including background
results) or in a file the agent downloads. The agent proves a hop by reading the
key through its tools; there is nothing to submit.

- **Value:** `hops_passed / hops_total`, as counted under the eval's scoring
  mode (`sequential` or `independent`). The answer reads
  `"<hops_passed>/<hops_total> hops"`.
- **Metadata:** `trial_id`, `hops_passed`, `hops_total`, `reached_step`,
  `scoring_mode`, `outcome`, `terminal_reason`, `timed_out`, and `per_hop`.
- **`per_hop.passed` is the raw check result**, not whether the hop counted. In
  `sequential` mode a hop after the first failed step does not count even when
  its check passed; use `hops_passed` and `reached_step` for the count.

| Trial as closed | Result |
|---|---|
| A server score with terminal reason `agent_success` or `agent_fail` | Scored. |
| A server score with terminal reason `timeout` | Scored normally, with `timed_out: true`. |
| Terminal reason `infra_error` (even with a score), or a `cancelled` outcome | Raises `TrialNotScored`: the sample errors and never enters the mean. |
| No server score (reaped/expired/unscored) | Raises `TrialNotScored`. |

With Inspect's default `fail_on_error`, one such infrastructure outcome aborts
the whole run. To tolerate them, pass `--no-fail-on-error`, or a threshold such
as `--fail-on-error 0.1`. From Python, pass `fail_on_error=False` to
`inspect_ai.eval()`. To retry, `--retry-on-error N` claims a fresh trial for
each retry, so leave headroom in `max_trials`.

## Python client

`RogueArenaClient.from_env()` reads `ROGUE_ARENA_BASE_URL` and
`ROGUE_ARENA_TOKEN`. It covers the whole eval API: `whoami`, `list_evals`,
`list_runs`, `create_job`, `get_job`, `cancel_job`, `list_job_trials`,
`claim_trial`, `get_trial`, `heartbeat_trial`, `run_command` (and the
single-request `exec_trial`), `upload_file`, `download_file`, `close_trial` and
`list_trial_events`.

```python
import asyncio

from inspect_rogue_arena import RogueArenaClient


async def show_evals() -> None:
    client = RogueArenaClient.from_env()
    for entry in await client.list_evals():
        print(
            entry.environment_id, entry.name, entry.hop_count, entry.scoring_mode.value
        )


asyncio.run(show_evals())
```

## Development

```bash
pip install -e '.[all-providers]'
python -m unittest discover -s tests
```

The tests run offline: HTTP goes to a mock transport and an in-memory fake
server, and the Inspect runs use `mockllm`, including a run of the shipped
example task.

## License

Apache License 2.0. See
[LICENSE](https://github.com/rogue-labs-cyber/inspect-rogue-arena/blob/main/LICENSE)
and
[NOTICE](https://github.com/rogue-labs-cyber/inspect-rogue-arena/blob/main/NOTICE).
The license grants no rights to the "Rogue Arena" or "Rogue Labs" names or
marks.

## Changelog

### 0.4.0

- First public release.
