Metadata-Version: 2.4
Name: mazebench
Version: 0.2.0
Summary: MazeBench: ice-maze puzzle site, world editor, and coding-agent benchmark runner. `mazebench launch` serves the website locally.
Project-URL: Homepage, https://mazebench.com
Project-URL: Source, https://github.com/mazebench/MazeBench
License-Expression: MIT
License-File: LICENSE
Keywords: agents,benchmark,game,maze,puzzle
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Games/Entertainment :: Puzzle Games
Classifier: Topic :: Software Development :: Testing
Requires-Python: <3.14,>=3.11
Provides-Extra: prime
Requires-Dist: verifiers>=0.1.14; extra == 'prime'
Description-Content-Type: text/markdown

# MazeBench

The MazeBench env can run in the browser, in the terminal, driven by a local
coding agent (Codex CLI or Claude Code), and through Prime Intellect Verifiers
v1. The Verifiers integration includes both normal multi-turn chat-model runs
and Codex CLI runs.

## Install from PyPI and launch

```bash
pip install mazebench     # or: uv tool install mazebench
mazebench launch          # serves the site and opens your browser
```

The wheel bundles the whole Node site/runtime; `mazebench launch` materializes
a writable workspace at `~/.mazebench/site` (drafts, run outputs, and account
state live there) and serves it. Node.js must be installed; `ffmpeg` + a
Chromium-family browser are needed for replay videos, and the `codex` /
`claude` CLIs for agent runs. Options: `mazebench launch port=3000
host=127.0.0.1 open=false`. From a repo checkout, `mazebench launch` serves the
checkout instead (same as `npm run dev`).

Releases are published by `.github/workflows/publish.yml` (PyPI trusted
publishing) whenever a GitHub Release is created; see
[docs/packaging.md](docs/packaging.md).

## The local site: Play, Build, and Agent modes

`npm run dev` (or `mazebench launch`) serves the full site at
`http://localhost:3000` with three modes on the home page (the frontend design
matches mazebench.com — the chrome is ported from the MazeJam repo):

- **Play Mode** (`/play`) — play the master world (the 256-room world agents
  are benchmarked on), any local draft world, or downloaded online worlds.
- **Build Mode** (`/build`) — make and save your own worlds **locally, without
  publishing anywhere**. Each draft world is a real game directory under
  `games/draft-*` (gitignored), so the level editor (`/author/<world>/<level>`),
  world-map editor, play page, flyover, and the agent runner all work on it.
  You can also edit the master world itself (its levels live in
  `games/maze/`), copy the master world into a draft, and import/export worlds
  as JSON in the same `mazebench-build-world-v1` format the hosted site uses.
- **Agent Mode** (`/agent`) — launch Codex CLI, Claude Code (your
  subscriptions, via the same runner as `mazebench model=...`), or Prime
  Verifiers runs **from the browser**, against the master world, any local
  draft, or a downloaded community world. Runs stream live: per-move actions,
  gems, the ASCII board, the agent's reasoning, the runner log, and the replay
  video when the run finishes. Run artifacts land under
  `outputs/maze-local/site/<run-id>/`.

### Sync drafts with mazebench.com

Build Mode can connect to your account on the hosted site
(`https://dev.mazebench.com`, soon `mazebench.com`): the local server holds a
session token in `data/remote.json` and talks to the site server-to-server —
push local drafts up, pull site drafts down, and download published community
worlds for agent runs. Connect either via the browser link flow
(`/link-local` on the hosted site) or by pasting your `mazebench_session`
cookie value manually. Drafts stay drafts — publishing remains a deliberate
action on the site.

## The `mazebench` command

There is a small `mazebench` CLI that wraps every workflow. It is a launcher:
it shells out to the repo's Node scripts (game engine, local-agent runner,
replay/video) and, for the Prime path, to the `prime`/`uv` CLIs. Node.js is a
prerequisite for all runs.

Install it from a checkout (recommended — the CLI needs the repo's Node
scripts at runtime):

```bash
# with uv (this repo already uses a uv-managed .venv)
uv pip install -e .

# or with plain pip
pip install -e .

# add the Prime Intellect Verifiers integration
uv pip install -e ".[prime]"
```

Local agent runs execute **inside a container by default** so the agent is
isolated from your filesystem. Build the image once (needs Docker):

```bash
mazebench build          # or: npm run maze:build-image  /  docker build -t mazebench-agent .
```

Then, without Prime, drive the maze with your own local agent and get a replay
video out the other end (credentials are read from `OPENAI_API_KEY` /
`ANTHROPIC_API_KEY` in your environment):

```bash
mazebench model=codex moves=10
mazebench model=claude moves=10 level=HxI
```

See [Run with a local coding agent](#run-with-a-local-coding-agent-no-prime)
for details, and [docs/packaging.md](docs/packaging.md) for publishing to PyPI.

## Run with a local coding agent (no Prime)

This path uses **your** Codex or Claude Code auth. There is no Prime intercept
server and no reward scoring — the agent just plays the game by shelling out to
`scripts/codex-play.js` (a stateful CLI over `scripts/maze-bridge.js`), and the
runner then renders a replay video from the session.

Prefer not to remember the flags? Run the **interactive setup** and pick each
option with the arrow keys:

```bash
mazebench wizard            # or: npm run maze:wizard
```

The wizard's "Custom…" model choice lists your actual Codex models (read from
`~/.codex/models_cache.json`), then — for Codex — lets you pick the reasoning
effort (`low`/`medium`/`high`/`xhigh`) and toggle Fast mode. You can also set
these directly:

```bash
mazebench model=codex model_name=gpt-5.5 reasoning=xhigh codex_fast=true moves=10
mazebench model=claude model_name=opus moves=10
```

`model_name` is forwarded to `codex -m` / `claude --model`; `reasoning` maps to
`codex -c model_reasoning_effort` and `codex_fast=true` to the `priority`
service tier.

One-liners:

```bash
# via the mazebench CLI
mazebench model=codex moves=10
mazebench model=claude moves=10

# equivalently, via npm
npm run maze:codex-local -- moves=10
npm run maze:claude-local -- moves=10
```

Common options (all accept `key=value` or `--key value`):

| key | meaning | default |
| --- | --- | --- |
| `model` | `codex` or `claude` | required |
| `container` | `true` (run in Docker, host FS isolated) or `false` (host) | `true` |
| `tools` | `false` (sandboxed to the maze) or `true` (full access) | `false` |
| `mode` | `text` (ASCII board) or `vision` (rendered PNGs) | `text` |
| `moves` | maze action budget shown to the agent | `20` |
| `game` | game dir under `games/`, e.g. a Build Mode draft world id | `maze` |
| `level` | world level id, e.g. `HxI` or `level_HxI` | `level_HxI` |
| `view` | `top` … `side` camera pitch | `top-diagonal` |
| `yaw` | `0`–`3` camera yaw | `0` |
| `gems` | unique gems required for `game_won` | `100` |
| `vision_width`, `vision_height` | PNG size in vision mode | `512` |
| `model_name` | underlying LLM id (`codex -m` / `claude --model`) | agent default |
| `reasoning` | Codex reasoning effort: `low` / `medium` / `high` / `xhigh` | model default |
| `codex_fast` | Codex Fast mode (priority tier, ~1.5× speed) | `false` |
| `video` | `on` / `off` | `on` |
| `fast`, `draft` | faster/cheaper video capture | off |
| `dry_run` | print the agent command + prompt and exit | off |

Use `dry_run=on` to preview exactly what will run (the container/agent command
and the full play prompt) without spending any tokens or needing Docker.

### Isolation: runs happen in a container by default

Every local agent run executes inside a container (`container=true`, the
default), so the agent **cannot touch your filesystem** — only an output
directory is mounted, and API credentials are passed via environment variables.
This is the strong, OS-level guarantee; it does not depend on the agent CLI
behaving.

- One-time build: `mazebench build` (or `npm run maze:build-image`). The image
  bundles Node, the maze runtime, a headless Chromium + ffmpeg (for vision and
  video), and the `codex` / `claude` CLIs. See the [Dockerfile](Dockerfile).
- Credentials (forwarded into the container as env, or mounted):
  - **Codex:** `model=codex` **auto-mounts your Codex subscription login**
    (`~/.codex/auth.json`, read-only) when it exists — no setup needed.
    Otherwise set `OPENAI_API_KEY`, or pass `codex_auth=<path>`.
  - **Claude:** on macOS, `model=claude` **auto-detects your Claude Code
    subscription login from the Keychain** and mounts just that credential (a
    short-lived, read-only temp file, deleted after the run) — no setup needed.
    Otherwise set `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`, or pass
    `claude_auth=<path>` to a `.credentials.json`.
  - Only the single credential is exposed — never your whole `~/.codex` or
    `~/.claude` (which hold history, memories and logs).
- What's mounted: only `outputs/maze-local` (writable, for artifacts). Nothing
  else on your disk is visible to the container. The network stays on because
  the agent needs to reach the model API.
- Escape hatch: `container=false` runs on the host instead, protected only by
  the per-CLI `tools` sandbox below (weaker — see the Codex read caveat).
- `docker_bin=podman` and `image=<tag>` override the runtime and image tag.

### Sandboxing (`tools`)

Inside the container (or on the host with `container=false`), `tools` is a
second layer. With `tools=false` (default) the agent is meant to only drive the
maze helper; how strictly that is enforced depends on the agent (below).
`tools=true` removes the guardrails (write files, run any command, use the
network) — inside the container that is still safe for your host.

- **Codex** — On the **host** (`container=false`, `tools=false`), Codex runs under
  `codex exec --sandbox workspace-write`: writes confined to the run folder,
  network disabled by default. Caveat: Codex's OS sandbox does **not restrict
  reads**, so a determined agent could still read files it knows the path to.
  **Inside the container**, Codex's sandbox (bubblewrap) can't create user
  namespaces under Docker, so Codex runs with
  `--dangerously-bypass-approvals-and-sandbox` — the container itself is the
  sandbox. So for Codex-in-container, `tools=false` is a prompt-level instruction
  ("only play the maze"), not an OS guarantee; host isolation comes from the
  container. `tools=true` always uses `--dangerously-bypass-approvals-and-sandbox`.
- **Claude Code** (`tools=false`): runs under `--permission-mode dontAsk` with an
  allowlist of **only** the maze helper command (`Bash(node <helper> *)`), so
  every other tool — Read, Write, WebFetch, other Bash — is auto-denied. Claude
  also blocks command chaining per-subcommand, so the allowlist can't be widened
  with `; other-cmd`. This is a true "maze-only" lockdown at the agent level.
  `tools=true` uses `--permission-mode bypassPermissions`.

Notes:

- In vision mode with `tools=false`, Claude is additionally allowed to read the
  rendered frames (`Read(<run>/frames/**)`) so it can see the maze. Under Codex's
  sandbox the headless-browser render may be blocked; if a frame fails, the
  helper falls back to the ASCII board for that turn (`frame_error`).
- These are the best guardrails each CLI offers; neither replaces running
  untrusted agents in a real VM/container if that is your threat model.

### Text vs vision observations

By default the agent plays from the ASCII board (`mode=text`). In `mode=vision`
the helper renders a perspective PNG of the current room each turn (via the same
`scripts/maze-render-frame.js` renderer the Verifiers vision taskset uses),
drops the ASCII board, and prints a `frame_image` path in the JSON. The agent —
Codex CLI and Claude Code are both multimodal — opens that PNG to decide its
move. Frames are saved under `<run>/frames/`.

```bash
mazebench model=codex moves=10 mode=vision
mazebench model=claude moves=10 mode=vision vision_width=768 vision_height=768
npm run maze:codex-local -- moves=10 mode=vision
```

Vision mode boots a headless browser to render each frame, so it is noticeably
slower than text mode and needs a Chromium-family browser (the same dependency
as the replay video). If a frame fails to render, the helper falls back to the
ASCII board for that turn and reports `frame_error`.

Each run writes a timestamped directory under `outputs/maze-local/<model>/`:

- `session.json` — full state and per-action replay
- `actions.jsonl` — per-turn action log
- `scorecard.json`, `maze_scorecard.json`, `maze_actions.txt`
- `maze_replay.mp4` — the replay video (unless `video=off`)
- `reasoning.json`, `agent.log`, `agent-events.jsonl` — the agent's reasoning
  (see below)

### Reasoning logs

For **both** `model=codex` and `model=claude`, the runner captures the agent's
structured event stream (`codex exec --json` / `claude -p --output-format
stream-json`) and distills a per-move reasoning log. `reasoning.json` is exactly
what you'd want to skim:

```json
[
  { "move": 1, "action": "right", "reasoning": "The exit is east, so...",
    "moved": true, "gems": 0, "room": "level_HxI" }
]
```

The `reasoning` for each move is the agent's commentary (Codex `agent_message` /
Claude `text` blocks, plus Claude `thinking` blocks if extended thinking is on)
since its previous action. `agent.log` is the same trace in human-readable form,
`agent-events.jsonl` is the raw event stream, and the runner prints a per-move
summary to the terminal at the end of the run.

Prerequisites: Node.js, plus `ffmpeg` and a Chromium-family browser for the
video (already used by `npm run maze:replay`). The agent binary must be on your
PATH — `codex` for `model=codex`, `claude` for `model=claude` — or pass
`codex_bin=`/`claude_bin=` with an explicit path.

To (re)build a video from any finished run or Prime eval directory:

```bash
mazebench replay outputs/maze-local/codex/<run-dir>/
# or
npm run maze:replay -- outputs/maze-local/codex/<run-dir>/
```

## 1. Set up the web app with Node

Install dependencies, run tests, then start the local server:

```bash
npm install
npm test
npm run dev
```

Open the app at:

```text
http://localhost:3000
```

## 2. Run the game in the terminal, see the scorecard, and save video

Start an interactive terminal game:

```bash
npm run maze:terminal
```

Controls:

- Arrow keys: move
- `i` / `k`: rotate camera up / down
- `j` / `l`: rotate camera left / right
- `z` or `u`: undo
- `r`: reset
- `q`: quit and print the scorecard

Interactive runs write replay files under `outputs/maze-terminal/<timestamp>/`.
When the run ends, answer the video prompt to save `maze_replay.mp4`.

For a non-interactive run that saves the scorecard and video:

```bash
npm run maze:terminal -- --moves UDLR --once --record-replay --video --fast --draft --fps 20 --width 400 --height 400
```

The output folder contains:

- `maze_scorecard.json`
- `maze_actions.txt`
- `maze_replay.json`
- `results.jsonl`
- `maze_replay.mp4` when video is enabled

## 3. Run with Prime Intellect Verifiers and see results

The `mazebench prime` subcommands wrap the steps below (they print the exact
underlying command before running it):

```bash
mazebench prime install                                   # prime env install mazebench
mazebench prime eval   model=openai/gpt-5-nano n=1 r=1    # normal chat-model eval
mazebench prime codex  model=openai/gpt-5-codex           # Codex CLI via Verifiers (section 4)
mazebench prime vision model=openai/gpt-4.1-mini          # vision observations (section 5)
```

Install `uv` if you do not already have it:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

On macOS, Homebrew also works:

```bash
brew install uv
```

Install the local environment from `./environments/mazebench`:

```bash
prime env install mazebench
```

Run a small eval:

```bash
prime eval run mazebench -m openai/gpt-5-nano -n 1 -r 1 -s --max-turns 8 -d
```

Use your own configured model after `-m`, or omit `-m` to use your Prime
default. The terminal prints the run summary. MazeBench v1 stores replay
artifacts under `info.maze_actions`, `info.maze_scorecard`, and
`info.maze_replay` in `results.jsonl`.

View saved evals:

```bash
prime eval view
```

Export scorecard files and a replay video from a saved eval directory:

```bash
npm run maze:replay -- environments/mazebench/outputs/evals/<model>/<run-id>
```

## 4. Run Codex through Verifiers v1

MazeBench includes a `mazebench_codex` v1 plugin. It uses your local `codex`
CLI as the harness, routes Codex model calls through the Verifiers v1
interception server, and writes replay artifacts under `outputs/maze-codex-v1/`.

```bash
cd environments/mazebench
uv run eval mazebench_codex \
  -m openai/gpt-5-codex \
  -n 1 -r 1 \
  --taskset.max-actions 100 \
  --max-turns 40 \
  --rich false
```

`--taskset.max-actions` is the maze action budget. `--max-turns` is still useful
because Verifiers counts Codex's internal model/tool-call turns while the CLI is
working. The harness strips the `openai/` prefix before invoking `codex exec` so
Codex can use its local model metadata, while Verifiers still routes/bills the
Prime model.

Export the saved v1 trace to scorecard files and video the same way as normal
MazeBench evals:

```bash
npm run maze:replay -- environments/mazebench/outputs/evals/<model>/<run-id>
```

## 5. Run vision observations

The normal `mazebench` taskset can send perspective PNG observations instead of
ASCII boards:

```bash
cd environments/mazebench
uv run eval mazebench \
  -m openai/gpt-4.1-mini \
  -n 1 -r 1 \
  --taskset.observation-mode vision \
  --taskset.vision-width 512 \
  --taskset.vision-height 512 \
  --max-turns 8 \
  --rich false
```

Vision mode keeps the same JS game state, allowed commands, terminal conditions,
and reward functions as ASCII mode. It sends a short text status plus a
perspective image of the current room, with no ASCII board.

## License

MIT. See [LICENSE](LICENSE).
