Metadata-Version: 2.4
Name: runboard
Version: 0.3.0
Summary: Self-hosted, W&B-style live training dashboards for any cluster or VM, viewable in your browser without SSH.
Author-email: Mohammad Zbeeb <mhmdzbyb829@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Moe-Zbeeb/runboard
Project-URL: Documentation, https://github.com/Moe-Zbeeb/runboard#readme
Project-URL: Issues, https://github.com/Moe-Zbeeb/runboard/issues
Project-URL: Changelog, https://github.com/Moe-Zbeeb/runboard/blob/main/CHANGELOG.md
Keywords: machine-learning,experiment-tracking,training,dashboard,metrics,slurm,cluster,wandb
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

<h1 align="center">runboard</h1>

<p align="center">
  <b>Your own live experiment dashboard on Cloudflare, from any cluster or VM.</b>
</p>

<p align="center">
  <a href="https://github.com/Moe-Zbeeb/runboard/actions/workflows/ci.yml"><img src="https://github.com/Moe-Zbeeb/runboard/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/runboard/"><img src="https://img.shields.io/pypi/v/runboard" alt="PyPI"></a>
  <a href="https://pypi.org/project/runboard/"><img src="https://img.shields.io/pypi/pyversions/runboard" alt="Python versions"></a>
  <img src="https://img.shields.io/badge/dependencies-0-brightgreen" alt="Zero dependencies">
  <a href="https://github.com/Moe-Zbeeb/runboard/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Moe-Zbeeb/runboard" alt="License"></a>
</p>

<p align="center">
  <a href="https://deploy.workers.cloudflare.com/?url=https://github.com/Moe-Zbeeb/runboard"><img src="https://deploy.workers.cloudflare.com/button" alt="Deploy to Cloudflare"></a>
</p>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Moe-Zbeeb/runboard/main/docs/assets/dashboard-dark.png">
  <img alt="runboard dashboard comparing three training runs" src="https://raw.githubusercontent.com/Moe-Zbeeb/runboard/main/docs/assets/dashboard-light.png">
</picture>

runboard is a personal, W&B-style experiment dashboard. Each researcher deploys one small backend to
their own Cloudflare account. Training jobs send metrics directly over HTTPS, so the cluster does not
need to host a server and the dashboard keeps the same URL when jobs, login nodes, or laptops restart.

- **Two lines in your training code.** `runboard.init(...)` and `runboard.log({...})`.
- **Bring your own Cloudflare.** One click creates a Worker, D1 database, and static dashboard
  in your account. No domain or always-on cluster process is required.
- **One stable URL.** Jobs and browsers connect to the same `workers.dev` HTTPS endpoint from anywhere.
- **Zero dependencies.** The client, server, and dashboard use only the Python standard library.
  `pip install runboard` works on any cluster, even behind a restrictive proxy.
- **Keeps network and disk work out of training.** `log()` appends metrics in memory; a background
  thread sends batches, retries outages, and spills excess backlog to disk. Remaining metrics are
  saved when the run finishes.
- **A local mode when you need it.** The same package can store plain JSONL files and serve them from a
  cluster or laptop without Cloudflare storage.

## Contents

- [Quickstart](#quickstart)
- [Zero-code integration](#zero-code-integration)
- [For coding agents](#for-coding-agents)
- [How it works](#how-it-works)
- [Logging API](#logging-api)
- [Cloudflare deployment](#cloudflare-deployment)
- [Running on a cluster](#running-on-a-cluster)
- [The dashboard](#the-dashboard)
- [Configuration](#configuration)
- [CLI reference](#cli-reference)
- [Security](#security)
- [FAQ](#faq)
- [Development](#development)

## Quickstart

**1. Deploy your personal backend.** Click the button, sign in to Cloudflare, and choose a long random
`RUNBOARD_TOKEN` when prompted. Cloudflare creates the Worker, database, and dashboard.

[![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/Moe-Zbeeb/runboard)

Cloudflare gives you a URL such as `https://runboard.<account>.workers.dev`.

**2. Install and connect the Python client** on each machine or shared cluster home:

```bash
pip install runboard
runboard configure https://runboard.<account>.workers.dev
```

The command asks for the same token, verifies the deployment, and saves both values in
`~/.runboard/server.json` with mode `600`. Run `runboard doctor` at any time to check the connection;
it sends a test run and ends with `PASS`, or `FAIL` and the fix.

**3. Stream your training job.** If it already writes TensorBoard, JSONL, CSV, or Hugging Face logs, or
prints lines like `loss=0.42`, wrap the launch command and change nothing else:

```bash
runboard run --project resnet -- python train.py
```

To log custom metrics from code instead:

```python
import runboard

runboard.init(project="resnet", config={"lr": 3e-4, "batch_size": 256})

for step in range(num_steps):
    loss = train_step()
    runboard.log({"train/loss": loss}, step=step)

    if step % 500 == 0:
        runboard.log({"eval/accuracy": evaluate()}, step=step)

runboard.finish()
```

**4. Open the URL printed by `runboard configure`.** The dashboard refreshes every 5 seconds.

`runboard.init()` reads the saved endpoint automatically. On clusters with a shared home directory,
configure it once and every compute node uses it. For containers or separate machines, set
`RUNBOARD_SERVER` and `RUNBOARD_TOKEN` as environment variables.

## Zero-code integration

Already logging to TensorBoard, a JSONL or CSV file, the Hugging Face Trainer, or just printing
`loss=...` lines? Put `runboard run --` in front of the launch command and leave the training code alone:

```bash
runboard run --project resnet -- python train.py --lr 3e-4
runboard run --project llm -- torchrun --nproc_per_node 8 train.py
```

The wrapper starts the command, finds the metric files it writes in the current directory (add more
with `--watch DIR`), and streams them to your dashboard. It prefers TensorBoard event files, then
`*.jsonl` and `*metrics*.csv`, then Hugging Face `trainer_state.json`, and reads printed metrics only
when none of those appear. Output and exit code pass through unchanged, and the run is marked
`finished`, `crashed`, or `killed` from the exit code.

To stream a job that is already running, or to upload a finished run's logs:

```bash
runboard watch ./outputs          # follow until Ctrl-C
runboard watch ./outputs --once   # upload what exists and exit
```

Running `watch` again on the same path continues the same run without duplicating rows.

## For coding agents

`runboard guide` prints step-by-step integration instructions for coding agents such as Claude Code and
Codex; the same text is in [docs/integrating.md](docs/integrating.md). In short: run `runboard doctor`
(it ends with `PASS` or `FAIL` and a fix), put `runboard run --` in front of the launch command, and
check that the wrapper's last line reports logged rows.

## How it works

```text
  cluster / VM                         your Cloudflare account                 anywhere
 ┌──────────────────┐    HTTPS    ┌───────────────────────────────┐        ┌───────────┐
 │ train.py         │ ──────────► │ Worker: auth + REST API       │ ◄────► │  browser  │
 │   runboard.log() │             │ D1: runs + metric batches     │ HTTPS  │ dashboard │
 │   retry + spool  │             │ Assets: dashboard             │        └───────────┘
 └──────────────────┘             │                               │
                                  └───────────────────────────────┘
```

1. Your script appends metrics to an in-memory buffer. A background thread sends them to the
   server every 10 seconds (every second in local file mode), in chunks of up to 5,000 rows.
2. The Cloudflare Worker authenticates the request and stores run metadata and metric batches in D1.
   Retried batches have deterministic IDs and are stored once.
3. The dashboard is served by the same Worker. It polls every 5 seconds and fetches only batches it
   has not seen.
4. If Cloudflare or the network is unavailable, the client retries with backoff and writes remaining
   data to `~/.runboard/spool/`. `runboard sync` sends it later.

More detail is in [docs/architecture.md](docs/architecture.md).

## Logging API

```python
import runboard

run = runboard.init(
    project="my-project",      # groups runs in the dashboard
    name="baseline-lr3e-4",    # display name (defaults to the run id)
    config={"lr": 3e-4},       # hyperparameters, shown in run details
    tags=["baseline"],         # searchable in the sidebar
    run_id=None,               # pass an existing id to resume a run
    server=None, token=None,   # override server discovery
    dir=None,                  # write to a shared directory instead of HTTP
)

runboard.log({"train/loss": 0.42, "train/lr": 1e-4}, step=100)   # step is optional; auto-increments
runboard.finish()                                                # also runs automatically at exit
```

| Behavior | Details |
|---|---|
| Values | Python numbers, NumPy scalars, and 0-d torch tensors all work. NaN and inf become gaps; non-numeric values are ignored. |
| Sections | Metrics named `group/name` are grouped into dashboard sections (`train/…`, `eval/…`). |
| Status | Runs are marked `running`, `finished`, `crashed` (uncaught exception), or `killed` (Ctrl-C). A run that stops logging for 10 minutes is shown as `stale`. |
| Metadata | Hostname, command line, and `SLURM_JOB_ID` are recorded automatically. |
| Multi-GPU | Under `torchrun` or DDP, only rank 0 logs. `init()` and `log()` are no-ops where `RANK` or `LOCAL_RANK` is non-zero, so you can call them unconditionally. Pass `all_ranks=True` to log from every rank. |
| Context manager | `with runboard.Run(project="p") as run: run.log(...)` marks the run `crashed` if the block raises. |

### When the server is unreachable

Network failures never run inside `log()`. If the server is down, rows are kept in memory and retried
with backoff; excess backlog is saved in the background. Rows still unsent when the job exits are written to
`~/.runboard/spool/`. Upload them later with:

```bash
runboard sync
```

## Cloudflare deployment

The deploy button is the normal path. It uses [Cloudflare's automatic resource provisioning](https://developers.cloudflare.com/workers/platform/deploy-buttons/),
so every user receives isolated resources in their own account. A custom domain is optional; the
generated `workers.dev` address is stable and sufficient.

To deploy from a terminal instead:

```bash
git clone https://github.com/Moe-Zbeeb/runboard && cd runboard
npm install
npx wrangler login
npm run deploy
```

Generate a token, then store it as a Worker secret when Wrangler prompts for its value:

```bash
python -c 'import secrets; print(secrets.token_urlsafe(32))'
npx wrangler secret put RUNBOARD_TOKEN
```

The Worker creates its schema on the first authenticated request. The SQL migration is also kept in
`cloudflare/migrations/` for inspection and future upgrades.

For local Worker development, copy `.dev.vars.example` to `.dev.vars`, replace its value, and run
`npm run dev`. Wrangler keeps local D1 data under `.wrangler/`.

Cloudflare's free plan is enough for personal use and normal research runs, subject to its current
[Workers](https://developers.cloudflare.com/workers/platform/pricing/),
[D1](https://developers.cloudflare.com/d1/platform/pricing/) quotas. Large sweeps or very frequent logging can
exceed those quotas. Runboard batches writes and, in HTTP mode, sends at most one batch every 10 seconds
per run by default (`flush_interval`). The dashboard polls every 5 seconds, pauses in hidden tabs, and only
downloads metrics for runs that changed. Log at a useful interval instead of every inner-loop operation.

## Running on a cluster

### Recommended: send directly to Cloudflare

If the cluster home is shared, run `runboard configure` once on a login node. Otherwise inject the
settings into the job:

```bash
export RUNBOARD_SERVER="https://runboard.<account>.workers.dev"
export RUNBOARD_TOKEN="your-token"
runboard run --project resnet -- python train.py
```

Use `python train.py` directly when the script calls `runboard.init()` itself.

Only outbound HTTPS is required. There is no Runboard daemon on the login node and no tunnel to keep
alive.

### Local server mode

When experiments must remain inside the cluster, run the included local server:

Run it in `tmux` or with `nohup` on the login node:

```bash
nohup runboard serve --tunnel > ~/runboard.log 2>&1 &
grep public: ~/runboard.log
```

If your cluster kills long processes on login nodes, run the server as a low-resource Slurm job
([examples/slurm/serve.sbatch](examples/slurm/serve.sbatch)):

```bash
sbatch examples/slurm/serve.sbatch
grep public: runboard-*.out
```

### Quick-tunnel URL notifications

The quick-tunnel URL changes whenever the server restarts. Use `--notify` to have each new URL pushed
to your phone through [ntfy](https://ntfy.sh). Install the app and subscribe to a private,
hard-to-guess topic:

```bash
runboard serve --tunnel --notify https://ntfy.sh/my-private-topic-8f3k2
```

### Local server discovery

Nothing extra is needed. Jobs discover the server through `~/.runboard/server.json`. See
[examples/slurm/train.sbatch](examples/slurm/train.sbatch) and
[examples/slurm/sweep.sbatch](examples/slurm/sweep.sbatch) (a job-array learning-rate sweep).

### Compute nodes can't reach the login node?

Some clusters block traffic between nodes. If they share a filesystem, skip HTTP and write straight
into the server's run directory:

```python
runboard.init(project="resnet", dir="~/runboard-runs")
```

or set `export RUNBOARD_DIR=~/runboard-runs` in your job script. The server picks the files up.

## The dashboard

- **Runs sidebar.** Search, project filter, and status badges. Click a run's name to see its config and metadata.
- **Compare runs.** Select up to 8 runs. Each run keeps its color while it stays selected.
- **One chart per metric.** Hover for exact values, drag to zoom, double-click to reset. Legends show each run's latest value.
- **Controls.** EMA smoothing, log-scale y, x axis as step, relative time, or wall clock, a regex metric filter, and system/light/dark themes.
- **Long runs.** Charts retain peaks and dips while reducing very large series to a display-sized sample. Original metrics remain in the JSONL files.
- **Summary table.** Latest value of every metric for the selected runs.
- A responsive layout for phones and desktops.

### Binned relationship plots

A run can declare `config.runboard_binned_charts` with entries containing `title`,
`prefix`, `x_label` and optional `y` (default `mean`). Log numeric metrics named
`<prefix>/bin_00/count`, `.../x_mean` and `.../mean` at the actual training step.
The dashboard adds a relationship chart using the latest step's nonempty bins.
It never mixes bins from older steps or applies time-series smoothing to these
plots. A custom `y` such as `p50` can display a median series. Both hosted and local
dashboards use the same format; older dashboards continue to display the scalar
metrics. These charts summarize bins; they are not raw scatter plots or box plots.

Raw bin-series cards are hidden when the metric filter is empty. Enter a filter
such as `bin_` to inspect their counts and individual values. Relationship legends
include each run's source step, so comparisons across runs do not imply identical
training progress. The mean and quantile fields must share a common step with the
bin count and x value; missing values are omitted rather than borrowed from a
previous step. Run the JavaScript contract check with `node tests/js/binned.test.cjs`.

## Configuration

| Environment variable | Default | Purpose |
|---|---|---|
| `RUNBOARD_SERVER` | from `~/.runboard/server.json` | Server URL used by `init()` |
| `RUNBOARD_TOKEN` | from `~/.runboard/server.json` | Access token used by `init()` |
| `RUNBOARD_DIR` | unset | If set, `init()` writes files to this directory instead of using HTTP |
| `RUNBOARD_HOME` | `~/.runboard` | Where the token, server info, and spool files live |

## CLI reference

| Command | Description |
|---|---|
| `runboard configure URL` | Verify and save a personal Cloudflare or other hosted endpoint. Prompts securely for its token; `--token` is available for automation. |
| `runboard run [options] -- CMD` | Run a training command and stream the metrics it logs. Options: `--project`, `--name`, `--run-id`, `--watch DIR`, `--source FILE`, `--stdout auto\|on\|off`, `--config KEY=VALUE`, `--config-json FILE` |
| `runboard watch [PATH…]` | Stream metrics from existing log files or directories. `--once` uploads and exits; `--stdout-file FILE` reads a text log |
| `runboard doctor` | Show where metrics go, send a test run, and print `PASS` or `FAIL` with a fix |
| `runboard guide` | Print the integration guide for coding agents |
| `runboard serve` | Start the server. Options: `--dir` (default `./runboard-runs`), `--port` (default 8080, falls back to a free port if taken), `--host`, `--tunnel`, `--notify URL`, `--advertise URL` (address jobs should use) |
| `runboard ls [--dir D] [--server URL]` | List runs on the configured server, or in a local directory |
| `runboard sync [files…]` | Upload spooled offline metrics |
| `runboard url` | Print the current dashboard URL |
| `runboard --version` | Print the version |

## Security

- Every request, including the dashboard itself, needs the access token. For Cloudflare, each user
  chooses it during deployment and stores it as a Worker secret. The local server generates one.
- Opening `/?token=…` stores the token in an `HttpOnly` cookie and removes it from the address bar.
- The hosted endpoint uses HTTPS. Run metadata and metric batches are stored in D1 inside the user's
  Cloudflare account.
- **Anyone with the URL and the token can read your metrics.** Treat the full URL like a password. To
  rotate a Cloudflare token, run `npx wrangler secret put RUNBOARD_TOKEN` from the repository and
  re-run `runboard configure`. For a local server, delete `~/.runboard/token` and restart it.

## FAQ

**How is this different from W&B or TensorBoard?**
W&B is a hosted service, so your metrics leave your infrastructure and you need an account.
TensorBoard needs port forwarding or a tunnel you set up yourself, and it reads event files.
runboard gives every user their own small Cloudflare backend and keeps the Python client dependency-free.

**Does it work on clusters with no internet access?**
Cloudflare mode needs outbound HTTPS from compute nodes. If that is blocked, use local file mode with
`RUNBOARD_DIR` and `runboard serve`, then access it over the internal network or an SSH tunnel.

**Can I log from any experiment?**
Yes. Log numeric metrics with any names, group them with `/`, and include any JSON-compatible
hyperparameters in the run config. The tracker has no framework dependency, so it can live in a plain
Python loop or a PyTorch, JAX, or other training script. Each backend retains the run's full metric
history.

**Can I analyze the data myself?**
Local mode stores every run as `<dir>/<project>/<run_id>/metrics.jsonl`:

```python
import pandas as pd
df = pd.read_json("runboard-runs/resnet/20260924-101500-a1b2c3/metrics.jsonl", lines=True)
```

## Development

```bash
git clone https://github.com/Moe-Zbeeb/runboard && cd runboard
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
npm install
npm run check:cloud
node tests/js/binned.test.cjs && node tests/js/app.test.cjs
```

Try it end to end with `runboard serve` in one terminal and `python examples/quickstart.py` in another.
See [CONTRIBUTING.md](CONTRIBUTING.md) for details.

## License

[MIT](LICENSE). Bundles [uPlot](https://github.com/leeoniya/uPlot) (MIT).
