Metadata-Version: 2.4
Name: iperf-orchestrator
Version: 2.5.0
Summary: Full-mesh iperf2 throughput testing across a fleet, with CSV/pivot/heatmap analysis
Author: Martin J. Gallagher
Maintainer: Martin J. Gallagher
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/MartinGallagher-code/iperf_orchestrator
Project-URL: Repository, https://github.com/MartinGallagher-code/iperf_orchestrator
Project-URL: Issues, https://github.com/MartinGallagher-code/iperf_orchestrator/issues
Project-URL: Changelog, https://github.com/MartinGallagher-code/iperf_orchestrator/blob/main/CHANGELOG.md
Keywords: iperf,iperf2,network,throughput,bandwidth,benchmark,mesh,fabric,load-testing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Networking
Classifier: Topic :: System :: Networking :: Monitoring
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: matplotlib
Dynamic: license-file

# iperf-orchestrator

[![PyPI version](https://img.shields.io/pypi/v/iperf-orchestrator.svg)](https://pypi.org/project/iperf-orchestrator/)
[![Python versions](https://img.shields.io/pypi/pyversions/iperf-orchestrator.svg)](https://pypi.org/project/iperf-orchestrator/)
[![CI](https://github.com/MartinGallagher-code/iperf_orchestrator/actions/workflows/ci.yml/badge.svg)](https://github.com/MartinGallagher-code/iperf_orchestrator/actions/workflows/ci.yml)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![REUSE status](https://api.reuse.software/badge/github.com/MartinGallagher-code/iperf_orchestrator)](https://api.reuse.software/info/github.com/MartinGallagher-code/iperf_orchestrator)
[![Documentation Status](https://readthedocs.org/projects/iperf-orchestrator/badge/?version=latest)](https://iperf-orchestrator.readthedocs.io/en/latest/)

A bash orchestrator for running full-mesh iperf2 throughput tests across a list of servers, collecting per-host CPU samples during the run, and producing a CSV, pivot table, and heatmap + bar chart visualization of the results.

Built primarily for **network fabric stress testing**: load every link in both directions simultaneously and find out what breaks or degrades. Also useful for one-off "is the network healthy" surveys of a fleet.

<!-- docs:end -->

📖 **Full documentation:** [iperf-orchestrator.readthedocs.io](https://iperf-orchestrator.readthedocs.io/en/latest/) — the same content as this README, split into pages and searchable.

---

<!-- docs:installation -->
## Installation

### With pip (recommended)

```bash
pip install iperf-orchestrator
```

This puts an `iperf-orchestrator` command on your PATH and installs the Python
libraries the heatmap step needs (`numpy`, `matplotlib`). Requires
`bash` and Python 3.6+ on the orchestrator host. Everywhere below, wherever you
see `./iperf_orchestrator.sh`, you can use the `iperf-orchestrator` command
instead — e.g. `iperf-orchestrator --servers servers.txt all`. `python -m
iperf_orchestrator` also works.

When run as the installed command, results are written to `./results/` in your
current working directory (and a `./servers.txt` there is picked up
automatically if you don't pass `--servers`).

### From a source checkout

The orchestrator itself is a single self-contained bash script; you can run it
directly without installing anything:

```bash
./iperf_orchestrator/iperf_orchestrator.sh --servers servers.txt all
```

The analysis steps need Python 3.6+. Only `make-heatmap` needs third-party
packages (`numpy` and `matplotlib`); `make-pivot`, `parse-cpu`,
`collect-results` and `results-summary` are stdlib-only. Run `doctor` to check.

<!-- docs:end -->
---

<!-- docs:quick-start -->
## Quick Start

```bash
# 1. List your servers, one IP or hostname per line ('#' for comments)
cat > servers.txt <<EOF
10.0.0.10
10.0.0.11
10.0.0.12
10.0.0.13
EOF

# 2. Make sure key-based SSH to every host is already set up, e.g.
#    for h in $(grep -v '^#' servers.txt); do ssh-copy-id "$h"; done

# 3. Write the plan file once: hosts + every setting in one artifact.
#    Every later command reads it, so no flag needs repeating.
./iperf-orchestrator.sh gen --servers servers.txt

# 4. Run everything end-to-end (start + summarize + stop + clean)
./iperf-orchestrator.sh run
```

Or drive it step by step with the six core commands (mirroring
[`matrix_orchestrator`](https://github.com/MartinGallagher-code/matrix_orchestrator)'s
workflow):

| Command | What it does |
|---|---|
| `gen` | Build `iperf_plan.conf` from your server list — hosts + settings in one file. |
| `start` | Start the iperf2 daemons everywhere and run the tests. |
| `status` | One line per host: probes, daemons, and live test progress. |
| `summarize` | Collect results; render CSV, pivot, heatmap; print the summary and what to do next. |
| `stop` | Stop the daemons (test logs stay on the hosts). |
| `clean` | Stop, then delete the remote work dir everywhere — verified gone. |

Not sure what to ask for? `./iperf-orchestrator.sh hints` turns a goal into
the command that gets you there. The classic one-liner still works too:
`./iperf-orchestrator.sh --servers servers.txt all`.

Each invocation that produces results creates a fresh timestamped run directory under `./results/<run-id>/`, and a `./results/latest` symlink is updated to point at it. Analysis subcommands default to following `latest`; pass `--run-id <id>` to address an older run.

Results in `./results/<run-id>/`:
- `iperf_results.csv` — every test, both directions, fully parsed
- `cpu_summary.csv` — per-host CPU peaks during the run
- `iperf_pivot.txt` — text pivot table of throughput
- `iperf_heatmap.png` — heatmap + sorted bar chart with CPU annotations
- `iperf_overlay.tsv` — datacenter layout overlay samples (`export-overlay`, or `--overlay`)

<!-- docs:end -->
---

<!-- docs:requirements -->
## Requirements

### On the orchestrator host (where you run the script)
- bash 4+
- ssh, scp
- Python 3.6+ (stdlib only), plus `numpy` and `matplotlib` for the heatmap step
- tar, gzip

### On every server in the mesh
- **iperf2** (binary name `iperf`, *not* `iperf3`) — version 2.0.13+ recommended for `--full-duplex`
- **sysstat** providing `mpstat` (recommended) — falls back to `/proc/stat` sampling if missing
- ssh access from the orchestrator

The script's `check-iperf` subcommand verifies both iperf2 and mpstat before you start.

### Why iperf2 and not iperf3
iperf3's server is single-threaded and accepts only one client at a time. A full mesh of N hosts would need N iperf3 servers per host on different ports just to function, plus a port-assignment scheme, plus N times the firewall holes. iperf2's multi-threaded server handles concurrent clients on a single port — one daemon per host on port 5001 and you're done. iperf2's `--full-duplex` also gives you a true single-socket bidirectional test, which is what fabric stress testing actually wants.

<!-- docs:end -->
---

<!-- docs:subcommands -->
## Subcommands

Key-based SSH to every host must already be configured (the orchestrator
connects non-interactively with `BatchMode=yes`).

```
PLAN WORKFLOW (each verb reads the plan file, so flags never repeat):
  gen [MODE]             Write iperf_plan.conf: host list + settings in one file
  start [MODE]           start-servers + run-tests in one verb
  status                 Probes, daemons, and a live progress line per host
  summarize              process + pivot + results-summary (with what-next hints)
  stop                   stop-servers, plus the next steps spelled out
  clean                  stop + remove $REMOTE_DIR everywhere, then verify
  run [MODE] [--for N]   all + results-summary; --for pins total-time (rolling)
                         or per-test duration (other modes)
  hints                  What you want to know -> the command that gets you there

SETUP:
  check-iperf            Verify iperf2 + mpstat presence on every host
  check-servers          Check which hosts have iperf -s currently running

EXECUTION:
  start-servers          Start iperf2 -s on every host (port 5001)
  run-tests [MODE]       Run the tests. Auto-runs create-scripts and
                         distribute-scripts for non-rolling modes. MODE is
                         parallel | sequential-host | sequential-pair | rolling.
  collect-results        Pull logs back as a tar archive per host
  stop-servers           Kill iperf -s on every host
  cleanup --yes          Remove $REMOTE_DIR on every host

ANALYSIS:
  parse-csv              Parse iperf2 CSV logs into iperf_results.csv (2 rows per test)
  parse-cpu              Parse mpstat samples into cpu_summary.csv
  make-pivot             Text pivot table at iperf_pivot.txt
  make-heatmap           Heatmap + bar chart at iperf_heatmap.png
  process                = collect-results + parse-csv + parse-cpu + make-pivot + make-heatmap
  results-summary        P50/P95/min/mean/max throughput + 5 slowest pairs
  export-overlay         Render the run as datacenter layout viewer overlays:
                         throughput, relative-to-median, pair asymmetry,
                         per-host success rate and CPU

INTERNAL (run-tests calls these for you; available standalone if needed):
  create-scripts         Generate per-host client run scripts locally
  distribute-scripts     Push each host's script out

CONVENIENCE:
  all [MODE] [--keep-going]  Run the full sequence end-to-end
                             (start + run-tests + process + stop)
  status                     Probe hosts live + list available runs
  doctor                     Check local prerequisites
  help                       Common commands and flags
  help-advanced              Every command, every flag, every env var
```

The orchestrator is **stateless**: nothing persists between invocations except the contents of the results directory. `status` derives state by probing hosts directly. Server lists are passed via `--servers`/`IPERF_SERVERS`/`./servers.txt` — or carried by the plan file. Each pipeline run creates a fresh `<results>/<run-id>/` directory; `<results>/latest` is updated to point at the most recent one.

### The plan file

`gen` writes a single plan file (default `./iperf_plan.conf`; `--plan FILE`
or `$IPERF_PLAN` override) that every other command reads, so a run is
reproducible from one artifact and no flag needs repeating:

```text
# iperf-orchestrator plan v1 -- one file drives every command
# mode=parallel
# port=5001 duration=10 streams=1 host_flows=1 total_time=300
# bandwidth= length= window= mss= no_nagle=0
# bind= server_bind=
# ssh_user=root remote_dir=/tmp/iperf_orchestrator
10.0.0.10
10.0.0.11
10.0.0.12
```

Hosts are plain lines (the plan doubles as the server list); settings ride
in the `key=value` tokens. Edit either by hand and just re-run. Precedence
is `CLI flag > env var > plan file > built-in default`, so a one-off
`--duration 60` still wins without touching the plan, and re-running `gen`
preserves any setting you don't override.

**Partial mesh.** `gen --grid` writes the hosts as an mx-style pair grid
instead of a plain list — rows send, columns receive, and a non-empty cell
tests that directed pair:

```text
src\dst,10.0.0.10,10.0.0.11,10.0.0.12
10.0.0.10,,x,x
10.0.0.11,x,,x
10.0.0.12,,x,
```

Blank a cell to skip a direction (here `10.0.0.12` never sends to
`10.0.0.10`); blank both cells to drop the pair entirely. Every mode —
parallel, sequential, and rolling — honors the grid, `status` shows each
host's own expected test count, `hints` sizes its estimates from the
enabled edges, and re-running `gen` on a grid plan keeps your blanked
cells. A plain host list simply means full mesh, and plain `servers.txt`
files keep working everywhere.

### Configuration

Every setting can be supplied either as an environment variable or as a CLI flag. The CLI flag wins when both are set. Flags accept both `--flag value` and `--flag=value` forms, and may appear before the subcommand:

```bash
./iperf-orchestrator.sh --duration 60 --ssh-jobs 32 all
./iperf-orchestrator.sh -d 60 -j 32 all parallel
IPERF_DURATION=60 ./iperf-orchestrator.sh all          # env var still works
```

| Env var | Flag | Default | Purpose |
|---|---|---|---|
| `IPERF_PLAN` | `--plan` | `./iperf_plan.conf` when it exists | plan file written by `gen` |
| `IPERF_MODE` | *(plan `mode=` key)* | `parallel` | default run mode when none is passed |
| `IPERF_SERVERS` | `--servers`, `-s` | plan hosts, else `<script-dir>/servers.txt` | server list path |
| `RESULTS_BASE` | `--output`, `-o` | `<script-dir>/results` | base directory for run subdirs |
| `IPERF_RUN_ID` | `--run-id` | auto-timestamp on write; `latest` symlink on read | which run subdir to address |
| `IPERF_PORT` | `--port` | `5001` | iperf2 listening port |
| `IPERF_DURATION` | `--duration`, `-d` | `10` | seconds per test |
| `IPERF_TEST_TIMEOUT` | `--test-timeout` | `duration + 30` | hard cap per iperf -c, enforced with coreutils `timeout` on the remote host; `0` disables |
| `IPERF_SINGLE_SERVER` | `--single-server` | *(unset)* | sequential-host mode only: one all→one round against this host instead of the host-at-a-time sweep |
| `IPERF_STREAMS` | `--streams`, `-P` | `1` | parallel TCP streams within each test (iperf2 `-P`) |
| `IPERF_SSH_JOBS` | `--ssh-jobs`, `-j` | 4×cores, capped at 32 | max concurrent SSH/SCP fan-out (capped concurrency) |
| `IPERF_TOTAL_TIME` | `--total-time` | `300` | rolling mode wall-time (seconds) |
| `IPERF_HOST_FLOWS` | `--host-flows` | `1` | concurrent iperf processes per directed edge (parallel/sequential) or per-host concurrency cap (rolling) |
| `IPERF_BANDWIDTH` | `--bandwidth`, `-b` | *(unset)* | cap per-flow target rate, e.g. `100M`, `1G` (iperf2 `-b`) |
| `IPERF_LENGTH` | `--length`, `-l` | *(unset)* | TCP read/write buffer size, e.g. `128K` (iperf2 `-l`) |
| `IPERF_WINDOW` | `--window`, `-w` | *(unset)* | TCP window / socket buffer (iperf2 `-w`) |
| `IPERF_MSS` | `--mss`, `-M` | *(unset)* | TCP maximum segment size (iperf2 `-M`) |
| `IPERF_NO_NAGLE` | `--no-nagle`, `-N` | `0` | disable Nagle's algorithm (iperf2 `-N`) |
| `IPERF_BIND` | `--bind`, `-B` | *(unset)* | bind clients to the matching NIC; substring-matched against `ip -o -4 addr show` |
| `IPERF_SERVER_BIND` | `--server-bind` | mirrors `--bind` | bind the daemon side too; `--server-bind=''` keeps 0.0.0.0 |
| `IPERF_DRY_RUN` | `--dry-run`, `-n` | `0` | print SSH/SCP commands instead of executing |
| `IPERF_VERBOSITY` | `--verbose`/`-v`, `--quiet`/`-q` | `1` | `-v` prints every ssh/scp invocation; `-q` suppresses non-WARN/ERROR logs |
| `SSH_USER` | `--ssh-user`, `-u` | `$USER` | SSH login user |
| `START_DELAY` | `--start-delay` | `30` | seconds in the future to schedule the synchronized start |
| `REMOTE_DIR` | `--remote-dir` | `/tmp/iperf_orchestrator` | remote working dir; safe to point at a shared FS (every remote-side file embeds `<host>_<run-id>`) |
| `PYTHON_BIN` | `--python` | `python3` | Python interpreter for analysis steps |

#### `--ssh-jobs` and capped-concurrency parallel SSH

Setup and teardown subcommands (`check-iperf`, `check-servers`, `start-servers`, `distribute-scripts`, `collect-results`, `stop-servers`, `cleanup`) fan out to all hosts in parallel, capped at `IPERF_SSH_JOBS` concurrent SSH/SCP sessions. Per-host output is captured in worker buffers and replayed in server-list order so the screen output stays readable.

The default is derived from the orchestrator host's core count (4× cores, floor 4, capped at 32), which keeps its SSH agent and the per-host sshd happy on most fleets. Bump it (e.g. `--ssh-jobs 64`) when you have hundreds of hosts and the orchestrator's CPU/network can absorb it; lower it if `MaxStartups` on your sshds rejects connections.

The actual `run-tests parallel` mode is *not* throttled by `--ssh-jobs` — it has to open one SSH session per host simultaneously to hit the synchronized start barrier. `--ssh-jobs` only caps the setup/teardown fan-outs.

#### SSH key setup

The orchestrator does not distribute SSH keys. Configure key-based, non-interactive SSH to every host before running it — for example with `ssh-copy-id`:

```bash
for h in $(grep -v '^#' servers.txt); do ssh-copy-id "$h"; done
```

Every connection uses `BatchMode=yes`, so any host still requiring a password will simply fail rather than prompt.

#### `--keep-going` for `all`

`all --keep-going` continues past per-host failures (a single host failing `start-servers` no longer aborts the whole pipeline). Without it, the first step that records per-host failures aborts the pipeline.

#### `--dry-run`, `--verbose`, `--quiet`

`--dry-run` prints every SSH/SCP command without executing — useful for inspecting what `all` would do before unleashing it on a fleet. `--verbose` echoes every ssh/scp invocation as it runs; `--quiet` drops normal INFO logs and keeps only WARN/ERROR.

<!-- docs:end -->
---

<!-- docs:run-modes -->
## Run modes

`run-tests` (and therefore `all`) takes a mode argument that controls how the tests are scheduled. The first three modes use canonical-pair generation — each unordered pair `{A, B}` is tested exactly once, with `--full-duplex` measuring both directions concurrently on a single TCP socket. `rolling` is structured differently (see below).

| Mode | What runs concurrently | Wall-clock at N=100, DUR=10 | When to use |
|---|---|---|---|
| `parallel` (default) | all hosts launch all of their clients at once after a synchronized start | ~1 × DURATION (~50s) | fabric stress testing: load everything at once and see what breaks |
| `sequential-host` | one host at a time runs all of its clients in parallel | ~N × DURATION (~17 min) | clean numbers per host without inter-host interference |
| `sequential-host --single-server HOST` | every *other* host targets just HOST, simultaneously (all→one) | ~1 × DURATION | incast: what one server's inbound looks like with the whole fleet converging on it |
| `sequential-pair` | exactly one connection on the wire at any moment | ~N(N-1)/2 × DURATION (~14 hr) | cleanest possible per-pair numbers; usually overkill |
| `rolling` | each host independently picks its least-tested peer, runs one short iperf, repeats for `--total-time`; up to `--host-flows` concurrent flows per host | bounded by `--total-time` | only practical mode at very large N: per-host load is `--host-flows`, independent of fleet size |

Every iperf client is additionally wrapped in a hard per-test time limit
(coreutils `timeout` on the remote host, when present): iperf2's `-t` bounds
the send window, but a client that cannot connect or wedges on a dead peer
would otherwise stall the whole round. The default cap is `--duration` + 30
seconds; `--test-timeout SECONDS` overrides it, and `--test-timeout 0`
disables the cap. A test killed by the cap is reported as a `FAIL` with
`killed by test timeout` in its log.

<!-- docs:end -->
---

<!-- docs:how-it-works -->
## How it works

### Synchronized start (`parallel` mode)
The orchestrator computes `start_time = now + START_DELAY` once locally, pushes that epoch timestamp to every host as a script argument, and each remote run-script busy-waits until that epoch before launching iperf. All hosts start within a fraction of a second of each other.

### Per-host run scripts
Generated locally with each host's targets baked in, then distributed once via scp. The remote side has no orchestration logic — it's just a target list, a synchronization barrier, mpstat in the background, and a fan-out of `iperf -c ... --full-duplex` calls (one per target, all backgrounded and `wait`-ed).

### Balanced pair assignment (the parity rule)
For an unordered pair `{A, B}` somebody has to be the iperf2 client. Naive "lex-smaller is always the client" gives terrible load imbalance: at N=100 the lex-first host runs 99 clients and the lex-last runs 0.

The fix is the **parity rule on host indices**: for indices `i, j` (positions in the sorted server list), the client is the smaller index when `(i+j)` is even, the larger when `(i+j)` is odd. Both endpoints compute the same answer independently, so no coordination is needed.

| N | spread | meaning |
|---|---|---|
| odd | 0 | every host runs exactly `(N-1)/2` clients |
| even | 1 | every host runs `(N-2)/2` or `N/2` clients |
| 100 | 1 | every host runs 49 or 50 clients (was 0..99 before) |

### CPU sampling
Each host runs `mpstat -P ALL 1 N` in the background, started right before iperf and running for `DURATION + 4` seconds. The fallback (`/proc/stat` deltas) kicks in if mpstat isn't installed; the parser detects which format it's reading.

The bar chart annotates each host's bar with peak CPU. Three patterns to watch for:

| Pattern | What it usually means |
|---|---|
| `peak_total_pct` low but `peak_softirq_pct` high on one core | RSS isn't spreading NIC IRQs — one core is doing all the packet work |
| `peak_idle_floor_pct` near 0 but `peak_total_pct` low | One specific core is pinned (usually by softirq) while others sit idle |
| `peak_total_pct` near 100 across all hosts | True CPU bound — throughput numbers measure the CPU, not the fabric |

### Result collection
For each host, one ssh + one scp + one local untar — instead of N-1 individual scp calls. At N=100 that's roughly 300 SSH/SCP operations across the whole pipeline instead of 5,000.

### Hostname sanitization in filenames
Server-list entries like `host.example.com`, `2001:db8::1`, or `user@10.0.0.1` are sanitized when used in filenames (`iperf_test_<src>_to_<dst>.log`, `cpu_<host>.log`, `run_<host>.sh`, the per-host tarballs). Slashes, colons, and `@` are replaced so the path is well-formed on every filesystem; the parser reverses the mapping when reading filenames back into the CSV. This means an IPv6 address or a `user@host` entry no longer produces broken paths or silently dropped logs.

### Heatmap auto-degradation
Cell annotations, axis labels, and figure size adapt to N:

| N | Cell labels | Axis labels | Bar value labels | Figure size |
|---|---|---|---|---|
| ≤ 30 | yes | every host | yes | scales linearly |
| 31–50 | no | every host | yes | scales linearly |
| 51–60 | no | every host (smaller font) | no | capped |
| > 60 | no | every Nth (~30 shown total) | no | capped at 36×32 inches |

At N=100 the heatmap renders as a ~280KB PNG in a few seconds, with slow hosts visible as red rows and columns.

<!-- docs:end -->
---

<!-- docs:output-schema -->
## Output schema

### `iperf_results.csv` (one row per direction, two per test file)

| Column | Meaning |
|---|---|
| timestamp | iperf2's reported test time |
| source | sender host for this row's direction |
| target | receiver host for this row's direction |
| status | `OK`, `NO_HEADER`, `NO_SUMMARY`, `DIRECTION_MISSING`, `READ_ERROR` |
| protocol | `TCP` |
| duration_s, parallel_streams | from the run-script header |
| bytes_transferred, bps, mbps | the throughput numbers |
| src_port, dst_port | raw from iperf2 |
| pair_a, pair_b | the canonical pair this row came from |
| filename, error | log file and any error text |

### `cpu_summary.csv` (one row per host)

| Column | Meaning |
|---|---|
| host | from the cpu log filename |
| source | `mpstat` or `proc_stat` (which parser handled it) |
| n_cpus | core count (mpstat only) |
| peak_total_pct | max box-wide `100 - %idle` across samples |
| mean_total_pct | mean box-wide `100 - %idle` |
| peak_softirq_pct | max `%soft` on any single core |
| peak_softirq_cpu | which core (mpstat only) |
| peak_sys_pct | max `%sys` (box-wide) |
| peak_user_pct | max `%usr` (box-wide) |
| peak_idle_floor_pct | lowest `%idle` on any single core |

### Heatmap reading
- **Rows = source** (sender direction)
- **Columns = target** (receiver direction)
- Cell `(A, B)` is the throughput when A was sending to B during the full-duplex test
- A row that's all red → that host has bad outbound
- A column that's all red → that host has bad inbound
- The bar chart underneath ranks hosts by mean outgoing Mbps with peak CPU% labeled when available

<!-- docs:end -->
---

<!-- docs:overlays -->
## Datacenter layout overlays

A mesh test answers "which host is slow"; a floor plan answers "which rack".
`export-overlay` renders a run as overlays for the
[datacenter layout viewer](https://github.com/MartinGallagher-code/datacenter_visualization),
which paints them over a `.dc` floor plan, so the numbers land on the hardware
that produced them.

```bash
./iperf-orchestrator.sh export-overlay          # -> results/latest/iperf_overlay.tsv
./iperf-orchestrator.sh run --overlay           # write it as part of the pipeline
```

Then open the viewer with the layout and the samples:

```
http://localhost:8000/?layout=floor.dc&results=iperf_overlay.tsv
```

### The overlays

| Overlay | Per | What it is |
|---|---|---|
| `iperf_mbps_out` / `iperf_mbps_in` | direction | throughput, credited to the sender and to the receiver |
| `iperf_mbps_duplex` | host | everything that host carried at once, both directions |
| `iperf_gbytes` | host | total data carried; bytes add over time, so this is exact in every mode |
| `iperf_line_util` | direction | throughput as a % of the NIC's line rate (with `--overlay-line-rate`) |
| `iperf_achieved` | direction | throughput as a % of the `-b` rate the run asked for |
| `iperf_rel_median` | direction | that direction against the run's own median, in % |
| `iperf_asymmetry` | pair | the gap between a pair's two directions, in % of the faster one |
| `iperf_status` | direction | `OK`, `FAIL` (with the status, error text and log to open), or `NO-DATA` |
| `iperf_fail_kind` | direction | the failures only, coloured by why (`NO_SUMMARY`, `DIRECTION_MISSING`, …) |
| `iperf_ok_pct` | host | how much of that host's mesh actually measured |
| `iperf_peers` | host | how many distinct peers it exchanged data with |
| `iperf_coverage` | host | those peers as a % of the ones it was *planned* to reach |
| `iperf_tests` | host | how many directed tests it took part in |
| `iperf_state` | host | the roll call: `TESTED`, or `NO-DATA` for a host that never answered |
| `iperf_cpu_peak`, `_mean`, `_softirq`, `_sys`, `_user`, `_idle_floor` | host | from `cpu_summary.csv` |
| `iperf_bind_iface` | direction | which NIC the traffic rode, when `--bind` was used |

The three derived overlays are the ones worth opening first, because they say
things the raw Mb/s cannot:

- **`iperf_rel_median`** scores every direction against the median of the run
  it came from, so a bad link stands out at any fabric speed — 100% is normal
  here, 45% is half speed, and you do not have to know what "good" is for this
  hardware. It ships with a diverging palette pinned at 0–200%, so slower-than-
  normal and faster-than-normal read differently rather than as one ramp, and
  aggregates with `median`. That last choice matters on a mesh: every host
  talks to the slow host, so every host's *worst* direction is the one to it,
  and a `min` aggregation paints the whole floor red while hiding the host that
  is actually slow. The median distinguishes "I am slow" (all my directions
  are) from "I have a slow peer" (one is). Switch it to `min` in the viewer
  when you do want the worst link anywhere.
- **`iperf_asymmetry`** is the shape a duplex mismatch, a one-way policer or a
  congested return path makes — a pair whose two directions disagree. It is
  credited to both ends (either NIC can be the cause) and aggregates with
  `max`.
- **`iperf_ok_pct`** keeps a mostly-broken host from hiding behind its one good
  link. Sent and received directions are tallied separately and the **worse**
  side is the value, with `sent=`/`recv=` metadata naming which one failed: a
  host that receives fine and cannot send anything reads 0%, not the 50% an
  average of its two halves would suggest. Tallying both sides also means a
  receive-only host in a partial-mesh grid still gets a reading.

Every overlay arrives with units, ramp direction and a short name, and each
percentage states its real `0–100` range — an auto-scaled CPU overlay makes a
30% peak look alarming purely for being the highest number present. Sample
metadata carries the peer, the test's timestamp, which parser produced a CPU
row (`src=mpstat`), how many cores it saw, which core saturated (`core=7`),
and — on a failure — the error text and the log file to open (`err=`, `log=`).
The file header records the run id, its **mode** (a `parallel` run is the whole
fleet under load at once; `sequential-pair` is one link's uncontended maximum;
they are not comparable) and the duration/streams/protocol every sample shared,
so only a row that departs from them pays for saying so.

One measured row becomes one sample: nothing is averaged on the way out, so
the viewer's own aggregation menu does the reducing (`mean` reads as "across
peers", `max` as "the best peer", `min` as "the worst", `stdev` and `range` as
"how steady was it"). `--overlay-reduce` collapses to one median sample per
host per overlay when the raw row count matters.

### Rolling runs, and what "carried at once" means

`iperf_mbps_duplex` is the one figure that cannot be a plain sum. Parallel and
sequential modes fire a round's flows together, so those rates genuinely add;
rolling mode probes the same pair over and over, and adding those reports more
than the NIC can carry. So rows are clustered by overlapping test window,
summed inside a cluster and averaged across clusters — the same rule
`make-pivot` uses on its cells, for the same reason. A CSV with no
`test_start` cannot say what was in flight together, and there the overlay is
left out with a note on stderr rather than guessed at. `iperf_asymmetry`
compares each direction's *median* for the same reason: on a rolling run,
comparing one arbitrary probe against another turns ordinary variance into an
apparent duplex fault.

**A direction that produced no number is never exported as 0 Mb/s.** Zero is a
measurement, and averaging it in makes a broken link read as a slow one. Those
directions leave as `iperf_status=FAIL` and pull their host's `iperf_ok_pct`
down — visible on the rack, absent from the throughput math. The same holds for
blank cells in `cpu_summary.csv` (a `/proc/stat` host has no per-core columns):
skipped, not zeroed.

### Two blind spots this closes

**A host that never reported.** If a host in the server list produced no rows
at all — SSH refused, iperf2 missing, box down — it has nothing to paint, and
an element with no data on a floor plan reads as "not part of this test". That
is the one reading that is certainly wrong. Those hosts are exported as
`iperf_status=NO-DATA` with 0% coverage, so they sit on the same overlay as
every other broken host instead of disappearing.

**A fabric that is uniformly slow.** `iperf_rel_median` compares each direction
against the run's own median, which is exactly what you want when one link is
sick — and useless when *everything* is at half speed, because the median is
half speed too and every direction reads a comfortable 100%. The viewer's
auto-scaled colours hide it for the same reason. Tell the export what the NIC
is and both problems go away:

```bash
./iperf-orchestrator.sh export-overlay --overlay-line-rate 25000   # 25 GbE
```

Throughput overlays are then scaled `0..25000` (and duplex `0..50000`) instead
of to whatever this run produced, and `iperf_line_util` gives each direction as
a percentage of line rate. It is never guessed: without the flag the overlay is
absent, because a run cannot tell you what the hardware was capable of.

### Comparing two runs

Every overlay is named with a prefix (`iperf_` by default). Give two exports
different prefixes and they load into one file as separate overlays, so the
viewer shows them side by side instead of averaging them together:

```bash
./iperf-orchestrator.sh --run-id monday export-overlay \
    --overlay-out compare.tsv --overlay-test-prefix before_ --overlay-run before
./iperf-orchestrator.sh export-overlay \
    --overlay-out compare.tsv --overlay-append --overlay-test-prefix after_ --overlay-run after
```

`--overlay-run LABEL` sets the `run=` tag each sample carries, which is what
tells them apart in the inspector when they *do* share an overlay.

### Kinship with `mx export`

[`matrix_orchestrator`](https://github.com/MartinGallagher-code/matrix_orchestrator)
writes overlays into the same results file, and the two tools deliberately
answer the same questions the same way: `iperf_achieved` is `mx_achieved`
against a target rate, `iperf_coverage` is `mx_coverage`, `iperf_tests` is
`mx_intervals`, `iperf_state` is `mx_state`, and both keep per-host and
per-peer values in separate overlays so the two granularities never reduce
into each other. Numbers are written the same way too — four significant
digits, fixed notation, because `%g` turns a host's aggregate into
`1.163e+06`, which is correct and unreadable in a file people grep. Load an
`mx` export and an iperf export together and they read as one system.

### Matching hosts to the layout

Targets are the names this orchestrator tested with, and the viewer resolves
any unique suffix of an element path — so a hostname-shaped layout
(`wr12r06u15`) matches with no mapping at all, and generating `servers.txt`
from the `.dc` file is the way to keep it that way. When the two disagree
(an IP list against a hostname floor plan), map them:

```bash
cat > hosts.map <<'MAP'
10.0.0.10   wr01r01u01      # <host as tested>  <layout element>
10.0.0.11   wr01r01u02
MAP
./iperf-orchestrator.sh export-overlay --overlay-map hosts.map
```

Peers are renamed too, so the inspector stays in one namespace.
`--overlay-prefix DH1/A/` prepends a path instead, for addressing one row of a
bigger floor plan. Hosts missing from a map file keep their own name and are
reported on stderr rather than dropped.

### Appending runs

The viewer's results format is append-only: every sample carries `run=<run-id>`,
and a nightly export into one file accumulates history the viewer can aggregate
(`last` for right now, `min` for the worst night).

```bash
./iperf-orchestrator.sh export-overlay --overlay-out nightly.tsv --overlay-append
```

`--overlay-format ndjson` writes one JSON object per line instead — the same
overlays, for anything that would rather generate JSON than columns (the
extension picks it automatically, so `--overlay-out nightly.ndjson` is enough).
`--overlay-out -` writes to stdout, and `--overlay-no-meta` omits the `!test`
lines that carry units, ranges, palettes and short names.

<!-- docs:end -->
---

<!-- docs:run-directories -->
## Stateless mode and run directories

There is no state file. Each invocation that produces results creates a fresh `./results/<run-id>/` directory (run-id = timestamp), and `./results/latest` is updated to point at it. Read-side commands (`parse-csv`, `parse-cpu`, `make-pivot`, `make-heatmap`, `results-summary`, `export-overlay`) follow `latest` by default, or take `--run-id <id>` to address a specific run. `status` derives state by probing hosts live (running `iperf -v` and `pgrep iperf` on each).

Every subcommand is safe to re-run individually. Common workflows:

```bash
# Full pipeline (creates a new run-id)
./iperf-orchestrator.sh --servers servers.txt all

# Re-run just the analysis on the most recent run
./iperf-orchestrator.sh parse-csv
./iperf-orchestrator.sh parse-cpu
./iperf-orchestrator.sh make-pivot
./iperf-orchestrator.sh make-heatmap

# Re-render the heatmap for an older run
./iperf-orchestrator.sh --run-id 2026-05-05_14-22-01 make-heatmap

# Run two modes back-to-back; each gets its own results subdir
./iperf-orchestrator.sh --servers servers.txt all parallel
./iperf-orchestrator.sh --servers servers.txt all sequential-host
ls results/
```

### Shared-FS safety on the remotes

`REMOTE_DIR` (default `/tmp/iperf_orchestrator`) can safely point at a shared filesystem (NFS home, GPFS, etc.). Every remote-side file the orchestrator creates embeds both the sanitized hostname and the run-id, so simultaneous runs (or back-to-back runs sharing the same `REMOTE_DIR`) never overwrite each other. `cleanup` (without `--all`) only removes files matching the active run-id, leaving prior runs untouched.

<!-- docs:end -->
---

<!-- docs:design-decisions -->
## Design decisions and lessons learned

This section documents *why* the script is shaped the way it is. Most of these were arrived at by hitting a real problem and fixing it.

### iperf3 was the wrong tool for full-mesh testing
The first version of this script used iperf3 with JSON output. iperf3 has nicer reporting (TCP retransmits, CPU utilization in the output, structured JSON), but its server is single-threaded and accepts one client at a time. At 100 hosts, every other host trying to connect to one server simultaneously would mostly fail with "the server is busy running a test." Working around this means running 100 iperf3 daemons per host on 100 ports, plus a port-assignment scheme, plus 100× the firewall config. We switched to iperf2 and the architecture got dramatically simpler.

### Bidirectional testing per pair
Earlier versions ran two independent tests per host pair (A→B as one test, B→A as another). With iperf2 `--full-duplex`, one TCP socket carries traffic in both directions simultaneously, both numbers come out of the same test, and you halve the test count. For fabric stress testing this is also more realistic — real-world traffic isn't strictly unidirectional and the switch buffers don't see the same patterns.

### Parity rule for client assignment
The first canonical-pair scheme used "lex-smaller host is always the client." That's correct but pathologically imbalanced — lex-first runs N-1 clients, lex-last runs 0. The parity-on-indices rule gives every host roughly (N-1)/2 clients with a max-min spread of 0 (N odd) or 1 (N even). At N=100 this turns 0..99 client load into 49..50 client load. Both endpoints compute the rule independently and agree without coordination.

### Per-host iperf parallelism
A subtle bug in early versions: `parallel` mode synchronized the *start* of each host but each host then ran its iperf3 calls in a serial `for` loop. So all hosts started at T+0, but A→B finished before A→C started. The whole point of parallel mode is to load the wire all at once; we were missing it. Fixed by backgrounding each `iperf` call with `&` and `wait`-ing for the whole batch on each host.

### CPU sampling has to run on every host, including the no-client one
With balanced pair assignment, every host has *some* client work — but if N is odd or if you customize the rule, you can end up with a host whose only role is being a server for inbound flows. That host's CPU is still doing real work (handling N/2 inbound full-duplex sockets), so it still needs to be sampled. The run-script keeps the mpstat sampler running even when its target list is empty.

### Synchronized barrier instead of locks/coordination
The first design considered something like a GPIO-style "everyone signal ready" barrier. Vastly simpler: pick a future epoch timestamp, push it to every host, each host busy-waits until then. No coordination, no failure modes around partial-readiness, no protocol to debug. Just a number.

### Tar-batched result collection
N-1 sequential scp calls per host at N=100 is ~80 minutes of pure SSH handshake overhead. One tar + one scp + one local untar per host is closer to a few minutes. Same data, ~17× faster, and the tarball naming naturally namespaces server-side log files (which were originally going to collide on extract).

### Embedded Python for analysis
The bash script runs locally only; on remote hosts the only assumption is iperf2 (and ideally mpstat). The CSV parser, pivot generator, and matplotlib renderer are embedded as heredoc Python in the orchestrator. This keeps it one file you can scp and run, no `pip install -e .` ceremony.

### Fail open, not closed, on individual hosts
`set -e` is intentionally **not** set. The orchestrator tracks failures per host and per step, warns about them, and keeps going so one bad host doesn't abort a 100-host run. Each step's success criterion is "did *enough* of it work to make the next step useful," not "did every single host succeed."

### What CPU sampling reveals
The most common surprise on first runs: the heatmap shows a host with low throughput, you assume the network is bad, then `cpu_summary.csv` shows that host pegged at 100% CPU during the test. The throughput number was measuring the CPU, not the fabric. The peak %CPU annotation on the bar chart exists to surface this immediately rather than letting you misread the heatmap.

A second common surprise: `peak_total_pct` is 30% but `peak_softirq_pct` is 100% on core 0. The box-wide CPU looks fine, but RSS is hashing every flow to the same core. This is invisible without per-core data, which is why `mpstat -P ALL` is preferred over the `/proc/stat` fallback.

<!-- docs:end -->
---

<!-- docs:limitations -->
## Limitations and known gaps

- **No automatic retry on transient SSH failures.** If `start-servers` fails on one host, the orchestrator reports it and moves on (`all` aborts unless `--keep-going` is passed). Re-run the subcommand to pick up stragglers.
- **`sequential-pair` at N=100 takes ~14 hours.** The cleanest mode is also the slowest. If you want sequential-pair-quality numbers in less time, the right approach is round-robin tournament scheduling (pack all `N(N-1)/2` edges into ~N-1 rounds where every host has at most one flow per round). Not implemented; would be moderate complexity.
- **Heatmap above ~60 hosts loses cell labels.** This is by design — they're unreadable at that density — but it means you have to read the colormap or the CSV for exact values.
- **Asymmetric NIC speeds aren't auto-handled.** If half your fleet is 1G and half is 10G, the heatmap colormap is dominated by the 10G hosts and the 1G hosts all look very red. Acceptable for our use case ("uniform fleet"); for a heterogeneous fleet you'd want per-pair expected-bandwidth normalization.
- **No UDP testing.** Fabric stress testing usually wants TCP because that's what real workloads do; if you specifically need UDP loss/jitter measurements, the iperf invocation in the generated run script needs `-u` and the parser needs to read different CSV columns.

> **Resolved**: setup/teardown fan-out is now capped-concurrency parallel SSH (see `--ssh-jobs`).

<!-- docs:end -->
---

<!-- docs:troubleshooting -->
## Troubleshooting

**`check-iperf` says `WRONG_VERSION`.** Some distributions ship `iperf` as a symlink to `iperf3`. Verify with `iperf -v` on the host and install the actual iperf2 package (`apt install iperf` on Debian/Ubuntu, where `iperf` is iperf2 and `iperf3` is iperf3).

**`run-tests` finishes but logs are full of "the server is busy" errors.** Confirms an iperf3 instance is still listening on port 5001. `pkill -x iperf3` and re-run `start-servers`.

**One host's row is all NaN in the pivot.** Either it failed to start its iperf2 server, or it failed to run its client script. Check `./results/latest/logs/run_<host>.log` and `./results/latest/iperf_run_<host>_<run-id>.status`.

**Heatmap shows surprisingly low numbers everywhere.** Look at `cpu_summary.csv`. If `peak_total_pct` is near 100%, you're CPU-bound, not network-bound. Possible fixes: bigger hosts, more cores, RSS tuning, or run in `sequential-host` mode to see what each host can do without contention.

**`make-heatmap` errors out with `Missing Python package`.** `pip install matplotlib numpy` (or your distro's equivalent). Only the `make-heatmap` step needs these — the other steps work without them. Run `./iperf-orchestrator.sh doctor` to get a single report of every missing local prerequisite plus install hints.

**Every host fails with a password prompt or `Permission denied`.** The orchestrator connects with `BatchMode=yes` and never distributes keys itself. Set up key-based SSH first, e.g. `for h in $(grep -v '^#' servers.txt); do ssh-copy-id "$h"; done`.

**A run aborted halfway and I want to pick up where it left off.** `./iperf-orchestrator.sh all --resume` consults `~/.iperf_orchestrator/state` and skips the steps already marked done. If the failure was on a single flaky host and you want to barrel past it instead, add `--keep-going`.

<!-- docs:end -->
---

<!-- docs:file-layout -->
## File layout

### Local: `<script-dir>/results/<run-id>/`

```
results/
  latest -> 2026-05-05_14-22-01    # symlink to most recent run
  2026-05-05_14-22-01/
    iperf_installed.txt            # check-iperf output (this run only)
    iperf_running.txt              # check-servers output (this run only)
    .run_mode                      # parallel | sequential-host | sequential-pair
    scripts/
      run_<host>_<run-id>.sh       # generated per-host run scripts
    logs/
      orchestrator.log             # everything the orchestrator did this run
      run_<host>.log               # stdout/stderr of each host's run-tests session
      run_<src>_to_<dst>.log       # sequential-pair only
    iperf_test_<src>_to_<dst>_<run-id>.log    # raw iperf2 CSV with header
    iperf_server_<host>_<run-id>.log          # server-side iperf log per host
    iperf_run_<host>_<run-id>.status          # per-host status timeline
    cpu_<host>_<run-id>.log                   # mpstat or proc_stat samples
    iperf_results.csv                          # parsed throughput data
    cpu_summary.csv                            # parsed CPU data
    iperf_pivot.txt                            # text pivot
    iperf_heatmap.png                          # heatmap + bar chart
    iperf_overlay.tsv                          # layout-viewer overlay samples
                                               # (export-overlay / --overlay only)
```

### Remote: `$REMOTE_DIR/` (default `/tmp/iperf_orchestrator/`)

```
run_iperf_<host>_<run-id>.sh
iperf_test_<src>_to_<dst>_<run-id>.log
iperf_server_<host>_<run-id>.log
iperf_run_<host>_<run-id>.status
cpu_<host>_<run-id>.log
```

Hostnames are sanitized: `:`, `/`, `[`, `]`, whitespace are replaced with `_` so bracketed-IPv6 names like `[fe80::1]` produce safe filenames. The unsanitized name is preserved inside file headers (`# pair_a=...`, `# host=...`).

<!-- docs:end -->
---

<!-- docs:completions -->
## Shell completions

Tab-completion stubs for the subcommand and global flags live in `completions/`:

```bash
# bash
source completions/iperf_orchestrator.bash
# or install system-wide:
sudo cp completions/iperf_orchestrator.bash /etc/bash_completion.d/

# zsh
fpath+=("$PWD/completions")
autoload -Uz compinit && compinit
```

<!-- docs:end -->
---

<!-- docs:tests -->
## Tests

The repository ships an extensive bash-based test suite under `tests/` covering pair assignment, parallel fan-out, the generated remote run-script, all parsers, the run-tests modes, the `doctor` command, the `--keep-going` semantics on `all`, and a long tail of edge cases.

```bash
./tests/run_tests.sh                           # everything
./tests/test_pair_assignment.sh                # one suite
```

<!-- docs:end -->
---

<!-- docs:documentation -->
## Documentation

The rendered documentation lives at
[iperf-orchestrator.readthedocs.io](https://iperf-orchestrator.readthedocs.io/en/latest/)
and is rebuilt by Read the Docs on every push to `main` (configuration in
`.readthedocs.yaml`).

The site has no prose of its own. Every page pulls its body straight out of
this README with a MyST `include` directive, sliced on the invisible
`<!-- docs:* -->` HTML comments that sit above each section, so there is one
copy of the text and the site cannot drift from the repository. `CHANGELOG.md`
and `PUBLISHING.md` are included whole.

To build it locally:

```bash
pip install -r docs/requirements.txt
python -m sphinx -b html -W docs docs/_build/html
```

`-W` matches the `fail_on_warning: true` that Read the Docs and CI both use.
When you add a section to this README, give it a `<!-- docs:slug -->` marker
and include the slice from a page under `docs/` — `tests/test_docs_sources.sh`
fails if you forget, and the docs build fails if a marker an include names
goes missing.

<!-- docs:end -->
---

<!-- docs:utility-scripts -->
## Utility scripts

`merge.sh` bundles a directory tree into a single text file and `split.sh`
expands it again — handy for moving the tree through a channel that only
carries plain text.

**They no longer live here.** The canonical copies are in
[shared_tools](https://github.com/MartinGallagher-code/shared_tools), under
`scripts/`. This repository used to carry its own pair, which is how the two
copies that once existed drifted apart in the first place — one repository's
copy gained features the others never saw. Use the shared ones:

```bash
git clone https://github.com/MartinGallagher-code/shared_tools
shared_tools/scripts/merge.sh bundle.txt some/dir   # bundle a tree
shared_tools/scripts/split.sh bundle.txt restored/  # expand it
```

When a transport caps the size of a single file, `-m` keeps every part under
a limit:

```bash
merge.sh -m 390K bundle.txt some/dir   # bundle-part1-of-N.txt, ...
split.sh bundle-part1-of-3.txt out/    # expand the parts in any order
```

> **Note:** the copy that used to live here took `-n PARTS` and produced
> `bundle.part1of2.txt`. The canonical version takes `-m SIZE` instead and
> names parts `bundle-part1-of-N.txt`. A size limit is the more useful knob,
> because the constraint is nearly always "the upload is refused above N
> bytes" rather than a part count — but any script or muscle memory using
> `-n` needs updating.

Parts are cut on entry boundaries, so no file is ever chopped in half and
each part is a complete, independently valid bundle with its own header and
checksums.

The bundle format inlines text files verbatim and base64-encodes binaries
(and any text file whose content would collide with the section markers).
Permissions, symlinks, empty directories and missing trailing newlines are
preserved; every file carries a sha256 that `split.sh` verifies on expansion,
and the header entry count catches a bundle truncated mid-transfer.
`split.sh` refuses bundles containing absolute or `..` paths and never passes
bundle-controlled strings to a shell.

`bundle.txt` is generated, not committed — rebuild it whenever you need one.

<!-- docs:end -->
---

<!-- docs:license -->
## License and contribution

This project is licensed under the **GNU General Public License v3.0** (GPL-3.0). See the [LICENSE](https://github.com/MartinGallagher-code/iperf_orchestrator/blob/main/LICENSE) file for the full text.

In short: you are free to use, modify, and redistribute this software, but any distributed derivative work must also be licensed under GPL v3 and made available in source form. The software is provided without warranty.

Contributions are welcome and, by submitting them, you agree that they will be licensed under the same GPL v3 terms.
