Metadata-Version: 2.5
Name: datacenter-layout-viewer
Version: 1.54.2
Summary: A very light datacenter floor-plan viewer: one plain-text file describes the whole datacenter, a second paints test results over it.
Project-URL: Homepage, https://github.com/MartinGallagher-code/datacenter_visualization
Project-URL: Source, https://github.com/MartinGallagher-code/datacenter_visualization
Project-URL: Changelog, https://github.com/MartinGallagher-code/datacenter_visualization/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/MartinGallagher-code/datacenter_visualization/issues
Author: Martin J. Gallagher
Maintainer: Martin J. Gallagher
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: datacenter,floor-plan,network,overlay,rack,visualization
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: JavaScript
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Topic :: System :: Networking :: Monitoring
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# datacenter-layout-viewer

[![tests](https://github.com/MartinGallagher-code/datacenter_visualization/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/MartinGallagher-code/datacenter_visualization/actions/workflows/tests.yml)
[![PyPI](https://img.shields.io/pypi/v/datacenter-layout-viewer.svg)](https://pypi.org/project/datacenter-layout-viewer/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://pypi.org/project/datacenter-layout-viewer/)
[![dependencies](https://img.shields.io/badge/dependencies-none-success.svg)](https://github.com/MartinGallagher-code/datacenter_visualization/blob/main/pyproject.toml)
[![License: GPL-3.0-or-later](https://img.shields.io/badge/license-GPL--3.0--or--later-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.html)
[![REUSE status](https://api.reuse.software/badge/github.com/MartinGallagher-code/datacenter_visualization)](https://api.reuse.software/info/github.com/MartinGallagher-code/datacenter_visualization)

A very light datacenter viewer. One plain-text file describes the whole
datacenter — rooms, rows, racks, servers, and the logical networks between them
— compactly enough that a hyperscale campus fits on a single page. A second
append-only file carries test results, which the viewer paints over the layout
as any number of simultaneous color overlays. A wide TSV table —
`Timestamp  host  var1  var2 …`, which is what monitoring already writes — is
read as it stands, and a folder of them can be reloaded on a timer as a live
dashboard. Where the layout is not written yet, one can be read out of the
hostnames on request.

No build step, no dependencies, no server-side anything. Static files only.

Current version **1.54.2** — see [CHANGELOG.md](CHANGELOG.md). One number
covers the viewer, the file formats and the tools; the formats are the
compatibility promise, so a file that loads today loads on every later 1.x
([how a release is cut](docs/releasing.md)).

## Run it

Any static file server works, and a clone is already the whole app:

```sh
python3 -m http.server 8000        # in the clone
# open http://localhost:8000/
```

Or install it, and get the server and the two overlay tools with it:

```sh
pip install datacenter-layout-viewer
dcviz serve                        # opens the viewer in a browser
dcviz serve --dir ~/layouts        # ... with that directory at /files/
```

The package has **no runtime dependencies** — it is the same static files
plus a hundred lines of `http.server` — and neither route is more supported
than the other. A server is not optional, though: browsers refuse to load ES
modules from a `file://` URL, so opening `index.html` directly gives a page
with dead buttons and a banner saying so.

The viewer starts empty. Load files with the **Load files…** button, from the
**Files** browser in the left panel, by dragging them onto the window, or with
URL parameters. Where the browser supports it (Chromium), the button reopens in
the directory it was last used in; elsewhere it is the ordinary file input:

```
http://localhost:8000/?layout=examples/small.dc&results=examples/small-results.tsv
```

Try the scale test: `?layout=examples/mega.dc` is a ~256,000-element
campus with ~560,000 cables, described in 45 lines.

Or write a layout from nothing: **Edit layout** opens an editor under the
canvas that re-parses on every keystroke, so the floor plan, a per-kind tally
(`3 rows · 24 racks · 216 nodes`) and any warnings follow the line being
written. **Insert template** drops in a commented skeleton to start from, and
**Download .dc** saves the result as a file. The same editor works on a loaded
file, which makes it the quickest way to learn what a line expands to. Drag
the bar along its top edge to make it taller or shorter; double-click the bar
to put it back.

The editor completes as you type (or on **Ctrl+Space**): kinds at the start of
a line, then the keys that line takes, then values — enumerated ones like
`mode=` and `style=`, and ones harvested from the layout itself, so `role=`
offers the roles this file already uses and `+` offers its tags. **Syntax**
opens a reference panel beside the text with every construct as a
click-to-insert snippet.

## The layout file (`.dc`)

Indentation-based. Every line is `<kind> <id> [key=value ...] [+tag ...]`:

```
dc IAD1 name="Acme IAD1" +prod region=us-east

  room DH1 name="Data Hall 1"
    row A..D +compute
      rack R[01..06] u=42
        node tor at=42 role=tor +switch
        node u[01..20] role=server +x86 +gpu model=hgx-h100
```

Those eight lines are 504 elements: ranges expand (`A..D`, `R[01..06]`,
`u[01..20]`), and the children of an expanded line are created once per
expansion. That multiplication is what makes the biggest datacenter fit on a
page — the `examples/mega.dc` campus is 8 halls × 24 rows × 40 racks × 41
devices from one nesting.

### Kinds

`dc`, `room`, `row`, `rack`, `node` get purpose-built layout (rows line racks
up on a shared floor; racks stack their children into real U-slots). **Any
other word works too** — `pod`, `hall`, `cage`, `suite` — and is drawn as a
generic container. There is no schema to declare.

Indentation is the structure, so a line indented one level too shallow
re-parents everything below it. The tell is a `node` that ends up containing
other elements — its rack sits empty with the "contents" drawn beside it —
and the parser reports it as a warning. Children sit deeper than their
parent; siblings align. (A deliberate container should be a non-node kind
like `chassis`, which nests silently.)

### Ranges

| Spec | Expands to |
|---|---|
| `R[01..20]` | `R01 R02 … R20` (zero-padding kept) |
| `u[1..40x2]` | `u1 u3 … u39` (step) |
| `A..H` | `A B … H` (letters; bare or bracketed — both ends must be the same alphabet, so `A..z` is refused rather than expanded through the punctuation between `Z` and `a`) |
| `[web\|db\|cache]` | `web db cache` (alternatives) |
| `r[1..2]-[a\|b]` | `r1-a r1-b r2-a r2-b` (cartesian) |
| `R[1..4,7..10]` | `R1 … R4 R7 … R10` (comma-separated segments) |

A spec that expands to nothing (`r[,]`) is reported: it would otherwise take
the line and everything indented under it away without a trace.

A mistake in a line's attributes is reported once for the line, naming what
was written (`"R[01..40]"`) rather than once for each of the forty racks it
became.

Segments are how numbering with holes stays on one line: racks 1–4 and 7–10
around a gap are `rack R[1..4,7..10]`, with the children written once instead
of once per block — see `examples/three-rows.dc`, which also pins sparse
U-slots the same way (`node [7..15x2,25..31x2] id=u{id} at={id}`).

### Placeholders

`{placeholders}` in attributes refer to enclosing elements:
`name="Hall {id}"`, `power=grid-{i}`, `{room}`, `{row}`, `{parent}`.
A kind names the nearest element of that kind, and on that kind's own line
the nearest is the element itself — so `{dcm}` on a `dcm` line is its own
id, the same as `{id}`: `dcm [1,2] name=dcm-{dcm}` names `dcm-1` and
`dcm-2`. Nothing is inserted between the text and the id, so the hyphen is
yours to write (`name=dcm{dcm}` is `dcm1`). The built-in names — `{id}`,
`{i}`, `{i0}`, `{n}`, `{seq}`, `{kind}`, `{parent}`, `{path}` — always mean
what they say here, even under a kind of the same name.

This is how flat hostname-style names work: `node u[01..40]
name={room}{rack}{id}` names every server like `wr12r06u15`, results files
can then target that name directly, and the row stays expressed by nesting
without appearing in the name — see `examples/hostnames.dc`.

Any whole-number placeholder takes a width: `{i:3}` is `001` and `{seq:2}`
is `07`, padded as a range written `R[01..20]` is. A value that is not a
number is left as it is.

A placeholder naming something not in scope is reported rather than left in
the text — `{rak}` for `{rack}`, or `{id}` in an id spec, where the id does
not exist yet — and the warning lists the names that line could have used.

### Numbering through the floor: `{seq}`

`{i}` starts again under every parent; `{seq}` carries on, counting every
element the line has made so far across all the copies of it. So racks
numbered uniquely across a floor are one block, not one per row:

```
row [1..4]
  rack [1..5] id=R{seq:2} u=42          # row 1: R01-R05, row 2: R06-R10, ...
    node u[01..40] role=server name={rack}{id}     # R07u15
```

`[1..5]` says how many racks each row has, `id=` gives each its number, and
`{seq:2}` pads it to two digits. The count keeps going however deep the
repetition goes: under `room [A|B]` the same rack line runs on to R40.
`examples/dual-plane.dc` numbers its pods' racks this way. `{seq}` belongs
on an attribute (`id=`, `name=`): an id spec is expanded before anything is
counted, so `rack R{seq}` is reported.

**Lines side by side share the count.** Lines of one kind in the same block
number as one, so a row can hold different racks in different places and
still be numbered straight through — here the middle four racks of each row
are network racks:

```
row [1..4]
  rack [1..8] id=R{seq:2} u=42              # R01-R08
    node u[01..40] role=server
  rack [1..4] id=R{seq:2} u=42 +network     # R09-R12, the middle four
    node agg[1..4] u=2 role=agg
  rack [1..8] id=R{seq:2} u=42              # R13-R20; row 2 is R21-R40
    node u[01..40] role=server
```

Only lines that use `{seq}` take a number — a `node tor` line beside a
numbered server line does not use one up.

**Starting again in every room.** `seq=room` on a line gives it one count
per room: racks numbered R01–R80 in each room, however many dcms or systems
repeat the rooms, the way `scope=room` groups a link rule. Any enclosing kind
works — `seq=row`, `seq=dcm` — and what is counted is kept per kind, so racks
and servers numbered per room keep two counts. Lines sharing a `seq=room`
share the count within the room, so a row split into three rack lines still
runs in order:

```
dcm [1..2] name=dcm-{dcm}
  room [1..2] name="Room {room}"
    row [1..4]
      rack [1..8] id=R{seq:2} seq=room u=42           # R01-R08 ... in every room
      rack [1..4] id=R{seq:2} seq=room u=42 +special  # R09-R12
      rack [1..8] id=R{seq:2} seq=room u=42           # R13-R20, then R21- in row 2
```

Any other word names a count instead: `seq=net` keeps network racks
numbered N1, N2, … apart from the racks beside them, and lines anywhere with
the same name share one count.

### Lines that apply only in some places: `if=`

`if=` on a line makes its elements only where the condition holds, so one
block for four rows can still give the first row something the others do
not have — here rack 12 is filled in row 1 and stands empty in rows 2–4:

```
row [1..4]
  rack [1..11] id=R{seq:2} seq=room u=42
    node u[01..40] role=server
  rack 12 id=R{seq:2} seq=room u=42 if={row}=1      # row 1: R12, filled
    node g[01..10] u=4 role=server +gpu
  rack 12 id=R{seq:2} seq=room u=42 if={row}!=1     # rows 2-4: R32, R52, R72, empty
  rack [1..8] id=R{seq:2} seq=room u=42
    node u[01..40] role=server
```

A condition is `{placeholder}=value` or `!=`, and several joined by commas
must all hold (`if={room}=1,{row}=1` is only the first row of room 1). A
value may list alternatives (`{row}=1|3`) and use `*` and `?` as a selector
does; matching ignores case. An element not made gives its `{seq}` number
back, so the numbers stay in order, and anything indented under it is not
made either. A condition that cannot be read is reported and ignored, and
`if=` is not kept among the attributes of what it made.

### Attributes and tags

- `key=value` attributes are free-form and **inherit** downward (children see
  the parent's `region=us-east` unless they override it). Layout-only keys
  — `id`, `name`, `u`, `at`, `cols`, `dir`, `gap`, `align`, `seq`, `if` — do
  not inherit.
- Attributes the viewer does not draw are still worth writing: they show in
  the inspector, and other tools read the same file. `nic_gbps=` on servers
  and `uplinks=` / `uplink_gbps=` on racks are the hardware binnacle's
  [`reckon`](#reckon--binnacle-grades-a-run-against-the-hardware) grades a
  test run against — one `nic_gbps=25` on a row covers every server in it.
  What the viewer itself draws as capacity is `gbps=` on a net or a link
  rule — see [Capacity](#capacity-what-a-cable-carries).
- `+tag` tags are free-form keywords; every element can carry many, and every
  element also carries its ancestors' tags. `+a,b,c` adds three at once. One
  tag means something to the viewer: [cable load](#cable-load-where-a-flow-metric-lands)
  forwards traffic only through elements tagged `+switch`.
- Rack children: `u=4` gives a node 4 U of height, `at=42` pins it to a slot;
  unplaced children auto-fill the lowest free run of slots. A node that lands
  above the rack's declared `u=` height is reported as a warning — `at=42`
  only fits a rack at least 42 U tall. A rack with nothing in it is drawn as
  an empty rack, as tall as its slots, rather than as a collapsed stub.
- **`u=` is an element's height, in U, everywhere** — the scale racks are
  drawn to, where one U is the height of a 1U server. On a rack it is how
  many slots the rack has, and on a rack's child how many it fills, as
  above. Outside a rack, a device is exactly that tall (`node pdu u=4` in a
  cage) and a container is *at least* that tall (`cage C u=60`): what its
  children need always fits. A row made taller than its racks keeps them
  standing on the floor, with the room above. A collapsed container keeps
  its compact size whatever its `u=`.
- **`w=` is its width, in U, on the same scale** — what `u=` is to height:
  a device outside a rack is exactly that wide (`node PANEL w=96 u=8`, a
  patch panel as wide as the racks it serves), and a container is *at
  least* that wide. A rack and what is
  in it keep the rack's width, and a collapsed container its compact size.
  It is not inherited.
- `cols=2` / `dir=x|y` / `gap=` shape generic containers — `gap=0` packs
  children with no gutter, and omitting it keeps the per-kind default.
- `align=center` (or `right`; `left` is the default) places each line of a
  container's children across the room it has — a narrow network layer
  centred over the racks below it. An `align=` that is not one of those is
  reported and laid out from the left.
- A **row** lays its racks out in one line, until it says otherwise:
  `cols=` or `dir=y` makes it a grid like any other container. That is how a
  row carries a network layer above its racks — `cols=1` stacks the two, and
  `dir=x` on each keeps its own contents in one line:

  ```
  row [1..4] name="Row {row}" cols=1 align=center
    network dir=x
      spine [1..4] name=spine-{spine} role=spine
    servers dir=x
      rack [1..20] id=R{seq:2} u=42
        node u[01..40] role=server
  ```
- The numbers are checked: `u`, `w`, `at`, `size` and `cols` must be whole and
  between 1 and 1000 (`gap` from 0), and a net's `width` between 0.1 and 100. Anything else
  is reported and ignored rather than quietly coerced — `u=1e9` used to read
  as 1 and `u=-5` drew a rack of negative height.

### Networks

Logical fabrics, drawn as differently colored/styled cable layers. Rules match
elements by selector, so you never enumerate cables:

```
net data    label="Data / east-west"  color=#4fa3ff
net mgmt    label="Out-of-band mgmt"  color=#8b93a7 style=dashed

link data  role=server role=tor   scope=rack           # star, per rack
link data  role=tor    role=spine scope=hall           # every ToR to every spine
link storage +storage,role=server scope=row mode=mesh  # full mesh within a row
```

- Selectors: `+tag`, `^tag` (non-inherited), `kind=rack`, `attr=value`,
  bare glob against id/path/ancestors, `!` negation, `|` alternatives,
  `,` for AND.
- `color=` takes any CSS colour — hex (`#4fa3ff`, `#fff`), a name (`teal`),
  or a function (`rgb(1,2,3)`, quoted if it contains a space). One the browser
  cannot read is reported and dropped: the canvas ignores a colour it cannot
  parse and silently keeps the last one, which would paint the net in the
  previous net's colour.
- **Globs work in every part of a selector**, tags included: `*` stands for
  any run of characters and `?` for exactly one. `model=r76*` takes r760 and
  r7625 alike, `model=r762?` only the five-character one, `u1?` the slots
  u10–u19, and `+stor*` any tag starting `stor`.
- `scope=` groups matches per rack/row/room/… before wiring.
- `mode=` is `star` (A×B, default with two selectors), `mesh` (default with
  one), `chain`, `ring`, or `pair` (A[i] to B[i]; with one selector,
  consecutive matches pair off — 1st–2nd, 3rd–4th, …). `pair` between two
  selectors of different lengths stops at the shorter one and reports how
  many were left unwired — two spines paired with eight ToRs is two cables
  and six racks with no uplink, which `mode=star` is almost always what was
  meant. With one selector an odd count leaves one over, which is inherent
  and says nothing.
- `cap=` stops a rule after that many cables, which is how a selector that
  turns out to match half the floor is kept from wiring all of it. A cap that
  is not a whole number between 1 and 100,000,000 is reported and ignored, and
  a rule that hits its cap says so.
- `splice=N` gathers every N consecutive matches of the first selector into
  one cable to each element of the second — a breakout harness, four servers
  spliced into one cable to a ToR port. See [Splices](#splices) below.
- A rule wires **two** selectors. A third token is reported rather than
  dropped, which is what catches a mistyped option — `scoope=rack` cannot be
  told apart from selecting on an attribute called `scoope`, but it can be
  counted.
- A rule that wires nothing says so: a selector that matched no elements (the
  shape a typo makes) and a rule whose matches produced no cables are both
  reported as warnings rather than left as a silently empty fabric. A
  `scope=` that names no kind in the layout says which kinds there are
  (`scope=rw` — did you mean `scope=row`?), and one that keeps the two sides
  apart — ToRs in the rack rows, spines in a row of their own, `scope=row` —
  says so and names the scope that holds both, or none.
- A `net` line takes `label=`, `color=`, `style=`, `width=`, `show=`, `gbps=`
  and `traffic=`; anything else is reported with the setting it was most
  likely meant to be (`gpbs=` — did you mean `gbps=`?). A quote left open
  anywhere is reported too: it runs to the end of the line, so
  `"label="x" gbps=400` is a label reading `x gbps=400` and no capacity.
- Declared nets **start visible** on a modest floor (up to 20,000 cables in
  total), so wiring something draws something. Past that they start unticked —
  a hyperscale floor's first render should be the floor. `show=` on a `net`
  line (or `on=`) settles it either way, and takes any spelling of yes or no —
  `yes` `y` `on` `1` `true`, `no` `n` `off` `0` `false`. Anything else warns
  and leaves the decision to the size rule.

Networks toggle on and off in the UI, and modest fabrics start on (see
above). Collapsed or zoomed-out regions merge their cables into one thicker
line; very dense views fade automatically.

### How cables are drawn

A cable leaves its device from the **left or right edge**, never the middle,
steps out into a **lane** running beside the devices, and follows it — the
way cables run up a rack's cable manager. Everything in one rack (or any one
container) puts a given net on the same side at the same distance, so a
rack's twenty server-to-ToR cables share one line beside the servers, with a
short stub from each server into it, rather than twenty lines fanning across
the rack.

When a device carries **several nets, they leave from opposite sides**: data
on the right and mgmt on the left, say, so the two never draw on top of each
other. A third net takes a lane of its own further out, on whichever side has
fewer. The choice is made per container — each net goes to the side its
devices have fewer nets on — and racks that carry the same nets make the same
choice, so an uplink leaves its ToR on the side the spine's own lane is on.

Two devices in the same column of the same container (a server and its ToR)
are joined along their shared lane. Anything further apart steps out into its
lane at each end and crosses between the two **as a curve**: it leaves each
device sideways, the way that device's lane runs, and bends round to arrive
at the other end the same way. The cables sharing a port are **fanned out**
along its lane — ordered by where their other end is, a little apart, within
the device's height — so forty ToR uplinks arriving at a spine no longer meet
in one spot. **curved cables** in the Networks panel switches the crossings
back to straight lines (they stay fanned out), and a view dense enough to
fade to a haze — more than 10,000 cables — draws them straight anyway, since
there a curve costs half as much again to draw and cannot be told from a
line.

**A bus is met along its length.** `cables=bus` on an element — a long,
thin one, drawn across the devices it serves with `w=` — makes every cable
arrive at its top or bottom edge, whichever faces the device at the other
end, square to it and at the point nearest that device: the racks under an
overhead busway drop straight onto it where they stand, instead of all
meeting it at one end. The device end still leaves by
its side and lane. Two buses join edge to edge where they overlap.
`cables=sides` is the default, and anything else is reported.

```
node BUSWAY w=200 cables=bus +switch      # an overhead busway along the row
node PDU[1..8] role=pdu                   # each rack's power, under it
link power name=BUSWAY role=pdu
```

### Splices

In a rack of sixteen servers, every run of four is often cabled into a
harness that splices the four into a single cable, and that one cable runs to
the ToR. `splice=N` on a link rule says so:

```
link data role=server role=tor scope=rack splice=4
```

Each run of N servers gets a **slim marker** beside it, in the net's own
colour, with the N stubs drawn coming together inside it; **one heavier
cable** runs on from the marker to the ToR. The marker is part of the cable
layer — it has no label, takes no rack space, and toggles with its net — so it
never reads as a device.

- Runs are **consecutive in the rack, from the bottom** — U1–4, U5–8, … — in U
  order whatever order the lines were written in, and **never cross from one
  rack into the next**, even with a wider `scope=`. Eighteen servers make four
  runs of four and one of two; a run of one is simply a cable.
- Every server is **still wired to its ToR**: the links are the same ones the
  rule makes without `splice=`, and each carries the splice it runs through.
  Link counts, the inspector, and *show only this element's cables* work as
  before — isolating one server shows its own cable into the marker and on to
  the ToR, and none of the other three on the harness.
- The inspector names a server's splice (`data: 4 into one cable to tor`) and
  counts the splices arriving at a ToR or below a container
  (`data ×4 (16 cables into 4)`).
- A splice gathers the first selector's matches onto cables to the second, so
  it needs **two selectors and `mode=star`**; anywhere else it is reported and
  ignored. N is a whole number from 2 to 1000.
- Collapse the rack and its splices are inside it. A harness to a ToR in
  another rack folds into the collapsed rack's one thicker cable, as unspliced
  cables do.

`examples/splice.dc` is a small floor wired this way.

### Capacity: what a cable carries

`gbps=` says how much a cable can carry, in Gb/s. On a `net` line it covers
every cable in the net; on a `link` rule it covers that rule's cables and wins
over the net's:

```
net nica    gbps=25                            # 25G NICs
net fabric  gbps=100                           # 100G above
net mgmt    gbps=1 traffic=no                  # drawn, but carries no load
link nica   role=server role=tor scope=rack splice=4
link fabric role=tor role=spine
link fabric role=tor role=spine,model=new* gbps=400   # these uplinks are 400G
```

- Fractions are fine: a 100 Mb/s console port is `gbps=0.1`. A value that is
  not a number from 0.001 to 1,000,000 is reported and ignored.
- A cable with no `gbps=` from its rule or its net has **no capacity**, which
  is not zero: the views below say so rather than painting it full.
- A splice is a breakout: four 25G cables into one marker are a 100G trunk.
- Parallel cables are rules written twice (as in `examples/dual-plane.dc`,
  four cables from every spine to every superspine); each carries its own
  `gbps=`, and they add up.

Where the capacity shows:

- **The Networks panel** puts each net's speed beside its name (`25G`, or
  `25G–400G` where rules differ), and marks a `traffic=no` net.
- **The inspector** gives every element's cables with what they carry: a
  server's `nica ×1 · 25G each`; a rack's `nica ×8 (all internal) · 200G
  inside` against `fabric ×4 (all leaving) · 400G leaving` — its
  oversubscription, at a glance.
- **Between picks**, each route says the most the two could move over it —
  the capacity of its narrowest hop, all that hop's cables together (`data: 4
  hops · up to 25Gb/s` for two servers; a rack's uplinks for two racks).
- **width by capacity** in the Networks panel draws every cable as wide as
  what it carries, on a scale where each step of port speed — 1G, 10G, 25G,
  100G, 400G — is one step wider, with a key of the speeds in the layout. A
  lane beside a rack is as wide as the fastest cable in it.

### Cable load: where a flow metric lands

A flow metric (samples tagged `peer=` — what `iperf_orchestrator`, `mx export
--peers` and any `a -> b` table write) says how much one host sent to another.
Tick **load the cables** on its card and the viewer works out which cables
that traffic crossed and draws each one by what it carries. (Greyed out, it
says why under it: a percentage or a latency does not add up along a path.)

- **Width** is the traffic through that piece of cable. A rack's lane
  thickens on its way to the ToR, as it gathers its servers' cables.
- **Colour** is how full the fullest cable at that point is: its busier
  direction against its `gbps=`, green through yellow to red, and **magenta
  for a full cable** — one holding back the flows that cross it (see
  [Contention](#contention-a-full-cable-shares-itself-out)). Cables carrying
  traffic with no `gbps=` are grey; idle ones stay, faint, in their net's
  colour. With no capacity anywhere, colour is the load against the busiest
  cable.
- **The panel** names the busiest cables (`R02/torb → R02/s5 25G/25G · 100%
  · 8 held`; click one to go to it) — **Show more** lists more of them, 25 at
  a time, and **Show all** every one (up to 10,000; past that it says so) —
  and the busiest switches, how many flows were routed, and any
  that could not be and why. **The inspector** gives the selected element's
  traffic out and in against its capacity, per net — for a rack, over its
  uplinks — and its busiest cables.
- **Expected**, ticked in the inspector's Cable load, shows in place of the
  selected element's own figures the average of every element of its class
  that sits where it does, in every rack, row, room … (the kinds of
  container it is in; the box says how many each finds): the tor b of the
  first rack of every row, the third server of every rack. One draw of
  random peers can give a ToR 400 or 600 where one like it carries 587 on
  average; this is that average. Only the report changes — the load and how
  it is routed are as set. Of one class means the same kind and `role=`, or
  the same `model=` where roles are named per room.

How a flow is routed, which is a model, not a measurement — the measurement
is end to end, and only switch counters see the cables themselves:

- Each flow takes the **shortest routes**, in hops, from the sample's target
  to its peer: the target is the sender. Over `examples/dual-plane.dc` that is
  NIC, ToR, spine, plane and back down.
- **Only switches forward.** A server with two NICs is not a path between its
  two ToRs. Switches are the elements tagged `+switch`; a layout that tags
  nothing that way forwards through everything, and the panel says which. A
  flow with no way through switches — between servers cabled only to each
  other in a ring — is listed as not routed; tag the hosts that do forward
  `+switch` and it goes.
- **`traffic=no`** on a net keeps it out: out-of-band mgmt beside the data
  fabric would otherwise take an equal share of every flow it ties with.
- Where several cables continue a shortest route the flow is **shared**, by
  the panel's choice. The model tries to do what a real fabric does, so the
  choices are what real switches are set to do, and the default is what
  nearly every Ethernet fabric does out of the box:
  - **ECMP: hashed per connection** (the default) — each connection takes
    *one* cable at every fork, picked by a hash of its 5-tuple with a seed
    of each switch's own, as Arista, Cisco, Juniper and SONiC switches do. A
    handful of big flows can land on one uplink and overload it while its
    neighbour idles. **Re-roll** hashes again with another seed: the spread
    between rolls is the luck of the hash.
  - **ECMP, hashed alike at every tier (polarized)** — the same hash at
    every switch, as switches left on one factory seed (or one config
    copied to every tier) do: flows that collide on a ToR's uplink collide
    again on the spine's, so the hot spots line up tier after tier. A
    misconfiguration to test for, and one of the first things to suspect
    when superspines flood while the average looks fine.
  - **adaptive routing (per packet)** — InfiniBand adaptive routing,
    Spectrum-X, a switch's dynamic load balancing: each packet goes out of
    the emptier port, so every switch leans on its emptier cables until its
    choices are about as full as each other. A switch sees only its own
    cables, though: congestion further on is avoided only where an earlier
    choice leads round it, so with a ToR's uplink out the spine left
    carrying its traffic is no better off than under an even split. With
    one uplink degraded to 40G in `examples/dual-plane.dc`, all-to-all at
    40% fills the busiest uplink to 44% where an even split fills it to 77%.
    A host does not route adaptively: a bonded one still hashes.
  - **weighted by capacity (WCMP)** — shares in proportion to `gbps=`, as
    weighted ECMP (UCMP, the BGP link-bandwidth community) spreads over
    unequal links;
  - **packet spraying (even split)** — every flow's packets sprayed evenly
    over every shortest route, round-robin or at random, whatever the load:
    what a leaf that sprays does, its spines and superspines keeping to the
    route each packet was given. It needs NICs that put packets back in order,
    as RDMA NICs do. It is also the long-run average of hashing with endless
    connections, and the best any balancing could do on an even fabric: if a
    cable is full here, nothing will save it. Where spines also move traffic
    off a busy route, adaptive routing is the closer model; where they move
    it off a dead one, take that out with the what-if.
- **connections a flow**, beside the routing, is how many connections each
  flow between two hosts is made of — `iperf -P`, a test's parallel streams,
  an RDMA job's queue pairs — **1 by default**. Each is hashed on its own,
  so this is the biggest single thing a hashed fabric's outcome turns on:
  in `examples/dual-plane.dc`, every server sending half its NICs to the
  one sixteen places on (the **shift** pattern) fills the busiest uplink to
  125% at one connection a flow, 75% at four, 63% at thirty-two, and 50%
  split evenly. Under contention each connection is its own share of a full
  cable, as TCP shares it: a flow of eight beside a flow of one gets eight
  shares, and a flow is held back only on the connections whose own path is
  full. One connection a flow is also what a hash on the IP addresses alone
  does, however many connections there are.
- **`weight=`** on a link rule scales its cables' share in every mode —
  `weight=3` takes three times what a weight-1 cable beside it does.
- **`count=`** on a link rule makes that many cables between each pair — a
  bundle's members: `link plane1 role=spine role=superspine count=6
  gbps=400` is six 400G members per pair, drawn as one heavier line. Hashed,
  a flow lands on one member, so one 400G member can be full while the
  bundle reads 60% — what one 2.4T cable cannot show.
- **`bond=`** on nets bonds a host's cables on them: `net nica gbps=100
  bond=nic` and `net nicb gbps=100 bond=nic` make every server's NIC a and
  NIC b one bond, run as LACP with the layer3+4 hash: **each connection
  keeps to one NIC**, picked by its own hash in proportion to the NICs'
  capacity — under hashed and adaptive routing alike, since a host's bond
  hashes whatever its switches do. With one connection a flow two NICs can
  come out a flow or so apart; with many they even out, and under the ideal
  even split every flow is shared over the bond exactly. A connection
  arrives on the same side it left, NIC a to NIC a, so in one rack it never
  leaves the ToR. **`balance=even`** on the bond's nets (`net nica gbps=100
  bond=nic balance=even`) shares every flow over the bond by capacity instead,
  under every routing — for a bond whose NICs carry the same whatever the
  connections, as round-robin bonding and some NIC drivers do; `balance=hash`
  is the default. Take a ToR or a NIC's cable out and the rest of the bond
  carries everything. Only hosts bond: a switch's cables on the same nets
  are what the hosts hang off. A host with cables in two bonds uses the
  bigger.
- A pair's samples are reduced by the card's own aggregation (the mean of a
  run's passes, by default). The metric must be a rate (`unit=Gb/s`, `Mb/s`,
  `MB/s`, `pps`, … — or a name like `iperf_mbps_out`) or a volume; a bit or
  byte rate is set against capacity, anything else loads without one. A
  percentage or a latency does not add up along a path, and is not offered.
- Loads are kept per direction: a cable is full duplex, so 20 Gb/s each way
  on a 25G cable is 80%, not 160%.

`examples/dual-plane-flows.tsv` loads `examples/dual-plane.dc`: one rack's
servers sending 40 Gb/s each across the planes — 80% of their NICs and their
rack's uplinks — and an incast of eight servers into one, 160% of the two
NICs it lands on. Hashed, each 40 Gb/s flow takes one NIC, not two; under
the even split it is shared over both.

#### Contention: a full cable shares itself out

A cable cannot carry more than its line rate. What the flows *offer* — what
their senders want — can add up to more, and what happens then is what
happens on a real fabric: the cable's queue fills, packets are dropped (or
marked, with ECN), and every TCP sender crossing it slows down until they
share it. **contention: full cables share out**, on by default, models that
steady state:

- Every flow is raised from nothing at the same pace until it has all it
  asked for, or a cable on its way is full; there it stops, at its share of
  that cable. This is **max-min fairness**, the textbook idealization of TCP
  flows sharing a bottleneck: no flow can get more without taking it from
  one that has less.
- A flow held back at one full cable **carries less on every other cable
  along its path too**. In `examples/dual-plane-flows.tsv`, eight servers
  sending 10 Gb/s each into one server's two 25G NICs each get 6.25 Gb/s,
  and that is all that crosses their own cables and the uplinks on the way —
  not the 10 they offered.
- **Full cables are magenta**, and say how many flows they hold back
  (`25G/25G · 100% · 8 held`). The panel gives what contention cost:
  `contention: 2 cables full, holding back 8 of 17 flows · 385Gb/s carried
  of 415Gb/s offered (93%)`. Hovering a cable gives what it carries each way
  *of* what was offered, and each flow over it, held back or not.
- The **split** matters more with contention. **Even** shares every flow
  over every route, so a flow is held to its slowest one; **hashed** puts
  each connection on one path, as ECMP does with TCP, so only the
  connections hashed onto a full cable slow down — and two big flows hashed
  onto one uplink can leave its neighbour idle.

Untick it to see what the flows offer: then a cable can read past 100%, in
magenta, which says how oversubscribed it is rather than what it carries.

**bursts of × … for … µs** says how bursty the load is. The model is a
steady average, and real traffic comes in clumps — a synchronised start, an
incast all answering at once — that last microseconds and overflow a
cable while the second's average looks easy. Above 1, the panel counts the
cables a burst that many times the average would overflow, and what becomes
of each:

- What a cable cannot take at once **queues at the switch sending onto it**,
  in its shared packet buffer, `buffer=` (`buffer=132MB`). Every port
  overflowing at that switch shares it as a switch's dynamic threshold
  shares it: with one port congested it may fill half the buffer, with n of
  them 1/(n+1) each. A burst whose excess over its length fits that share is
  **absorbed**; one that does not **drops**.
- All-to-all at 50% over `examples/dual-plane.dc` with `buffer=2MB` on the
  ToRs, in 2.5× bursts: each 25G NIC is 6.25 Gb/s over for the burst, 78 KB
  in 100 µs, against the 222 KB its ToR gives each of its eight overflowing
  ports — `64 absorbed by their switch’s buffer`. At 1000 µs it is 781 KB,
  and `64 drop — more than their switch’s buffer holds`.
- A switch with no `buffer=` is counted apart (`at switches with no buffer=
  to say`), and a host sending more than its NIC takes queues in the host,
  dropping nothing on the wire.
- A cable's tooltip says what the burst would need of it, how much queues
  where, of how much: `78.1 KB queued at R03/tora, of the 222.2 KB its
  buffer gives this port: absorbed`.
- **What drops is lost**: the excess past the port's share, out of what the
  burst sends onto the cable while it lasts. The panel and the headline give
  it in all — the same load in 1000 µs bursts reads `a 2.5× burst for 1000
  µs drops 35.8 MB on 64 cables — 7% of what they carry while it lasts` —
  and a dropping cable's tooltip gives its own. It is a loss per burst, not
  a rate: how often bursts come is the traffic's, and TCP backs off after
  each, which the steady average does not see.

The buffer is the one figure that tells an incast a switch rides through
from one that loses packets; real switches add per-queue reserves and
priority queues the model leaves out, so read it as which bursts are close.

Sharing out costs in proportion to the cables every flow crosses, added up
over the flows. Flows that leave the same switches for the same server share
that part of the work, and the page shares out up to twenty million
crossings: a thousand servers each sending to 150 random peers over a
four-plane fabric is about nine million (fourteen with bonded NICs), and
takes a few seconds. Past that
the panel says **too many flows to share out here**, and draws what the flows
offer, as if contention were unticked.

The model is the steady state, and real fabrics are rougher. TCP favours
flows with shorter round trips; an incast of many senders can lose packets
all at once and collapse well below the fair share while senders wait out
their timeouts; UDP does not back off at all, so its excess is simply dropped
at the full queue, after it has crossed the cables before it; and a lossless
RoCE fabric drops nothing but pauses the senders upstream, which can spread
the congestion to flows that never crossed the full cable.

#### How full a switch is

With load on the cables, every element tagged `+switch` has a **gauge**
along the foot of its box: filled as far as it is busy, in the colour a cable
that full would be, magenta when it is full and holding flows back. Its
tooltip, the inspector and the panel's **Busiest switches** say how much
passes through it against what it can pass:

- With nothing more said, a switch can pass what its **ports** carry, one
  way — `switching 132.8Gb/s of the 500G its ports carry (27%)` — and it is
  full only where its ports are; the tooltip counts its full ports.
- **`gbps=` on a switch** is its own forwarding capacity, as switches are
  rated: 64 ports of 800G is `gbps=51200`. Where that is less than its ports
  add up to, the switch is oversubscribed — the tooltip says by how much —
  and contention shares it out like a cable: `switching 150Gb/s of the 150G
  it can forward (100%) · full — holding back 326 flows itself`.

```text
node spine1 role=spine +switch gbps=12800
```

- **`pps=` on a switch** is the packets a second it can forward. Small
  packets fill a switch long before its bandwidth — what a matrix test of
  small requests does to a fabric — and with a load that knows its packets
  (a packet rate with a size, mx's own or the generator's) contention holds
  flows back there too. Over `examples/dual-plane.dc` with its spines at
  `pps=2000000`, every server sending a million 64-byte requests a second
  out of its rack reads `passing 2 Mpps of the 2 Mpps it can (100%), of 13.3
  Mpps offered · out of packets — holding back 640 flows itself` with the
  spine's bandwidth all but idle. The busiest switches say `· 100% of its
  pps`, and the gauge fills by whichever is fuller.

```text
node superspine role=superspine +switch gbps=51200 pps=8000000000
```

- **`pps=` on a server** (or any host) is the packets a second it can send,
  and separately take in — what its NICs and network stack manage, which is
  often the real limit on a test of small packets. Contention holds its own
  flows back at that rate each way, before any switch is involved. Over
  `examples/dual-plane.dc` with `pps=500000` on every server and the same
  million 64-byte requests a second, a server's tooltip reads `packets:
  sending 500,000 pps, receiving 500,000 pps of the 500,000 pps it can each
  way (100%)` and `out of packets — holding back 48 flows itself`, and the
  panel's contention line says `32 hosts out of packets`. Like a switch's,
  it needs a load that knows its packets; one in bits alone leaves it be.

```text
node s[1..16] role=server pps=20000000
```

- **`gbps=` on a server** is what its NICs can move each way, whatever its
  cables add up to — a card in a PCIe slot narrower than its ports, two cards
  sharing one. Contention holds its own flows back there, as at a full
  cable, and its tooltip says `its NICs: 100% of the 20G they can move each
  way — full, holding back 62 flows itself`; the panel counts `N hosts full`.

A `gbps=` on a rack or a room is not passed down to what is in it.

#### Device models: a datasheet written once

A `model` line defines a device once, and every element with `model=` its
name takes its settings and tags — `u=`, `+switch`, `gbps=`, `pps=`,
`buffer=`, `latency=`, `ports=`, `breakout=` and anything else — wherever it
does not set its own. Names match in any case; a `model=` no `model` line
defines is kept as a note, and one a letter off a defined name is named as
a typo. The two Arista switches, from their datasheets:

```text
model 7050DX4-32S u=1 +switch gbps=12800 pps=5300000000 buffer=132MB latency=900ns ports=32x400,2x10,1x1 breakout=400:32,200:64,100:128,50:144,40:64,25:128,10:130
model 7060DX5-64S u=2 +switch gbps=25600 pps=10600000000 buffer=114MB latency=825ns ports=64x400,2x10,1x1 breakout=400:64,200:128,100:256,50:256,25:256,10:258

node leaf[1..8]  model=7050DX4-32S
node spine[1..4] model=7060DX5-64S
node leaf9 model=7050DX4-32S buffer=64MB      # its own setting wins
```

- **`gbps=` is one direction**: what the ports add up to, 32 × 400G = 12.8T.
  Datasheets also quote the two directions added (25.6 and 51.2 Tbps);
  that is not the figure. These two are non-blocking, so `gbps=` never
  binds; it is there for one that is oversubscribed.
- **`pps=` is the forwarding rate** — 5.3 and 10.6 billion packets a second.
  Both are rated with packets averaging over about 295 bytes: from there up
  they forward at line rate, and below it packets, not bits, are the limit.
  5.3 billion 64-byte requests a second, with framing, are about 5.5 Tb/s:
  a matrix run of small packets fills a 7050DX4-32S's packet rate at under
  half its bandwidth (see `pps=` above).
- **`ports=`** is what the device has, so many of so many Gb/s (`1x1` is
  the management port, for a `gbps=1` mgmt net). Its cables are checked
  against it: each takes a port at least as fast, and a fast port breaks out
  into several slower cables — a 400G port is four 100G cables, two 200G or
  eight 50G. **`breakout=`** is the datasheet's most ports of each speed,
  which is a limit of the chip as much as of the cages: a 7050DX4-32S breaks
  out to 128 × 100G but 144 × 50G, not 256. Too many cables, a cable faster
  than any port, or more of a speed than `breakout=` allows is a warning on
  the device's line, once however many devices the line made.
- **`buffer=`** is the shared packet buffer bursts queue in (see bursts,
  above), and **`latency=`** the switch's own time per packet (see
  [Between picks](#between-picks-only-their-flows-and-a-test-flow)).

A server's NICs are specified the same way, on the server: `ports=`,
`gbps=` each way, `pps=` each way and `latency=`. A Broadcom **BCM57508**
card in its 2 × 100G form (P2100G; 1 × 200G as the P1200G) sits on a PCIe
4.0 x16 slot, about 250 Gb/s each way before overhead — enough for its 200G.
Broadcom's DPDK reports measure it at about 106 million 64-byte packets a
second with no loss across both ports, which is the card's ceiling; a
kernel network stack is well below it, so measure your own servers for
`pps=` where the matrix runs use sockets:

```text
model P2100G ports=2x100 gbps=200 pps=106000000
node s[1..16] role=server model=P2100G
```

A server with two such cards has twice each figure: `ports=4x100
gbps=400 pps=212000000`, or a card a net if they are cabled apart.

A server is more than its NICs, and two of its own limits usually set what
a test reaches well before they do:

- **`stream=`** — the most one connection carries, in Gb/s. On a kernel
  stack a TCP stream is mostly one core's work, and tops out around 30–50
  Gb/s however empty the cables. It holds at either end, and a flow carries
  at most its connections (the routing box's *connections per flow*) times
  it: `stream=40` and four connections is 160.
- **`cpugbps=`** — what its CPU can send and receive together, in Gb/s:
  one budget for both directions, since the same cores do both. A server
  with `cpugbps=310` sends its 200 alone, but sending and receiving at once
  it manages 155 each way. **`rxcost=`** says how many times more receiving
  costs than sending (1 unless set): at `rxcost=1.3` it is about 135 each
  way.
  (`cpu=` stays free to name the part, `cpu=epyc-9654`, as layouts already
  use it.)
- **`cores=`** — how many cores it gives the network, `cores=32`. Each
  carries one connection at its `stream=`, so a server sends and receives
  at most cores × `stream=` together (receiving at its `rxcost=`, as
  below) — whichever of that and its `cpugbps=` is less. `stream=8.5
  cores=16` is 136 Gb/s in all: 68 each way, or all of it one way. With no
  `stream=` there is nothing for a core to carry, and it sets no ceiling.
- **`busy=`** — the share of its CPU other work takes, `busy=20%` (or a
  bare `busy=20`), off its `cpugbps=` (or its cores): `cpugbps=310 busy=20%` leaves 248 for
  the network. Write the idle figure once on the model and `busy=` on the
  servers or lines that are loaded. With no `cpugbps=` it has nothing to
  take from, and the tooltip says so. Measured is better than written:
  with servers that have a `cpugbps=`, the routing box offers **CPU busy
  elsewhere**, and choosing a loaded per-server CPU metric there — in per
  cent (`unit=%`), or unitless and named for the CPU (`cpu_util`,
  `busy`) with readings as shares — reads each server's share from it in
  place of its `busy=`. The metric must be ticked to count.

The routing box shows *connections a flow* whenever a server has a
`stream=`, packet spraying included: it sets what a flow may carry. The
**Load generator** has the same setting, always shown — **connections a
flow**, counted for each pair of hosts, not each server: a server holds
that many for every peer it sends to and every one sending to it. When
`stream=` holds generated flows below what they were asked, it says how
many, what one flow can carry at the connections set, and how many would
carry every flow in full, with a **Use N** button: 50 Gb/s from each server
over five peers is 10 Gb/s a flow, which `stream=8.5` holds to 8.5 with one
connection (42.5 from each server) and two carry whole. It is what the run
files write as iperf's `-P`, mx's `--streams` and netmesh's `flows=`.

Beside it, **cores per server** says how many cores the run may use on
each — blank for what each server's `cores=` says, or no limit. It caps
`cores=`, never raises it, and holds the way `cores=` does: cores ×
`stream=` for sending and receiving together. The generator says how many
cores the busiest server needs for what it sends and takes in — that
50 Gb/s each way at `stream=8.5` is twelve — and, when servers have fewer,
how many, with a **Use N** button. The run files give it to mx as
`--workers N`, one process a core.

So the three numbers are:

| Setting | Counted per | Holds | Run files |
|---|---|---|---|
| connections a flow | pair of hosts | `stream=` each | iperf `-P`, mx `--streams`, netmesh `flows=` |
| cores per server (or `cores=`) | server | cores × `stream=`, send and receive together | mx `--workers` |
| `cpugbps=` | server | the CPU's own budget, send and receive together | — |

Both hold flows back in [contention](#contention-a-full-cable-shares-itself-out)
just as a full cable does; hovering a server says which it is at, and the
headline counts *hosts out of CPU* and *hosts at their stream=*. Telling the
two apart is one test: an iperf run with `-P 1`, `-P 4` and `-P 8` gains with
each extra connection only where the limit is per connection.

```text
model P2100Gx2 ports=4x100 gbps=400 pps=212000000 stream=40 cpugbps=310
```

What the datasheets also give and the viewer does not model: power draw,
fans and power supplies, memory, table sizes (128-way ECMP, routes, MACs),
queues per port and QoS (PFC, ECN), MACsec, and the load-balancing features
the 7060X5 adds — dynamic load balancing is close to **adaptive routing**,
and Cluster Load Balancing places RDMA flows with a view of both leaf and
spine. Write any of them as attributes for the inspector.

#### Findings: what went wrong, and whether the load explains it

The **Findings** box under the cable load lists, by kind, every server
something went wrong on, and marks each on the floor plan with a corner — red
for the critical, amber for drops, magenta for contention. Its tooltip and
the inspector say what was found. Two sides feed it:

- **The files.** Any per-server metric that reports trouble: **drops**
  (`rx_drop_per_s`, `tcp_retrans_pct`, `loss_pct`, …), **soft lockups or hung
  tasks** (`soft_lockups`, `hung_tasks`), **kernel errors** (`edac_errors`,
  `mce_count`, `oom_kills`, …) and **failed checks** — any text metric reading
  `FAIL`, `CRITICAL`, `WARN` or `NO-DATA`, as `reckon_verdict` does. The kind
  is read from the metric's name; `finding=` on its `!test` line says it
  outright, or `finding=no` that a metric is not one. A zero, or an `OK`, is
  not a finding.
- **The load.** With load on the cables: servers **held back by
  contention** — sending less than they offer, or receiving less than is
  offered to them, an incast when many send to one — and **flows with no
  route**. With contention unticked, a server offered more than its NICs carry.

Side by side they answer the question binnacle's `reckon` asks with
`LOSS_BELOW_CAPACITY`: **is a server dropping packets because the load says it
must, or with room to spare?** A drop on a server whose traffic is held back is
*explained* — that is what a full cable does. A drop on a server whose cables
the load leaves short of full is *loss the load does not explain*, the one to
go and look at, and the box counts those first.

`examples/dual-plane-health.tsv` beside `examples/dual-plane-flows.tsv` shows
both: the incast target drops packets and the load explains it, one other
server drops with room, and two log soft lockups — one of them an EDAC error
and a failed check too. With the load generator, the same: run a load, and
the findings say which of the servers' troubles that load accounts for.

#### Between picks: only their flows, and a test flow

Picking two or more elements decides which cables are drawn — the routes
between them — but not which traffic is counted on those cables: every flow
in the metric that crosses them still adds to their width. With two or more
picks the picks box offers two things more:

- **count only the flows between the picks** loads the cables with just the
  metric's flows that start in one pick and end in another (either way; a
  rack picked holds everything in it). The panel says how many of the
  metric's flows that is. For per-host totals it keeps only the host pairs
  between the picks, each at the share it has in the whole estimate.
- **test flow [N] [unit] 1 → 2 · Send** routes N from the first pick to the
  second and draws it like any metric — with nothing loaded, on its own and
  in Gb/s, Mb/s or pps; with a metric loaded, on top of it and in its unit,
  so you can see whether one more flow fits. A rack picked sends (or
  receives) it shared evenly over its devices. **Stop**, Esc or Clear ends
  it.

A single test flow, hashed as the page routes it by default, takes one
uplink, so every cable from server to server is drawn the same width; the
even split (packet spraying) shares it over every uplink — four spines, a quarter each.

Between two devices the picks box also gives the **latency** floor of one
packet's trip, at 64 and 1500 bytes, over every shortest route: each
switch's `latency=`, and each cable's time to send the packet — once for a
run of cut-through hops, and again wherever a switch must hold the whole
packet first because the cable out is faster than the cable in. Over
`examples/dual-plane.dc` with `latency=900ns` on its switches, two servers
in one rack are `942 ns at 64 B, 1.4 µs at 1500 B · 1 switch`; in two racks,
`2.78 µs at 64 B, 3.7 µs at 1500 B · 3 switches` — the ToR stores each packet
where 25G steps up to 100G. A `latency=` on either host — its NIC and
stack — is added too. Not counted: the cables' length (about 5 ns a metre);
a switch with no `latency=` is counted as none, and said so.

With a load on the cables, a second line gives the trip **under this
load**: each cable's queue added. A cable a fraction ρ full holds a packet
about ρ/(1−ρ) of the load's packet times — the single queue's textbook
figure, 9 at 90% and 99 at 99% — and a full one as long as its port's share
of the sending switch's `buffer=` takes to drain, which is also the most
any queue can hold it. The load's packets are its own size where it says
one, else IMIX. Over the dual-plane fabric with `buffer=2MB` on its ToRs,
two racks apart, a load of 150% shared out fills the ToR's downlink, and a
1500-byte packet takes about 75 µs where it took 3.7 — 71 of them waiting
for 222 KB to drain at 25G. A full cable at a switch with no `buffer=` is
said to have no ceiling known, and a host's own full NIC queues in the host,
which is not counted.

#### The load generator: a load with no file

The **Load generator** box under the Networks panel puts a made-up load on
the cables — nothing to measure first, nothing to load. **Run** routes it and
draws it like a metric; **Stop**, or **Off** in the cable load panel, takes it
off.

- **The pattern:**
  - **all-to-all** — every host sends to every other host, an equal share
    each;
  - **to random hosts** — every host sends to `×N` hosts picked at random,
    an equal share each, **and is sent to by exactly N**: the peers are drawn
    as N random pairings, each one in which every host sends to one host and
    receives from one, so no host is picked more often than another and every
    server's own cables carry the same. It is the way to load a fabric too
    big for all-to-all — a few thousand servers at 16 or 64 peers each — and
    the panel says how many peers fit when all-to-all is too many. The same
    seed picks the same peers; **New peers** draws them again. One draw
    can land a server's peers inside its own rack or not, so a switch's
    figures move from draw to draw: the inspector's **expected** shows what
    one like it carries on average (see *Cable load*, above);
  - **incast: all to one** — every host sends to one target, the first pick
    or else the selection: many to one, the pattern that fills a single
    server's cable. A rack as the target shares it over the rack's devices;
  - **one to many** — the target sends to every other host, as a storage
    server feeding its clients: its own cables fill outward;
  - **hotspot: a share to one** — every host sends a share of its load to
    the target (20% unless set) and the rest to ×N random peers: a few
    servers everyone calls, over a background;
  - **shift: each to one, k on** — every host sends everything to the host
    k places on, in the layout's order, and is sent to by one. With k empty,
    half the hosts on, so every flow crosses the middle of the fabric: the
    classic worst case for ECMP;
  - **ring: each to the next** — shift by one, the last to the first: how an
    all-reduce passes its data round.
- **each sends** how much every host sends in all: in Gb/s, Mb/s or pps, or
  **% of its NICs** — a share of what the host's own cables can carry, their
  `gbps=` added up, which is what it can send one way — which is the
  default once the cables have a `gbps=`. 50% of two 25G NICs is 25 Gb/s,
  so 50% and 25 Gb/s are the same load; the box says what the share comes
  to from each host.
- **in all** or **to each peer** — who that amount is for:
  - **in all**, the default, makes it the host's total, whatever the
    pattern: **more random peers spread the same load thinner** — at ×8
    each flow is an eighth of it, at ×150 a 150th — so the cables carry much
    the same at any count, and only the number of flows changes;
  - **to each peer** (**to each other host** for all-to-all, **to each
    target device** for incast) gives every flow the whole amount, so a host
    sends it times its peers and **every peer adds load**: 1 Gb/s to each of
    150 peers is 150 Gb/s from every server, past two 25G NICs, and
    contention shows where it gives. A share of the NICs works the same way:
    1% to each of 150 peers is 150% of them.

  Read off a metric, the amount is always the host's total: a reading is
  what the server sent in all.
- **requests of … B, replies of … B** — in pps, the packets' size: a made-up
  matrix_orchestrator run. Each host sends its packet rate as requests of
  that payload (plus 66 bytes of framing on the wire), and every request is
  answered by a reply of its own size, the other way. The load is then a bit
  rate — set against `gbps=`, shared out where full — and the headline gives
  the packets beside the bits. Over `examples/dual-plane.dc`, all-to-all at
  1,000,000 pps of 500 B requests and 4096 B replies is 32 Mpps each way and
  fills every server's NICs to 76%, almost all of it the replies coming back.
  Choosing pps starts at **IMIX** requests and no replies: 316 B of payload,
  the average of the standard 7:4:1 mix of 64, 594 and 1518-byte frames.
  Real datacentre traffic is two-humped — small acks and requests beside
  full-size data — and averages a few hundred bytes, and switches are rated
  for their packet rate at about this size; 1500 B would make a packet-rate
  limit look harmless. One average size is exact here: bits and packets both
  add up linearly, so a mix and its mean load the cables the same. **64 B ·
  IMIX · 1500 B** beside the box set each at a click, here and on a measured
  packet rate's own row. With the request size emptied, pps is plain
  packets, drawn by load alone.
- **to hosts** — **anywhere**; **within its own rack** (row, pod, room —
  the kinds in your layout), where every rack is a load of its own, its
  servers sending to each other in the pattern as if it were the whole
  floor — a test run rack by rack, or row by row — and nothing goes above
  that level; or **outside its own rack**, the other way round: every flow
  leaves the sender's rack, which loads the uplinks and nothing inside a
  rack. Explaining results uses the same choice, so set it to how the test
  ran.
- **hosts** — which devices send and receive, as a [filter](#the-viewer):
  `role=server`, `+gpu`, `role=server in:A[01..04]`, or several sections at
  once, `role=server (in:A01 or in:B[03..05])`; the box counts the hosts it
  takes as you type. Empty means every device with cables
  that is not tagged `+switch`, and a filter only narrows those: naming a
  rack takes its servers, never its ToRs. A layout that tags nothing
  `+switch` cannot tell a server from a switch, so the box says so and
  every device with cables counts.
- **split** — the hosts as groups that share none, and what they are for:
  peers only within each, and host lists to save. Splitting them:

  | Split | Groups |
  |---|---|
  | **each** rack (row, pod, …) | a container each |
  | **each pick** | each Ctrl-click pick: a rack, a row, a server |
  | **each rule** | a filter each, one a line or `;` apart: `in:A01 ; in:A02 ; rack=R[05..08]` |
  | **into blocks** of N | N runs of neighbours, in the layout's order |
  | **groups of** N | runs of N neighbours, the last what is left |
  | **interleaved** N | dealt a host at a time, so every group spans every rack |
  | **at random** N | dealt at random; **Reshuffle** deals again |
  | **by attribute** | a group for each value of one: `model`, `plane`, … |

  No host is ever in two groups: picks or rules that overlap — a server
  inside a picked rack, two rules that both hold `r1h1` — are refused,
  naming where, rather than one of them quietly winning, and nothing runs
  until they are fixed. Hosts no pick or rule holds are counted, and send
  nothing. The row says what the split comes to ("3 groups: g1 2 · g2 2 ·
  g3 2").
  - **peers only within each group** — each group a load of its own, its
    hosts sending to each other in the pattern (all, random peers, shift,
    ring) as if it were the whole floor: the disjoint sets of a test that
    runs several at once. *Within its own rack* is the same thing with the
    racks for groups; this is any groups. **to hosts**, outside its own
    rack, still holds inside a group.
  - **Save host lists** — `hosts-lists.tar`: a `hosts-<group>.txt` a group,
    a host a line as the load tools take one (`name`, or `name=addr`), and
    all of them in `hosts-grouped.txt` with a `[group]` header each — the
    grouped `hosts.txt` model_network's autopilot, iperf2 mesh and topology
    discovery read. Taken-out hosts are left out. Each list goes as it is to
    `--servers`, or into binnacle's `muster add` to be handed out a batch at
    a time.
  - **An area each** — every group an area of its own (below), with this
    area's settings and the group's hosts as its filter: a container's
    `in:/D/A/r1`, a pick's, a rule's, an attribute's `plane=2`, or the hosts
    themselves by path, `path=[D/A/r1/h1,D/A/r2/h1]`.
- **area** — several loads at once, each with settings of its own on hosts
  of its own: an all-to-all at 20 Gb/s in one pod while another runs an
  incast of 64-byte packets. **+ area** adds one like the area shown, with
  no hosts yet; the tabs show each, a name box names it (its files are
  named for it), and **Remove** takes it away. **Run** runs every area;
  the cables carry them all, shared out together, and the panel says what
  each sends ("A: 3 of 3 hosts send · 6 flows · 60G").

  **No host is in two areas — unless you say so.** Two areas whose filters
  take a host — including an area with no filter, which takes every host
  — are refused, naming the areas and a host they share, and nothing runs
  until each has hosts of its own. An incast's or outcast's target is the
  first pick (or the selection) inside its area and in no other, so no
  area reaches into another's hosts. Connections a flow and cores per
  server are the whole run's.

  **areas may share hosts** lets them: two things running on some hosts
  at once — a storage load on every server while one rack also takes an
  incast. A host in two areas runs both, and the model shares its CPU and
  NICs between them as it does any two flows; the row says how many hosts
  are shared. What the tools need for it is ports of their own and stopping
  only their own, so with hosts shared each area's run listens a hundred
  ports on from the one before (mx 5300, 5400, …; netmesh 5310, 5410, …;
  iperf 5001, 5101, …), the notes list them to open, and the Host status
  pane's preflight checks them all. mx's workers share a port by
  `SO_REUSEPORT`, so two mx runs on one port would quietly take each
  other's packets. Stopping by pid is the other half: **mx 1.11.2** and
  netmesh from the binnacle after 0.8.0 stop only their own agents, where
  older ones also kill every other agent on the host when they stop.
  iperf-orchestrator already stops its servers by port.

  The run files make each area a run of its own, all started at once, so
  no run reads, writes or cleans up after another — the tools' own
  options for runs side by side, nothing new on the hosts:

  | Tool | Each area's own |
  |---|---|
  | mx | `matrix-A.csv`, `--remote-dir /var/tmp/mx-A`, `--reports reports-A`, `mx_results-A.tsv` |
  | iperf-orchestrator | `iperf_plan-A.conf` (`remote_dir=` in it), `--run-id dcviz-A`, `iperf_results-A.tsv` |
  | netmesh | `mesh-A.csv`, `--remote-dir /var/tmp/netmesh-A`, `--reports reports-A`, `netmesh_results-A.tsv` |

  Each area has its own `expected-A.tsv`, and its run file, `mx-run-A.tsv`,
  carries the layout and says which area it was (`#~run area A`): load
  them together and they land side by side. The script checks every area,
  starts every area's run in the background, each to its own log
  (`mx-run-A.log`), waits for them all, and then exports and collects
  each.

- **Take out at random** — servers, racks and switches, each by a
  number or **% of the** ones there are, and each among all of its kind or
  those a filter names (`role=tor`, `role=compute`, `ht11r[01..04]*`); the
  row says what it comes to ("→ 41"). **Take out** draws them into the
  what-if beside anything taken out by hand — their cables carry nothing,
  the load goes round them, and the panel says what it cost — and the what-if
  shows them as one chip, "41 at random". **Draw again** takes out a fresh
  set in their place, and **Put them back** returns them, leaving what was
  taken out by hand. A rack is any rack with a cabled device in it, the
  switches' racks included; a server inside a drawn rack still counts as a
  server drawn only if it was drawn itself.
- **from** — an amount of the generator's own, or **what a metric says**:
  pretend the servers make the load they measured. Any per-host metric that
  is loaded and ticked can be picked: a rate (`egress_gbps`, `tx_mbps`) is
  read as what each server sends, a percentage (`unit=%`, a NIC
  utilization, say) as that share of the server's own NICs. **each sends**
  is then a share of the reading — 100% as measured, 200% for twice it — and
  the pattern still says where it goes. Servers with no reading send
  nothing, and are counted. (**load the cables (estimated)** on a rate
  metric's card is the other way to the same idea: every host's total sent to
  the others by the gravity model.)

With a metric loaded onto the cables, the generated load goes on top of it,
in a unit that adds up with the metric's (a share of the NICs only where the
metric is a bit rate). A test flow between picks goes on top of either.

Every pattern is routed like any flow metric, by the panel's split. Over
`examples/dual-plane.dc`, all-to-all at 50% fills every server's NICs to 50%
and the ToR uplinks to 39%; outside its own rack, every flow takes an uplink
and they fill to 50%. Generating is
quick, routing is the cost: every destination is a walk over the whole
fabric, so a load too big to route in the page is refused with why. That
walk is shared by every server behind the same switches, so it is done once
per ToR (or ToR pair), not once per server: random peers over four thousand
servers route in about a second. All-to-all grows with the square of the
servers — a thousand of them is a million flows, past what the page will
route — so above a few hundred, send to random hosts instead, with as many
peers as the panel says fit, or narrow the hosts.

#### Run it for real: the files and the commands

Under a running load, **Run it for real** hands over what each load tool
needs to make that same load on the real servers — its input files, to save,
and the commands that run it and export what it measured for this viewer.
Nothing is run from the page. Set how long the run lasts (60 s unless
changed) and the fabric's MTU (1500 or 9000), pick a tool, and save its
files into one directory:

| Tool | Files | What it is given |
|---|---|---|
| **iperf-orchestrator** ([iperf_orchestrator](https://github.com/MartinGallagher-code/iperf_orchestrator)) | `iperf_plan.conf` (one per rate), `hosts.map` | a pair grid of who sends to whom, `-b` each stream's share of the pair's payload, `-P` the routing box's *connections a flow* |
| **mx** ([matrix_orchestrator](https://github.com/MartinGallagher-code/matrix_orchestrator)) | `matrix.csv` | each pair's requests a second; a sized load's request and reply sizes as they are, a bit rate as full frames at the MTU — 1472-byte requests at 1500, 8972 at 9000, a sixth of the packets — answered by mx's smallest reply, 32 B; `--streams` the connections |
| **netmesh** ([binnacle](https://github.com/MartinGallagher-code/binnacle)) | `mesh.csv` | the load's pairs, probed ten times a second for latency, jitter, loss and path MTU |

The commands come with them, ready to copy — for mx, say:

```sh
mx check --matrix matrix.csv            # what each host will send and take in
mx run --matrix matrix.csv --for 60 --streams 1
mx export --matrix matrix.csv --peers -o mx_results.tsv
```

**Save run-mx.tar** puts all of it — the tool's files, `expected.tsv`,
the layout as `model.dc` and the script — in one archive, a folder of its
own, the script executable:

```sh
tar -xf run-mx.tar && cd run-mx && ./run-mx.sh
```

One download rather than four, so a browser cannot quietly save a second
run's matrix as `matrix (1).csv` beside the first one's `matrix.csv`.
**Save run-mx.sh** saves the script alone: every note
a comment, every command as it is, and a check that its files are beside
it before anything starts. Save it with the files, change what you like —
`--for`, `--user root`, a line commented out — and run `bash run-mx.sh`.
For iperf at more than one rate its runs go together, in the background,
each into its own log, and are waited for before the export; for netmesh
the run under the load is there commented, to swap in for the idle one.
Each tool's export lands back on the floor plan, because **the hosts go
into the files by the layout's own names**: load `mx_results.tsv`,
`iperf_results.tsv` or `netmesh_results.tsv` (made by `dcimport`) beside the
layout, and compare what ran with what the model said.

- **The NIC to load** goes in the box beside the MTU, as mx's and iperf's
  `--bind` take it: an interface name (`eth1`, `mlx5`) or part of an
  address (`10.20.`), matched against `ip -o -4 addr show` on each host,
  for a fleet reached over a management NIC whose traffic should ride
  another. It goes into the files — iperf's plans as `bind=`, which every
  command run against the plan reads; mx's and netmesh's runs as
  `--bind`, the mx run under netmesh too — and is kept for next time, so
  the files need no editing. netmesh reads it as a NIC from the binnacle
  after 0.8.0, which also points each host's probes at its peers' matching
  addresses; an older netmesh took only a whole address and refused a
  name (`cannot bind data:5310 -- Name or service not known`). Empty, there is none. A pattern a shell would
  expand (`eth[12]`) goes quoted; one with a space, a quote or shell syntax
  in it is refused, and the box says why.
- **A host the tools cannot reach by its name** gets an `addr=` — the
  address they reach it at, `node s7 role=server addr=10.1.7.15`, or one
  per server from the line that makes them, `addr=10.1.{rack}.{id}`. An
  address is one device's own, so a rack's `addr=` is not its servers'.
  mx and netmesh take it as `name=address`; iperf tests the address, and
  `hosts.map` turns it back into the name for the export.
- **Names must tell the hosts apart**, and be letters, digits, `.`, `_`
  and `-`: a layout whose racks all hold an `s1` cannot say which `s1` a
  result is for, so the panel asks for names first (`name={rack}{id}`).
- **The cables carry exactly what the model shows.** iperf paces and reports
  TCP *payload*, and every packet also carries 52 bytes of TCP/IP headers
  (timestamps on, as Linux sends them) and 38 of Ethernet framing — 1448
  bytes of payload in every 1538 on the wire at MTU 1500. So `-b` asks for
  that share of the model's rate: 941.5m for each 1 Gb/s on the cable at
  1500 (94.1%), 990m at 9000 (99.0%). The comparison knows it: an iperf
  metric is set against the model's payload, which `expected.tsv` carries
  as `model_payload_gbps`, so a run that did as asked reads 100%. A stream
  slower than about 100 Mb/s is written 10 ms at a time (`length=`, iperf's
  `-l`): iperf2's own 128 KiB writes are refused under about 1 Mb/s, and
  overshoot a short run above it. mx needs
  no such allowance: its packets are counted with their framing on both
  sides.
- **iperf's `-b` is one rate a run.** A load where every pair sends the
  same — all-to-all, random peers, a ring at one amount — is one plan. Pairs
  at different rates make a plan per rate (up to eight; more are binned),
  each on its own port so they run at once in separate terminals, and their
  exports go into one `iperf_results.tsv`. Runs at once each keep a remote
  directory of their own (`remote_dir=/tmp/iperf_orchestrator-dcviz-2` in
  the plan): iperf-orchestrator ends a run by removing its `--remote-dir` on
  every host, and a host in two rates' runs would otherwise lose the files
  of the one still running. iperf sends TCP, so a sized load's replies go
  back as TCP the other way at their own bit rate.
- **mx's workers are sized to the load, or to the cores.** A pair's
  connections are mx's `--streams` — its rate split over that many
  sockets — but what does the work on each host is `--workers`, separate
  processes, one a core, that each sustain about 200k packets a second of
  request and reply. With **cores per server** set, that is the number:
  the run line says `--workers N` (even at 8 or fewer, since `auto` takes
  whatever the box has), and the notes say if the load would want more.
  Without it, the files size them as `mx hints` does — twice the busiest
  host's requests a second (out, or in, which it answers) over 200k, plus
  one, and never fewer than the connections a flow — capped at the fewest
  `cores=` among the run's hosts, since mx runs one number everywhere; the
  notes say when the cap cut it. `--workers auto` runs one a core and stops
  at 8, which a full-frame load at tens of Gb/s outruns (50 Gb/s of
  1472-byte requests is 4 million a second), so past 8 the run line says
  `--workers N`, the mx run under netmesh too. iperf has no cores setting:
  `-P` is threads within each iperf and the kernel spreads them, so its
  notes count the connections the busiest host holds against the cores it
  has — fewer connections than cores leaves cores idle, and the notes say
  to raise connections a flow. netmesh is one light agent process a host,
  whatever the cores; they go to the mx run under it.
- **netmesh makes no load.** It measures the network: run alone it probes
  the idle network; `netmesh run --mesh mesh.csv --baseline 20 -- mx run …`
  probes 20 s idle and then under the load, which is the latency the
  model's queues predict. Of the generator's settings it takes only **which
  hosts talk to which** (the pattern, the hosts, the peers), **how long**,
  and **the connections**, as probe flows a pair (`flows=`). Every pair gets
  ten 64-byte probes a second whatever the load's rate: the rate, its unit
  and the request and reply sizes go only to the mx run under it. Nor is
  the MTU given to it — it finds each path's MTU itself — and the MTU box is
  what the model expects it to find. Its notes and `mesh.csv` say exactly
  this, rather than describing the load as if netmesh sent it.
- Flows to or from anything taken out are left out — nothing runs there —
  and the panel says how many.
- **`expected.tsv`** goes with every tool's files: the model's own figures
  for this load — what it expects each flow to carry (`model_gbps`, and
  `model_pps` for a packet tool), what each was asked to send
  (`model_offered_gbps`), where it holds one back (`model_held`), and what
  each host sends and takes in. The last command collects the run with it,
  stamped with when the run began and ended, the host it ran from and the
  tool's own `--version`:

  ```sh
  {
    cat expected.tsv
    echo "#~ran started ${started:-unknown}"      # set as the script begins
    echo "#~ran finished $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    echo "#~ran host $(hostname 2>/dev/null || uname -n)"
    echo "#~ran mx $(mx --version 2>&1 | head -n 1)"
    cat mx_results.tsv
  } > mx-run.tsv                                  # one file: the run and its model
  ```

#### The run carries its layout

A run is only as good as knowing what it was run against, so **the layout
goes with it**. The tar holds it as `model.dc` (kept with the run, never
needed to run it: the script does not wait for it), and `expected.tsv` —
so every `mx-run.tsv` made from it — carries it whole, as `#~` lines that
every reader of a results file skips as comments, with its sha256 and how
the load was set up:

```
#~run made 2026-10-28T14:02:11Z
#~run viewer datacenter_visualization 1.42.0
#~run tool mx
#~run layout floor.dc
#~run load the random pattern to 1 peer, 1000 pps from each host
#~run seconds 60
#~run connections 1
#~run mtu 1500
#~run bind eth1
#~run out DC1/A/r3
#~model sha256=9f2c… lines=212 name=floor.dc
#~ dc DC1
#~   row A
…
#~end model
#~ran started 2026-10-28T14:03:00Z
#~ran finished 2026-10-28T14:04:05Z
#~ran host jumpbox
#~ran mx mx 1.11.1
```

The sha256 is `sha256sum model.dc`'s, so a layout file on disk can be
checked against a run without the viewer.

**Load a run file and its layout comes with it.** Into an empty viewer it
simply loads. Over the very same layout nothing changes, and the report
says it matched. Over a different one the viewer **asks first** — the
layout there now, and any edit not saved, would go — and **Cancel** keeps
it and loads the readings onto it, with a warning that they may land on
elements other than the ones they were measured on. A layout loaded in
the same drop as the run file is your choice and stays. A carried layout
that no longer matches its sha256, or was cut short, is not loaded, and
the report says why. The report also gives each stamped file's line: made
when and by which version, ran when, where, and with which tool.

#### Measured against the model

Load a run's results — `mx-run.tsv`, or `expected.tsv` and the tool's export
side by side — and **Measured against the model**, under Findings, sets
what was measured beside what the model said that load would do. It
appears whenever a rate is loaded, per flow (`peer=`: `iperf_mbps_out`,
`mx_peer_pps`) or per host (`mx_egress_gbps`), and compares it against
the `expected.tsv` loaded with it, or the load generator as it is set now
when there is none.

```
43Gb/s measured, of the 50Gb/s the model expects on the same flows: 86%
30 flows, within 15%: 20 as modeled, 5 short of it, 5 not measured
6 hosts: 4 as modeled, 1 short, 0 over, 1 with nothing measured
Where to look: what the short flows have in common
  r1h1 sending: 5 of its 5 flows short
1 host off the model by more than 15%
  r1h1 sent 3Gb/s of 10Gb/s (30%) · 5 of its 5 flows, every one
1 host with nothing measured
  r2h2
```

- **Flow by flow**, within the tolerance either side of the model (15%
  unless changed) is *as modeled*, below it *short*, above it *beat it*; a
  flow the model has and the run does not is *not measured*, one the run
  has and the model does not *not in the model*. A bit rate is compared in
  Gb/s; requests a second (`mx_peer_pps`) in requests a second — or, against
  a model in Gb/s, as bits at the size the run says its packets were.
- **Against the model, not against what was asked.** A flow the model
  expects to be held back at a full cable, measured so, is as modeled, and
  listed under **As the model said** with where it was held. What falls
  short of *that* is the surprise.
- **Where to look**: what the short flows have in common that the flows
  that kept up do not. A *place* — a server, a rack, a row — nearly every
  flow out of (or into) fell short: one slow server names the server, four
  racks at half rate name the four racks. A *switch* or a *cable* most
  short flows cross and almost no good one does: a rack's uplink, when what
  leaves the rack falls short and what stays inside it does not. The widest
  place is kept over everything inside it — a rack, not its ToR, its
  uplinks and its sixteen servers. Each side is routed one flow a pair, so
  below the layer where routes fork it is as exact as the layout; above it,
  as good as the model's guess of which way each flow hashed. Hosts are
  named as the run named them where that name is theirs alone.
- **Hosts**: one that is off the model as a whole is one line, not a line
  per flow; one the model has sending and the run has nothing from is
  listed apart — usually a host that never ran its part.
- **Loss**: with a loss metric per flow loaded too (`mx_peer_loss`,
  netmesh's `loss`), flows losing more than 1% are counted, and those
  where the model has room called out by pair: loss the load does not
  explain.
- **Round trips** — netmesh's `rtt_p50` and `rtt_p99`, or any per-pair
  metric in a time unit (`ns`, `us`, `ms`, `s`) whose name says round trip,
  RTT or latency — are set against the fabric's own round trip for each
  pair: there and back, 64-byte probes, every switch's `latency=` and every
  cable's time to send. That needs no load and no `expected.tsv` — the
  layout alone says it, so **Against** offers *the layout* — and netmesh's
  `expected.tsv` from **Run it for real** carries it too, as
  `model_rtt_us` (and `model_rtt_load_us`, with the generator's queues).
  A measured round trip is mostly the hosts' stacks and the prober, much
  the same for every pair, and rarely in the model; so that part is found,
  not assumed: the median of what each pair measured beyond the fabric is
  **common to every pair**, added to each pair's expectation, and a pair
  more than the tolerance past it is *slower* — a path, not the hosts. The
  panel says what to put in the model for the common part, a `latency=`
  on the servers (a quarter of it: a round trip crosses each end's stack
  twice). **Measured under the load** sets it against the fabric with the
  generator's queues, for a netmesh run beside an mx one.
- **Path MTU**: with netmesh's `path_mtu` loaded, the pairs whose path
  carries less than the fabric's MTU (the one in Run it for real, or
  `expected.tsv`'s `model_path_mtu`) are listed, with where to look — a
  port, a switch or a tunnel set lower. Its `loss` is listed as a rate
  tool's is.
- **Requests alone.** With replies on, what a host sends is its requests
  and the replies it owes the hosts that call it. mx's `mx_request_gbps`
  counts the requests alone, so it is set against the model's requests —
  `expected.tsv` carries them as `model_request_gbps` when there are
  replies — and the panel says so; `mx_egress_gbps` is the figure with the
  replies.
- **An agent at its ceiling.** With the run's `mx_agent_cpu` (or netmesh's
  `agent_cpu`) loaded, the hosts that fell short with their agent at 90% of
  a core or more are named, in amber, above the rest: mx's agent is Python,
  a core a worker, and short for want of CPU looks just like short for want
  of a cable. More workers (`mx start --workers`), more `--streams`, or
  bigger packets.
- **Show on the floor plan** loads the comparison as overlays of its own —
  `vs_model_pct`, each host's sending as a share of what the model expects,
  red to blue around 100%; `vs_model_flow_pct`, each flow's; and
  `vs_model_verdict` — and **Save vs-model.tsv** keeps them as a file to
  load again later, its summary and suspects in its header.

#### Hovering a cable

The pointer on a cable outlines it, and its tooltip says:

- its net and its two ends — `TOR to spine: R01/tora ↔ SP1/spine1`;
- what it can carry — `1 cable · 100G`, or for parallel cables drawn as one,
  `4 cables · 100G each, 400G in all`;
- with load on the cables: what it carries each way, and how full it is the
  busier way — or, comparing, now against before;
- and, once the pointer rests on it a moment, **the flows that cross it**,
  biggest first, with how much of each: routed exactly as the drawing was,
  so they add up to the load shown. A test flow is marked as one.

Where several cables are drawn as one line — the stub out of a ToR carries
every cable it has, a rack's lane every server's — the pointer is on all of
them, and the tooltip says so: `NIC a: R01/tora ↔ 4 spliced (s1…s4), …`, in
and out of the ToR. Inside a device's own box, the device is what is shown
unless the pointer is right on a cable.

Every cable is full duplex, and its load is told each way: a NIC carrying
88 Gb/s out and 88 Gb/s in is 88% full, not 176%. A flow listed over a
cable shows the part of it that crosses that cable — under an even split
over two NICs, half of it each.

#### The headline

At the top of the Cable load panel, one number: what the hosts deliver to
each other in all — with contention, of what they offered — and over how
many flows. Under it, each net's load, both ways, against what its cables
that carry any of it can carry: `plane-1 57.6Tb/s of 115.2T · 50%` says how
much crosses the superspines. The nets are not added up, since a flow
between two rows crosses NICs, uplinks and planes and is counted on each.

#### Hovering a server

With load on the cables, a server's tooltip says what it sends and receives
in all, and how full that is of its cables' capacity, the busier way —
`sending 25Gb/s · receiving 25Gb/s — 50% of its 50G` — then each of its
cables, out and in, and how many of the flows start and end there. Shared
out, it also says what it offered where a full cable held it back, and how
many of its flows were: an incast target reads `receiving 50Gb/s, of
775Gb/s offered — 100% of its 50G … 31 held back by a full cable`. A switch
says how much it switches instead (see **How full a switch is**).

#### Per-host totals: an estimate

A metric measured per host, with no `peer=` — `mx_egress_gbps`, or any
`egress_gbps <host> 30` — says how much each host sent but not to whom. Its
card offers **load the cables (estimated)**: each host's total is sent to the
other hosts measured, then routed like flows.

- **to each host by its own total** (the default) sends more to the hosts
  that are busy themselves — the *gravity model*, the standard first guess at
  a traffic matrix from per-host volumes. A host sending 10 among hosts
  totalling 10 and 20 sends them a third and two thirds.
- **evenly to every other host** sends each the same share.
- **to its peers, as the test paired them** sends each server's total to
  its random peers instead, in the layers set under it — the same setup
  explaining uses: × peers, within or outside its own rack (row, room, …),
  and a share for each layer. A few flows a server rather than one to
  every other, so it shares out and its replies come back however many
  servers there are. It is taken by itself where every pair would be too
  many flows to share out — a thousand servers are a million pairs — and
  the panel says so.
- Every host still sends exactly its total; what is a guess is only where it
  goes. The panel says the load is estimated and from how many hosts, and an
  estimate is always shared as a fluid — hashed or adaptive routes it as an
  even split, since one path per guessed flow would read as a measurement.
- Only devices count as hosts: a reading on a rack or a room is left out,
  and the panel says how many were.
- The metric must be a rate (`unit=Gb/s`, `pps`, …, or a name like
  `mx_egress_gbps`). A temperature or a bare count is not offered.
- **treat as duplex** reads each total as both directions added — what
  `iperf --bidir` reports — so half of it is what the host sent and half
  what it received. It is ticked by itself when the metric's name says
  `duplex` or `bidir`, or its `!test` line says `duplex=yes` (`duplex=no`
  says it is not). It is offered unticked when hosts read past what their
  cables carry one way — a measured 309 Gb/s from a server with two 100G
  NICs cannot be what it sent, and the panel says so in red, naming the
  worst. Ticked or unticked by hand, it stays as set for that metric. The
  same box sits in the load generator when it reads the metric.

`examples/dual-plane-hosts.tsv` gives each server of `examples/dual-plane.dc`
a total: a busy rack, and one hot server in a quiet one.

#### Explaining the results by the network

The estimate above asks *where would these totals go?* Ticking **explain
these results by the network**, under the same per-host load, asks the
question the other way round: if every server tried to do exactly the same
thing and the network is what stopped them, *what would the network have to
look like to produce these results?*

- **What every server tried** is set in the panel itself, to match how
  the test was run, at **the best server's result**:
  - **the pattern** — random peers (the default, one each), all-to-all, a
    ring, a shift;
  - **× peers** for random peers: how many each server sent to;
  - **where** — anywhere, *within its own* rack (row, room, …) or
    *outside its own* — the same choices as the Load generator's **to
    hosts**. A test run rack by rack is *within its own rack*: left at
    *anywhere*, every server is assumed to have sent across the whole
    floor, over the superspines.
  - **+ layer** adds another set on top, with **% of it** — the share of
    each server's traffic that layer carried. Two tests run at once, iperf
    to 4 peers inside the rack and mx to 8 anywhere, are two layers; each
    draws its own peers. "to 4 random peers within its own rack (25%) + to
    8 random peers (75%)" says what was assumed.
  - **New peers** draws every layer's peers again; **From the Load
    generator** starts from its settings as one layer.

  The setup matters: explaining two-layer results as one layer anywhere
  finds the right cable at the wrong speed and reproduces fewer servers. Change the pattern there, and
  the routing (packet spraying, hashed, …) in the routing box, to match the
  test that was run.
- Each server that fell short of that is a server the network has to
  explain. The search looks for the cables those servers share, and for
  each one the capacity that holds every server crossing it back to what it
  measured — running the same max-min contention as the load itself, so a
  cable shared with a fast server is not blamed for a slow one.
- A cable that comes out under its `gbps=` is **degraded** — a lane down, a
  bad optic, a speed mis-negotiated — and says what it runs at: "runs at
  30G of its 100G · 8 servers". One that only needs to be full at its rating
  is **full**: simply more offered than it carries. Click either to select
  its sending end; hover for the servers it explains.
- A server no shared cable explains is **held back by itself** — its NIC,
  CPU or stack — and is named as such rather than blamed on a cable.
- **As the layout already has it**: a short server that the layout as it
  stands already holds to its result — cables full at their `gbps=`, or
  its own `stream=`, `cpugbps=` less what it was busy with — is named as
  such, with how busy it was, and no cable is changed for it.
- A packet rate with its packets sized (mx's) is read in bits: each
  server's requests and the replies it sends back.
- The cables in **explained by the network** carry the load as the
  explanation shares it out, so none reads past its capacity but a suspect,
  against what the results give it.
- **The servers themselves**: when most of them stop at the same result,
  and the network as the layout has it would carry more of the same load,
  the limit is theirs — "32 of 32 servers stop at about 155Gb/s, where the
  network would carry 200Gb/s". It gives the `stream=` (the result over
  each server's flows and connections) or `cpugbps=` (sending and receiving
  the result at once) that would reproduce it; write either on the servers
  and the panel has nothing left to explain. Where cables explain the
  results too, it is offered as the other answer, after them.
- When even the best server sits on a degraded path (all-to-all, where every
  server crosses the bad cable), the best result is not what was tried: the
  search tries higher targets and says so — "above the best result of 31G:
  even the best was held back".
- The panel ends with how well it fits: how closely the model, with those
  cables changed, reproduces every server's result, and how many come out
  within 5%. A poor fit is in red — the pattern, peers or routing likely do
  not match the test.
- The cable view gains **explained by the network**: the load as it would
  be, with each suspect cable's share read against the capacity the results
  give it.
- It is worked out in turns, the page answering between them, and on
  thousands of servers stops after twenty seconds with the cables it was
  surest of — the panel says when it did.

It is an inference, not a measurement: it finds the *fewest* cables that
explain the results, and a different set may explain them as well. Switch
counters, below, are what settle it.

#### Switch counters: measured, on the cable

Interface counters say what each cable actually carried. Write one sample per
switch port, naming the neighbour the port faces with `link=`:

```
!test port_gbps unit=Gb/s label="Switch port counters"
port_gbps  R01/torb  200  link=SP1/spine1  net=fabric            # sent towards spine1
port_gbps  R01/torb   40  link=R01/s3      net=nicb    dir=in    # received from s3
```

- The value is what the device **sent** towards that neighbour; `dir=in`
  makes it what it **received** from it.
- `net=` picks the net when two cables of different nets join the same two
  elements (a server's data and mgmt to one ToR). Parallel cables of one net —
  four uplinks to the same spine — share the port's value in proportion to
  their capacity.
- Both ends of a cable often report it, one as sent and one as received:
  those are two readings of the same traffic, and are averaged, not added.
- Samples over time reduce by the card's aggregation, as everywhere.
- A counter naming two elements with no cable between them is listed in the
  panel rather than drawn somewhere wrong.

The card offers **draw on the cables**: the counters, measured rather than
routed, drawn in the same colours and widths as the model. With a routed
metric loaded too, the panel's **Show** choice offers the model, the
counters, and **measured − model**: each cable coloured by how far the
measurement is from what the model said — red where it carried more, blue
where less, as a share of the cable's capacity — with the biggest
disagreements listed. That is where hashing put more than its share, or where
traffic runs that the flow metric never mentioned.

`examples/dual-plane-counters.tsv` is the counters for the run in
`examples/dual-plane-flows.tsv`, as ECMP hashing really placed it plus two
flows the flows file does not know about: one R01 uplink carries 200 Gb/s
where the model has 80.

#### What-if: take something out

Select an element and press **Take out** in the inspector, or press the ✕ on
a cable in the panel's lists, and it is out of service:

- A cable taken out carries nothing; an element taken out carries nothing
  and neither does anything inside it — take out a rack and its servers'
  flows go with it, listed as *taken out*.
- The model routes round what is left, and the **Show** choice switches to
  **change since taking out**: every cable coloured by how its load moved —
  red where it took more, blue where it lost it — with the biggest changes
  listed. The panel says how many flows lost their route and how many cables
  are now past full against before.
- What is out stays on the floor, marked: an element crossed through, a cable
  as a dashed red line between its two ends. Routes between picks go round it
  too.
- Each is a chip in the panel's what-if box; click one to put it back, or
  **Put all back**. Taken-out cables and elements are remembered by name, so
  editing the layout keeps the what-if.

## The results file (`.tsv`)

One sample per line — `test  target  value  [key=value …]` — separated by tabs,
commas or spaces. It is **append-only by design**: to add results from a new
test run, append lines. `cat run47.tsv >> results.tsv` is a fully supported
workflow.

**Already have a table?** A file of `<stamp>  host  var1  var2 …` loads as
it stands, no conversion and no layout file — see
[A table you already have](#a-table-you-already-have). This format stays
exactly as it is described here; that is a second reader, chosen per file.

Converting data from somewhere else into this format — by hand or by handing
the job to an AI — is written up in
[docs/converting-data-to-overlays.md](docs/converting-data-to-overlays.md):
the target-resolution rules, what each `!test` key does, and the mistakes that
produce a file which loads cleanly and paints nothing.

```
temp_c      DH1/A/R01/u05   61.2    run=nightly-01
iperf_gbps  DH1/A/R01/u05   94.7
burnin      DH1/A/R01/u05   PASS
```

- Targets are element paths, but any **unique suffix** works (`A/R01/u05`,
  or just a hostname if node ids are hostnames). Matching is case-insensitive.
  A target that fits several elements goes to the first and the metric's card
  says so — *N ambiguous targets*, naming each one and where its reading went.
  Slot and role names repeat across racks (forty racks, forty `u01`s), so
  write enough of the path to be unique when they do.
- Values may be numbers or words (`PASS`/`WARN`/`FAIL` get traffic-light
  colors; other words get stable categorical colors).
- **How big can a results file be?** There is no limit in the viewer. A file
  is read as one string, so the hard wall is the browser's maximum string
  length — about 512 MB, past which `file.text()` throws. Measured on a
  laptop-class browser: 5 MB (142k samples) loads in under a second, 25 MB in
  ~3 s, 100 MB (2.8M samples) in ~12 s, 300 MB (8.5M samples) in ~34 s. Past
  that, split the run across several files and load them together — they merge
  exactly as appending would, and the panel groups them by file.
- The **same test + target may repeat freely** (many runs, many instances).
  All samples are kept, and the UI reduces them with the aggregation you pick:
  mean, median, min, max, sum, count, first, last, harmonic mean, geometric
  mean, p95, p05, stdev, range. A metric of **verdicts** is not averaged and
  offers no choice: the worst wins, so a failure under a collapsed rack stays
  visible, and where nothing is worse than anything else the most common value
  does. The card says so in place of the picker.
- Optional `!test` lines set display metadata:

```
!test temp_c  unit=C  higher=bad  min=18 max=78  short=TMP  label="Inlet temp"
```

### Standardizing a metric

An overlay's colours normally span the smallest and largest value present, so
the same colour means different things on different metrics, and a floor where
everything is fine still paints something red. **standardize** on the overlay
card colours by the **z-score** instead — how far each value sits from that
metric's own mean, in standard deviations:

| Setting | Values shown | Colour from |
|---|---|---|
| `off` | the metric's own values | the min…max range (as before) |
| `colour by z-score` | the metric's own values | z, across ±σ |
| `values as z-score` | the z-score, in σ | z, across ±σ |

Use the first when the numbers matter and the colours should say "unusual for
this metric"; the second when comparing metrics in different units side by
side, where +2σ on temperature and +2σ on latency should look alike.

**standardize all**, above the cards, switches every metric at once. It
*overrides* the per-metric settings rather than rewriting them: each card goes
on showing (and still lets you edit) the setting it was given, greyed to say it
is not what is being drawn, and switching the override back off puts every
metric back on its own setting — including the ones you had deliberately left
raw. `off` there means "no override", not "force every metric raw". A metric
loaded while it is on is standardized too. The ramp's spread stays per metric,
since a shared one would have to overwrite what each card holds.

**one scale** — beside it, and on by default once anything is standardized —
is what makes two metrics actually comparable by eye. Spanning ±σ is not
enough on its own: a per-metric `palette=`, a per-metric spread, or a
`higher=good` inversion each paint the same z-score green on one metric and
red on the next. Ticked, every standardized metric is drawn from one palette,
one spread and no inversion, so **the same z-score is the same colour
everywhere**; each card still shows the palette and spread it would use on its
own, greyed. The default ramp is `rdbu`, diverging, because a z-score is
signed and the mean belongs in the middle. Untick it to go back to per-metric
colours.

The mean and spread are measured over **one aggregated value per measured
element** — the devices being compared, not the raw sample rows, which repeat
per run — so changing the aggregation re-measures them. The overlay card shows
what they came out as (`mean 40.5C ± 8.73C over 480`), the ramp ends default to
±3σ and are adjustable, and a metric with no spread at all reads 0σ everywhere
rather than dividing by zero.

**A z-score says how unusual, never how much.** +2σ is 17 °C on one metric and
0.4% on another, and sharing one scale across metrics is exactly what makes
that worth knowing. Three things say it:

- The legend carries **two rows**: `≤ -3σ · mean · ≥ +3σ` over what those three
  points are worth here — `14.31C · 40.50C · 66.68C`.
- The inspector and the tooltip show **both numbers**: `38.8Gb/s  -4.82σ`. The
  floor plan still shows whichever single number the mode asks for, because a
  slice is too narrow for two.
- Hovering the units row says what one σ is worth on this metric.

**Is σ a fair yardstick here?** A shared scale assumes one σ means the same
thing on every metric, which holds only while their distributions are a
similar shape — so each card says how much of its metric sits beyond ±2σ
against the 4.5% a normal distribution puts there. Close to it and the
comparison is sound. It is highlighted only when the tail is further from
normal than chance explains **at that sample size**: on twenty elements the
expected tail is 0.9, so one element either side doubles the percentage, and
a mark that fires on that is a mark you learn to ignore. A split population
(σ set by the gap, nothing in the tails) and a fat tail (σ set by the bulk,
too much outside it) are both caught once there are enough elements to tell.

**The ends are where the colour stops, not where the data stops.** Everything
past ±3σ paints the same colour, so an outlier at -4.8σ is indistinguishable
from one at -3.1σ — at exactly the extreme worth looking at. The card counts
what ran off (`20 past the ends, worst -4.82σ`), those elements carry a bar on
the edge they left by, and **fit** widens the range until nothing is clamped.
On a shared scale, fit reaches past the furthest point on *any* standardized
metric, since one range has to hold them all.

(`higher=bad|good` picks the green↔red ramp direction; `palette=` chooses any
ramp: viridis, magma, plasma, turbo, health, cool, ember, gray, rdbu;
`agg=` presets the aggregation; `decimals=` fixes formatting, 0 to 10 places;
`finding=drops|lockups|errors|verdict` says what trouble a metric reports, for
the [findings](#findings-what-went-wrong-and-whether-the-load-explains-it)
list, where its name does not — and `finding=no` that it reports none;
`duplex=yes|no` says whether a per-host rate adds both directions, where its
name does not say `duplex` or `bidir`; `bytes=` and `reply_bytes=` size a
packet rate's requests and replies, in bytes of payload, so it loads the
cables as a bit rate.
`min=`/`max=` must be numbers with `min` below `max`; anything else is
reported in the load report and ignored rather than quietly applied.)

Fields split on tabs, commas or runs of spaces — except inside quotes, so a
value that needs a space is written `label="Inlet temp"` (single quotes work
too) and the quotes are not part of it.

### `dcadd` — appending made even easier

Optional helper; plain `>>` works too. `tools/dcadd` runs it from a clone
with nothing installed; `pip install` puts it on `PATH` as `dcadd`.

```sh
dcadd results.tsv temp_c DH1/A/R01/u05 61.2 run=nightly   # one sample
my_test | dcadd results.tsv --stdin temp_c                      # target value per line
dcadd results.tsv --merge run1.tsv run2.tsv                     # concat other files
dcadd results.tsv --csv fio.csv --test iops --target host --value write_iops
dcadd results.tsv --meta temp_c unit=C higher=bad min=15 max=95 'label=Inlet temp'
```

A metadata or `key=value` field whose value carries a space or a comma is
quoted on the way out, so `label=Inlet temp` writes `label="Inlet temp"` and
reads back whole. The format has no escape, so the quote used is the one the
value does not contain; a value holding both is refused rather than written
as something that reads back differently.

### JSON results

Anything that starts with `{` or `[` is read as JSON instead — no flag, no
separate extension, and a `#` comment above it does not count. The overlays
that come out are identical to the text form's, so the two are
interchangeable and can be loaded together.

**NDJSON — one object per line — is the form to generate.** It keeps the
append-only property that makes `cat run47.ndjson >> results.ndjson` work,
which a top-level `[ ... ]` array would lose:

```json
{"!test":"rtt_p50","unit":"us","higher":"bad","palette":"turbo"}
{"test":"rtt_p50","target":"wr12r06u15","value":184.2,"meta":{"peer":"wr12r06u16"}}
{"test":"burnin","target":"wr12r06u15","value":"PASS"}
```

An object with `!test` is the JSON spelling of a `!test` line. Everything
else needs `test`, `target` and `value`; `meta` is optional and shows up in
the inspector. A number stays a number and a string that reads as one is
treated as numeric, so a value quoted by whatever generated the file behaves
the same as an unquoted one.

A whole document also works, for tools that would rather emit one value:

```json
{"tests":   {"rtt_p50": {"unit": "us", "higher": "bad"}},
 "samples": [{"test": "rtt_p50", "target": "wr12r06u15", "value": 184.2}]}
```

A bare `[ ... ]` array of samples is accepted too. Malformed lines are
reported with their line number rather than dropped, so a broken generator
shows up as a warning instead of an empty overlay.

### `mx export` — matrix_orchestrator writes this format itself

[`matrix_orchestrator`](https://github.com/MartinGallagher-code/matrix_orchestrator)
needs no importer: it exports overlay samples directly, so there is nothing
in between to guess at its reports.

```sh
mx run --for 120                       # measure
mx export --window 120 >> results.tsv  # colour the floor plan with it
```

The numbers are the ones `mx summarize` prints, reduced by the run's own
rules — a blank cell is *not measured* and never zero, a layered run's rates
come from the host rows, latency is the worst peer's rather than a
percentile of percentiles. Those rules are not visible from outside a
`reports/` directory, which is why this direction is an export and not an
import.

#### What arrives on the floor plan

One sample per host per run, targeted at the host's name. Every overlay
carries its own units and palette direction, so the first render is already
readable.

| Overlay | Unit | What it is |
|---|---|---|
| `mx_pps` | pps | requests this host sent |
| `mx_rep_pps` | pps | replies it got back |
| `mx_served_pps` | pps | requests that *arrived* here from its peers — the receiver-side truth a sender cannot see |
| `mx_request_gbps` | Gb/s | its own requests on the wire, Ethernet/IP/UDP framing included |
| `mx_egress_gbps` | Gb/s | everything it puts on the wire: those requests plus the replies it owes the hosts that call it |
| `mx_rel_median` | % | its packet rate against the fleet's own median — 100% is "normal for this fabric" |
| `mx_line_util` | % | egress against the NIC's line rate (only with `mx export --nic-gbps`) |
| `mx_loss` | % | round-trip loss |
| `mx_forward_loss` | % | of what it sent, the share that never arrived |
| `mx_return_loss` | % | the share that arrived but whose reply never came back |
| `mx_achieved` | % | delivered rate against the matrix's target |
| `mx_coverage` | % | layered runs only: the share of its peers measured so far |
| `mx_rtt_avg` | µs | mean latency over its flows |
| `mx_rtt_p50` `mx_rtt_p99` `mx_rtt_max` | µs | latency of its **worst** peer |
| `mx_cpu` | % | the whole box, averaged over its cores |
| `mx_cpu_core` | % | its busiest single core |
| `mx_agent_cpu` | % | the busiest mx worker, as a share of *one* core — near 100% means mx itself is the ceiling |
| `mx_peers` | | flows this host sends |
| `mx_workers` | | agent worker processes, which is what makes `mx_agent_cpu` readable |
| `mx_intervals` | | report intervals it contributed to the window |
| `mx_state` | | `REPORTING`; `SILENT` for a host that reported earlier in the run but nothing inside the window; `NO-DATA` for one in the matrix that never reported at all — the reading a missing report cannot give you |

`mx export --peers` adds a per-flow overlay — `mx_peer_pps`, `mx_peer_loss`,
`mx_peer_rtt_p99` — one sample per flow, each tagged `peer=` (and `layer=`
on a layered run). They live under their own names so a `mean` over the
per-host overlays can never quietly include per-flow rows, and they ask for
`max` by default, which reads as *the worst peer of this host*.

**The matrix on the cables, in packets and in bits.** `mx export --peers`
writes `mx_peer_pps`, one sample per flow tagged `peer=` — the matrix itself.
Tick **load the cables** on it and every flow is routed, src to peer. A
packet rate alone cannot be set against a cable's `gbps=`, so the panel
asks for its packets' size: **packets of [256] B, replies of [4096] B**,
filled in by itself from the `# run shape: 256 bytes out -> 4096 bytes back`
line mx export writes at the top of the file. With a size:

- every request is routed out at its payload plus 66 bytes of framing
  (Ethernet, IP, UDP, preamble and gap — mx's own `mx_request_gbps` is
  exactly that), and **its reply is routed back** at the reply's size, so a
  matrix of small requests and big replies loads the return path hardest;
- the load is a bit rate: cables read as a share of their `gbps=`,
  contention shares out the full ones, and switches say how full they are;
- the headline gives both: `168.7Gb/s delivered between hosts · 132 flows`,
  then `4.7 Mpps requests · 4.7 Mpps replies — 256 B out, 4096 B back`.

Change either size in the panel to ask what a different payload would do;
empty the reply size to route the requests alone. `bytes=` and
`reply_bytes=` on a `!test` line size any packet-rate metric, mx's or not.
Per-host totals (`mx_pps`) work the same way, with each host's requests
spread as the estimate spreads them and every reply sent back to whoever
asked — up to about 380 hosts, the most that can be paired here; past that
the panel says the replies are left out, and `mx_peer_pps` is the one to
load.

**Rates you can read without knowing the hardware.** `mx_rel_median` is the
one to open first on an unfamiliar fleet: it puts every host against the
run's own median on a diverging ramp, so 100% is "normal here" and a slow
rack stands out whatever the absolute numbers are. It aggregates by
**median** deliberately — every host in a mesh talks to the sick host, so
`min` would redden the whole floor and hide it, while a host that is itself
slow has all of its flows slow. That is "I am slow" against "I have a slow
peer". (`iperf_rel_median` below is the same idea for throughput, and reads
the same way.)

What a relative overlay cannot see is a floor that is *uniformly* slow —
every host reads 100% of a median that is itself wrong. `mx export
--nic-gbps 25` fixes the scale to the hardware: it adds `mx_line_util` and
pins the throughput overlays absolutely, so half speed looks like half
speed.

**Every overlay arrives ready to read.** Units, palette direction, decimal
places, a 0-100 range pinned where the value really is a percentage of
something (auto-fitting makes a 30% CPU peak look alarming for no reason
but being the highest), and the aggregation that answers each overlay's own
question when a rack or room is collapsed: `max` for the worst peer's
latency and the busiest agent worker, `min` for coverage and intervals,
`median` for the two that diverge around 100%. All of it is overridable in
the overlay panel.

**An overlay is present only when its number was measured.** A host whose
receive side nobody reported gets no `mx_served_pps` and no
`mx_egress_gbps` — rather than a zero, which the viewer would average into
the rack and room above it — while `mx_request_gbps` is known from the
host's own rows and is always there. The loss split needs every peer's
receive rows, `mx_achieved` needs a paced matrix, and `mx_coverage` only
means anything on a layered run. Sparse overlays are normal here and are
the format working as intended.

**Reading `mx_rtt_p99`.** mx's latency histogram holds four buckets per
octave and a percentile reports its bucket's upper edge, so `mx_rtt_p99`
rounds *up* — by up to ~25%, and it can read a little above `mx_rtt_max`,
which is exact. That is a property of mx's report, not of the export;
`mx summarize` prints the same pair.

#### Making the names line up

Targets are the mx host names. The viewer resolves a bare name, a full
path, or any unique tail of one, so hosts named `wr12r06u15` in
`servers.txt` land on the right node with nothing else to do. A server list
of bare IPs will not resolve — the overlay panel shows them as *unmatched
targets* — so either name the hosts in `servers.txt` (`name=addr`) or map
them on the way out:

```sh
mx export --names hosts.map --target-prefix DH1/A/ -o results.tsv
```

#### The rest of the switches

`--raw` adds one sample per host per report *interval* for the columns a
host row carries, so the viewer can show min/max/p95 over a run rather than
the window's mean; the overlays derived from more than one row are still
written once per host. `--json` writes the same samples as NDJSON, one
object per line. `--window 0` reduces the whole report history instead of
the last 60 s. `--run LABEL` tags every sample `run=LABEL`, and
`--test-prefix` renames the overlays if `mx_` would collide with something
else in the same file.

### `export-overlay` — iperf_orchestrator writes this format itself

[`iperf_orchestrator`](https://github.com/MartinGallagher-code/iperf_orchestrator)
2.2+ needs no importer either. A mesh test answers "which host is slow"; this
floor plan answers "which rack", and the orchestrator exports the samples that
join them:

```sh
iperf-orchestrator run                          # measure
iperf-orchestrator export-overlay               # -> results/latest/iperf_overlay.tsv
iperf-orchestrator run --overlay                # or as part of the pipeline
```

**Throughput**

| Overlay | Per | What it is |
|---|---|---|
| `iperf_mbps_out` / `iperf_mbps_in` | direction | one test's rate, credited to the sender and to the receiver |
| `iperf_mbps_duplex` | host | what that host carried *at once*, both directions |
| `iperf_gbytes` | host | total data carried across the run |
| `iperf_line_util` | direction | rate as a % of the NIC's line rate (only with `--overlay-line-rate`) |
| `iperf_achieved` | direction | rate as a % of the `-b` target the run asked for |

**Relative — the overlays that say something a rate cannot**

| Overlay | Per | What it is |
|---|---|---|
| `iperf_rel_median` | direction | that direction against the run's own median, in % |
| `iperf_asymmetry` | pair | the gap between a pair's two directions, as a % of the faster one |

`iperf_rel_median` is the one to open first: 100% is "normal for this fabric",
45% is half speed, and you need not know what good looks like on this hardware.
It arrives with a diverging palette pinned at 0–200% and aggregates by
**median**, which matters more than it sounds — on a full mesh every host talks
to the sick host, so every host's *worst* direction is the one to it, and a
`min` aggregation reddens the whole floor while hiding the host that is
actually slow. The median separates "I am slow" (all my directions are) from "I
have a slow peer" (one is). Switch it to `min` when you do want the worst link
anywhere. `iperf_asymmetry` is the shape a duplex mismatch, a one-way policer
or a congested return path makes, credited to both ends and aggregating by
`max`.

**Reliability**

| Overlay | Per | What it is |
|---|---|---|
| `iperf_status` | direction | `OK`; `FAIL` with the status, error text and log file to open; or `NO-DATA` |
| `iperf_fail_kind` | direction | the failures only, valued by why (`NO_SUMMARY`, `DIRECTION_MISSING`, …) |
| `iperf_state` | host | the roll call: `TESTED`, or `NO-DATA` for a host that never answered |
| `iperf_ok_pct` | host | success rate, the **worse** of the host's send and receive sides |
| `iperf_peers` | host | how many distinct peers it exchanged data with |
| `iperf_coverage` | host | those peers as a % of the ones it was *planned* to reach |
| `iperf_tests` | host | how many directed tests it took part in — sample count is confidence |

`iperf_state` is the roll call, and `NO-DATA` is a host that was in the run's
server list and produced no row at all — SSH refused, iperf2 missing, box down.
Without it that host has nothing to paint and reads here as "not part of this
test", which is the one reading that is certainly wrong; it comes with 0% on
`iperf_ok_pct` so it lands on a numeric overlay too. The roll call is a
per-host overlay kept apart from the per-direction `iperf_status` for the same
reason `mx export` keeps `mx_state` apart from its per-peer overlays: two
granularities in one overlay would reduce into each other. `iperf_ok_pct` reports the worse side
rather than the pooled rate because a host that receives fine and cannot send
anything is broken, not half-well — the `sent=`/`recv=` metadata names which
side failed.

**The host itself**

| Overlay | Per | What it is |
|---|---|---|
| `iperf_cpu_peak`, `_mean`, `_softirq`, `_sys`, `_user`, `_idle_floor` | host | from `cpu_summary.csv`; softirq and idle floor are per-core, and say which core |
| `iperf_bind_iface` | direction | which NIC the traffic rode, when the run used `--bind` |

Every overlay arrives with its unit, ramp direction, short name and — for the
percentages — its real `0–100` range, because an auto-fitted scale makes a 30%
CPU peak look alarming purely for being the highest number present. Sample
metadata carries the peer, the test's timestamp, and on a failure the error
text and log file; the header records the run id, its **mode** (a `parallel`
run is the whole fleet under load at once, `sequential-pair` is one link's
uncontended maximum — the numbers are not comparable) and the shape every
sample shared.

Two things are deliberately never invented. A direction that produced no number
is **never** exported as 0 Mb/s — zero is a measurement, and averaging it in
makes a broken link read as merely slow — so it leaves as a `FAIL` verdict and
pulls its host's coverage down instead. And `iperf_mbps_duplex` is not a plain
sum: parallel and sequential runs fire a round's flows together so those rates
genuinely add, while rolling mode probes one pair over and over and adding
*those* reports more than the NIC can carry. Rows are clustered by overlapping
test window, summed inside a cluster and averaged across clusters; when the CSV
carries no test windows the overlay is left out rather than guessed at.

**This export and `mx export` are built to the same rules**, so an `mx` run
and an iperf run can share one results file and read as one system:
`iperf_achieved` is `mx_achieved` against a target rate, `iperf_coverage` is
`mx_coverage`, `iperf_tests` is `mx_intervals`, `iperf_state` is `mx_state`,
both keep per-host and per-peer values in separate overlays, and both write
numbers to four significant digits in fixed notation — `%g` renders a host's
aggregate as `1.163e+06`, which is correct and unreadable in a file people
grep. `--overlay-test-prefix` (like mx's `--test-prefix`) namespaces every
overlay, so two runs exported with different prefixes load side by side for
comparison instead of averaging into each other.

One measured row becomes one sample, so the aggregation menu here does the
reducing (`mean` across peers, `max` the best, `min` the worst, `stdev` how
steady). Every sample carries `run=<run-id>`, so a nightly
`export-overlay --overlay-append` accumulates history in one file that `first`
and `last` read as then-and-now. `--overlay-map` and `--overlay-prefix` rename
hosts onto whatever the layout calls those nodes, and `--overlay-format ndjson`
writes NDJSON instead.

### `reckon` — binnacle grades a run against the hardware

The overlays above are relative: `mx_achieved` to the target the run asked
for, `mx_rel_median` and `iperf_rel_median` to the fleet's own median. A
fleet running at half its NICs sits at 100% of both. The absolute reading
needs the hardware, and the hardware can live in the layout as ordinary
attributes:

```text
row A +compute nic_gbps=25
  rack r[01..04] u=42 uplinks=4 uplink_gbps=100
```

[binnacle](https://github.com/MartinGallagher-code/binnacle)'s
[`reckon`](https://binnacle.readthedocs.io/en/latest/tools/reckon.html) reads
that, works out what every flow of an mx or iperf run could have had — its
fair share of the NICs and rack uplinks it crosses, or its target if lower —
and writes this format itself:

```sh
reckon floor.dc --mx reports/ --overlay reckon.tsv
dcviz serve --layout floor.dc --results reckon.tsv
```

| Overlay | Per | What it is |
|---|---|---|
| `reckon_efficiency` | host | its flows' achieved vs expected, median, on a ramp pinned at 0–200% |
| `reckon_expected` / `reckon_achieved` | host | pps for an mx run, Mb/s for iperf |
| `reckon_limit` | host | what the hardware limits it to: `TARGET`, `NIC`, `UPLINK`, … |
| `reckon_verdict` | host | `OK`, `WARN`, `FAIL`, or `NO-DATA` for a host that never reported |
| `reckon_nic_gbps` | host | the speed it was graded against, `source=` layout, measured or flag |
| `reckon_peer_efficiency` | flow | one flow, with `peer=`, so it draws as a measured flow |
| `reckon_change` / `reckon_peer_change` | host / flow | with `--baseline`: efficiency now minus then, in points, on a ramp from −50 to +50 |

`reckon_efficiency` is the one to open beside `mx_rel_median`: the median
overlay says which hosts are unlike the rest, and this one says whether the
rest are where the hardware puts them. A host above 100% is a finding too —
it means the layout's hardware is wrong. Every assumption the model made is
written into the file's `#` header.

`reckon --baseline` takes an earlier run's reckoning — its `--json`, or the
overlay you kept for the viewer — and adds `reckon_change`: how far each host
moved since then. Efficiency compares across runs where raw rates cannot,
because each run is graded against its own expectation, so the change layer
shows a host slipping before any absolute layer turns red.

```sh
reckon floor.dc --mx reports/ --run wed --baseline tue.tsv --overlay wed.tsv
```

### `dcimport` — netmesh output, directly

`dcadd --csv` already imports any CSV with a target column and a value
column. `dcimport` is for netmesh, whose output does not have that shape,
because what it measures is a **pair** — `(src, dst)` — while the viewer
paints **elements**.

```sh
dcimport results.tsv --tidy reports/          # netmesh
tools/dcimport results.tsv --tidy reports/    # ... or straight from a clone
```

| Source | What arrives |
|---|---|
| [`netmesh`](https://github.com/MartinGallagher-code/binnacle) `reports/` | `rtt_p50` `rtt_p99` `jitter` `loss` `path_mtu` per peer, `agent_cpu` per host |

**Nothing is averaged on the way in.** One measured row becomes one sample,
so the viewer's own aggregation menu does the reducing: `mean` reads as
"across peers and intervals", `max` as "the worst peer", `last` as "right
now". `--reduce` collapses to one sample per host per metric (median) for
meshes big enough that the raw row count matters.

**Only while every agent was running.** `netmesh start` and `stop` reach
the hosts over ssh a batch at a time, so a run's first and last seconds are
not a mesh: a probe to a peer not yet up, or already gone, is never
answered and reads as loss on a network that lost nothing. `dcimport`
imports the rows written while every agent was running, read off the
reports the way netmesh's own summary reads them — from the first row of
the last host to start to the last row of the first host to stop — and
says what it left out, on stderr and as a `#` line in the file. A host
that starts or stops 15 s (three report intervals, if longer) or more
apart from the rest came late or died, and its loss stays in.
`--whole-run` imports every row.

A blank cell means "not measured" and is skipped rather than read as zero —
averaging a blank as 0 is the one mistake that quietly makes every one of these
numbers look better than it is.

`iperf_orchestrator` is not imported here either, for the same reason as `mx`:
it writes this format itself — see `export-overlay` above.
`tests/fixtures/iperf-overlay.tsv` is that export, kept as the contract the
suite checks.

Targets are whatever the tools called the hosts, so **generate the server
list from the layout and the names already match**. binnacle's `manifest`
reads this project's `.dc` format for exactly that:

```sh
manifest floor.dc 'room[1]' | netmesh gen --servers -
netmesh run --for 60 && dcimport results.tsv --tidy reports/
```

## A table you already have

Most monitoring already writes a wide table: a timestamp, a host, and a column
per thing measured. The viewer reads that directly — no conversion step. Load
your `.dc` file beside it as usual, or, if the layout is not written yet, ask
for one to be [built from the names in the data](#building-a-floor-plan-from-the-data).

```
Timestamp             host          rtt (us)   loss %   cpu %
2026-09-16T12:00:00   wr01r01u05    184.2      0.01     37
2026-09-16T12:00:00   wr01r01u06    191.0      0.00     41
2026-09-16T12:00:10   wr01r01u05    186.9      0.00     39
```

Columns are separated by **tabs** where the row has them — that is what lets a
heading be `rtt (us)` rather than three columns — and by **runs of spaces**
where it has none, so a table typed by hand or written by `awk` reads too. Try
it, with a floor plan read out of the hostnames:

```
http://localhost:8000/?build=1&results=examples/live/room-wr01.tsv
```

- **The first column is the stamp, and it is only the stamp.** A date, a time,
  an epoch, or a plain number counting the passes — `1`, `2`, `3` is as much a
  first column as an ISO date is. It is kept on every sample and it never
  names anything: a file like

  ```
  1   host_1   5
  2   host_1   6
  3   host_1   3
  ```

  is **one** metric, on one host, with three samples — not three metrics
  called 1, 2 and 3.
- **The name does not matter.** Which reader a file gets is worked out from
  what is inside it, so a table written to `today.log`, `metrics.dat` or a
  file with no extension at all is the same table it would be in `run.tsv`.
  The Files panel lists the usual data extensions by default and everything
  else behind **all files** — but it will open anything that could be text,
  and only refuses what cannot be (`.png`, `.gz`, `.so`). Going the other way,
  a floor plan called `floor` that opens with `dc DC1` is read as a layout.
- **The header is optional.** Without one the columns are called `A`, `B`,
  `C`… — so give the file a header line, or a `!test` line per column, when
  the metric deserves a name. A header may be commented out
  (`# Timestamp<TAB>host<TAB>…`), so the file stays a table to `awk`, and a
  second header part-way down the file is read as a new one — concatenated
  days each bring their own.
- **A heading carries its unit** when it is bracketed (`rtt (us)`, `loss [%]`)
  or ends in `%`. The metric keeps the whole heading as its label.
- **A blank cell is "not measured"**, never zero. Averaging a gap as zero is
  the one mistake that makes every number on a floor look better than it is.
- **A value may be a word** (`PASS`/`FAIL`), exactly as in the results format.
- **A host may be a flow**: `wr01r01u05 -> wr01r02u09` measures the path
  between two machines rather than a property of one. The sample belongs to
  the host it started at, with the far end as its `peer=`, which is what the
  viewer already draws as a [measured flow](#the-viewer) and filters with
  `peer=`. `->`, `=>` and `→` all work, and `a -> b -> c` keeps its hops.
- **A column can still be given display metadata.** A table may carry the
  results format's own `!test` lines, checked exactly as they are there, so a
  column gets a unit, a palette, a range or a short name without a second
  syntax for it — and without stopping the file being read as a table:

  ```
  !test  rtt   unit=us  higher=bad  min=0 max=500  short=RTT
  !test  loss  unit=%   higher=bad  slug=loss_pct
  Timestamp             host          rtt   loss
  2026-09-16T12:00:00   wr01r01u05    184   0.01
  ```

- **Several files combine.** Tables in the *same folder* are one dashboard:
  one file per host, or per metric, or per hour — a column of the same name in
  two of them is one metric carrying the samples of both. (Every other format
  keeps a file's metrics to itself, deliberately; this is the format where the
  opposite is what you meant. Two folders stay two dashboards.)

## Building a floor plan from the data

The `.dc` file is what says where the machines are, and normally that is the
file you load. But the names in a results file often carry their position
anyway, and sometimes the layout is not written yet — so **Build from data**,
in the Structure panel, reads a floor plan out of them. It is a button and
never anything else: nothing here invents a floor plan because a file arrived.

It works on **any** loaded results, not only a table: a results target is
already a path through a floor plan somebody wrote, rows and all.

```
DH1/A/R01/u05      ->  room DH1, row A, rack R01, machine u05
wr12r06u15         ->  room wr12, rack r06, machine u15
dc1-hall2-r03-u05  ->  room dc1, row hall2, rack r03, machine u05
rack01-server05    ->  rack rack01, machine server05
mailserver         ->  one rack of hosts
web01.dc.acme.com  ->  the domain the hosts share is dropped, not read as racks
```

The last part of a name is the machine, the part before it its rack, then its
row, and the rest its room. `-`, `_`, `/`, `:` and `.` split a name; a name
with no separator is split at its letter-and-digit runs, which is what makes
`wr12r06u15` three levels. Racks that name no row are dealt into rows of
twelve, so a flat fleet is a floor plan rather than one very long line.

- It is an **ordinary layout**: `dc DATA … +generated`, which **Edit layout**
  opens and **Download .dc** saves to correct by hand.
- While it is the floor plan it **follows the data** — a host that starts
  reporting gets a place without being asked for again — and **Clear** takes
  it away, leaving the data loaded.
- **Loading a `.dc` file replaces it**, every overlay still bound to the same
  hosts. Going the other way, replacing a floor plan you loaded, takes two
  clicks: that file is what the data is meant to be read against.
- When a layout is loaded, the panel also says how many targets in the data
  are **not on it** — the number that otherwise only reads as "nothing was
  measured".
- `&build=1` in the URL does the same thing on load, which is what makes a
  link to a folder of tables a dashboard somebody else can open.

### `dcprobe` — how every host is doing

`dcprobe` reaches every host over ssh and writes how each is doing as
results the viewer paints. One shell script goes to each host; nothing is
installed there and nothing is left behind. `tools/dcprobe` runs it from a
clone; `pip install` puts it on `PATH`. **Host status**, in the right
panel, writes its `hosts.txt` and commands for the layout's hosts, and
reads a loaded scan back into the hosts worth a look.

```sh
dcprobe preflight --servers hosts.txt -o preflight.tsv --tools iperf --ports 5300,5310,5001 --bind data
dcprobe scan      --servers hosts.txt -o host_status.tsv --nic data --mtu 9000 [--why-slow]
dcprobe watch     --servers hosts.txt -o host_status.tsv --every 30 --history host_history.tsv
dcprobe diff      host_status-before.tsv host_status.tsv -o changes.tsv
```

**Save probe-hosts.tar** puts it all in one archive, as **Run it for real**
does for a load: `hosts.txt`, the layout as `model.dc`, and
`probe-hosts.sh`, executable, in a folder of their own. **The layout goes
with the results**: every command passes `--model model.dc`, which writes
it, whole, into `host_status.tsv` as the `#~` lines a run file carries
(see [The run carries its layout](#the-run-carries-its-layout)), so
loading the scan brings back the floor plan its hosts were probed in —
asking first when another is in the viewer. The script says so and stops if `dcprobe` is not on `PATH`;
otherwise it runs preflight and then one scan (the watch and diff lines are
there commented, to swap in). **Save probe-hosts.sh** saves the script
alone.

```sh
tar -xf probe-hosts.tar && ./probe-hosts/probe-hosts.sh
```

`hosts.txt` is the grammar mx and netmesh read — `name[=addr]` a line —
with what the host should be beside it: `r1h1=10.1.1.11 gbps=25`, and
optionally `mtu=` and `nic=`. Host status writes it from the layout: each
host by the layout's name, at its `addr=`, with the speed of its fastest
NIC. **The hosts are the load generator's**: those its **hosts** filter
picks (every host when it is empty), less anything taken out — the hosts a
load would run on, so a probe looks at the machines the run will use. The
pane, and the script's comments, say which filter picked them and how
many there are; a filter that picks none says so instead.

| Verb | What it does |
|---|---|
| `scan` | one pass: everything below, for every host |
| `watch` | a pass every `--every` seconds; the file is rewritten each pass, so it always says how things are **now** and Live follows it; `--history` keeps every pass |
| `preflight` | can every host run the load? ssh, python3 (mx and netmesh copy their agents over), the tools the hosts need (`iperf`), `--bind` matching an interface, the run's UDP ports free, the clock against this machine's, open files allowed |
| `--model FILE` | (scan, watch, preflight) the layout, written whole into the results, named by `hosts.txt`'s `# layout:` line |
| `diff` | what changed between two scans — kernel, OS, CPU, memory, NIC driver and firmware, NIC speed, MTU and link — host by host, and `-o` as `probe_changed` |

**Hosts are probed in parallel, 256 at once** (`--jobs`): a scan of 1,000
hosts is four waves of about two seconds each — the ssh connection and the
one-second sample. Each host at once is an ssh child holding a few open
files here, so `dcprobe` raises its own open-file limit to fit, as far as
the system lets it; where `ulimit -n` will not stretch that far it probes
as many at once as fit, says so, and still probes every host.

What a scan measures, over a one-second sample (`--sample`):

| Metric | What it is |
|---|---|
| `probe_reach`, `probe_why` | answered the probe, PASS or FAIL, and ssh's reason when not — a host that does not answer is a sample, never a gap |
| `host_load1`, `host_load_per_core` | load, and load a core |
| `host_cpu_pct`, `host_iowait_pct`, `host_steal_pct` | CPU busy, waiting on I/O, stolen by the hypervisor |
| `host_mem_pct`, `host_swap_pct`, `host_disk_pct` | memory in use (from MemAvailable), swap in use, the fullest real disk |
| `host_softnet_drop_ps`, `host_softnet_squeeze_ps` | the kernel's receive backlog: packets dropped, and passes out of budget, a second |
| `nic_link`, `nic_speed_gbps`, `nic_mtu` | each physical NIC (no `lo`, veth or bridges) |
| `nic_rx_gbps`, `nic_tx_gbps` | what it moved |
| `nic_rx_drop_ps`, `nic_tx_drop_ps`, `nic_err_ps`, `nic_ring_drop_ps` | what it dropped and errored, and its ring and buffer drops (`ethtool -S`), a second |
| `nic_check`, `nic_why` | the NIC as the layout says: FAIL when none is up or none is at the layout's `gbps=`, WARN for one slower or at another MTU than `--mtu` |
| `host_kernel_errors`, `host_kernel_last` | kernel errors logged in the last hour (`--dmesg-since`; watch: since the last pass), and the last of them |
| `host_top` | the busiest processes |
| `inv_kernel`, `inv_os`, `inv_cpu`, `inv_cores`, `inv_mem_gb`, `inv_nic_driver`, `inv_nic_fw` | what the host is |
| `whyslow_verdict`, `whyslow_top` | with `--why-slow`: [binnacle's why-slow](https://github.com/MartinGallagher-code/binnacle), piped to each host over the same ssh, its worst finding as PASS, WARN or FAIL and its first serious one |

**Pass `--nic` the run's `--bind`** (Host status does): only the NICs it
matches are judged, so the run's NIC is held to the layout's speed and the
run's MTU, and a port nobody cabled is not a WARN. Without it, a NIC that
is down is listed but not held against the host.

**Drops, errors and verdicts go to [Findings](#findings-what-went-wrong-and-whether-the-load-explains-it)**
on their own (`finding=` on their `!test` lines). **Host status** lists, from
a loaded scan, who did not answer, who is not ready to run, whose NIC is not
as the layout says, what why-slow found, kernel errors, and who is busy
(CPU 90%+, load a core 1.5+, iowait 20%+), nearly out of memory (90%+),
swapping (25%+ of swap) or nearly full (a disk at 90%+) — each host a chip
with what it said, a click away. Folded, its title says how many answered
and how many need a look.

### Live: reload on a timer

The **Live** panel (left, under Files) turns loaded files into a dashboard:

- **reload every N s** — re-reads every loaded results file on a timer. The
  same reader and the same rules as opening one by hand, so a file being
  appended to replaces what it brought last time instead of counting its rows
  twice. A pass still running when the next is due is simply the pass that is
  running: they never overlap.
- **last N records per file** — reads only the end of each file, the way
  `tail -n` does, so a log that grows all day stays the size of what is on
  screen. Headers, comments and `!test` lines are kept whatever their age —
  losing them would take the column names and units with them. Empty means the
  whole file.
- **replace what it brought** / **add the rows since last time** — what a pass
  *does* with what it reads.

  *Replace* reads each file again from scratch: the file is the truth, which
  is what a file rewritten in place needs, and what stops a growing log from
  counting its rows twice.

  *Append* takes only the records that were not in the last read and adds them
  to what is loaded — so the view accumulates past the tail, which is the
  point: read the last 500 rows every ten seconds, keep the whole hour. Where
  the new read starts is found by looking for the **end of the last read
  inside it** (the last few records, matched as a block, since one row is not
  an identity — a table on a timer writes the same line twice the moment two
  passes measure the same values). A file that cannot be lined up — rewritten
  from the top, or grown by more than the tail being read — is *replaced*
  instead and the report says so, because taking rows that cannot be lined up
  would count some of them twice. Past 400,000 accumulated samples on a metric
  the oldest are dropped as new ones arrive, making it a rolling window.
- **Reload now** — one pass, on demand.
- **Load all**, in the Files panel, loads every results file in the open
  folder matching the name filter (`*.tsv` by default) and then **follows that
  folder**: on each pass it is listed again, so a file written while the
  dashboard is up joins it, and one that disappears takes its samples with it.
  The × beside the watched folder in Live stops that; nothing already loaded
  is unloaded.

Files opened through the folder browser reload best — the browser hands over a
handle that reads the file as it is now. Files chosen through the plain file
input are re-read by path, which works until the file is replaced rather than
appended to; a file that cannot be re-read is named in the panel rather than
quietly left stale.

## The viewer

- **Every titled part of the side panels folds**: a section (Files,
  Structure, Overlays, Networks, Load generator, Cable load, Findings,
  Measured against the model, Inspector), a list inside one (Busiest
  cables, Run it for real, Where to look …) and each part of the
  inspector (Readings, Cable load, Findings, Measured flows). A click on a
  title folds it to that title, and what is folded stays folded across
  redraws and reloads. A folded section still says in a few words how
  things stand — the cables' total, how many servers have findings, how
  many flows are off the model — and the buttons in its title (Run/Stop,
  Off, mark on the floor) work without unfolding it. A section with nothing
  to show is not shown, and a section's title stays at the top of the panel
  while its contents scroll under it. **Alt+click** a title to open that one
  alone, every other beside it folded; Alt+click it again to open them all.
  **Collapse all**, at the top of each side panel, folds every section of
  that panel to its title; once they all are, it reads **Open all** and
  opens them again. The parts inside a section keep what they were.
  The **Inspector** is the right panel's first section — what was clicked
  is what is being looked at — and its own parts sit a step below the
  panel's sections.
- **Light or dark**: the **Light** button in the top bar turns the panels
  and the floor plan light, for a bright room or a printout, and **Dark**
  turns them back. The choice is kept for next time.
- **What connects these?** Shift-click (or Ctrl/⌘-click) elements on the
  floor plan or in the tree to pick them; each pick gets a numbered tag.
  With two or more picked, only the cables between them are drawn: for each
  pair, every network that joins the two on its own, by its shortest routes
  — and every equally short alternative, so two servers in different rows
  show all the spines they could cross. The inspector lists the picks and,
  per pair, each network and how many hops it takes (`u05 ↔ u07 · data: 4
  hops`). A route crosses from one network to another only when no single
  network joins the two, as in a layout that gives each tier its own net; it
  never borrows a server's mgmt cable to reach the data fabric. A pick can be
  a container — two racks show the uplinks between them. Untick a network to
  route without it; Shift-click a pick again, or its chip, to drop it; **Clear**
  or Esc forgets them all. With cable load on, the picks box can also count
  only the flows between the picks, and send a test flow from the first to
  the second — see [Between picks](#between-picks-only-their-flows-and-a-test-flow).
- **Hover a cable** to see it: its ends, what it can carry, what it carries
  and the flows over it — see [Hovering a cable](#hovering-a-cable). With
  nothing to load, the [load generator](#the-load-generator-a-load-with-no-file)
  makes a load up.
- **Files** — the panel's **Open folder…** keeps a directory open beside the
  canvas, instead of a dialog that shows one and forgets it. Layouts and
  results are listed with their sizes (worth seeing before clicking a 300 MB
  file); a click loads one — a `.dc` as the floor plan, anything else as
  overlays added to what is already there — and a ✓ marks what is in. The
  extension only decides what is *listed*, never what can be opened: tick
  **all files** and a `today.log`, a `metrics.dat` or a file with no extension
  opens exactly the same way, because the format is read from the file. Subdirectories
  open, the breadcrumb goes back up, `⟳` re-reads a folder that has gained a
  file, and the name filter takes globs (`mx*.tsv`). Dropping a folder on the
  window opens it here. In Chromium the folder and the path inside it come
  back after a reload, behind one click for the permission the browser will
  not keep; elsewhere it falls back to a directory input, which lists the same
  tree but has to be chosen again each session. Nothing is read until it is
  clicked, and clicking a results file that is already loaded re-reads it —
  replacing its overlays rather than counting every sample twice, since the
  format is append-only. A file is known by its path below the open folder, so
  `monday/results.tsv` and `tuesday/results.tsv` are two files and stay two
  sets of overlays; where a load genuinely cannot tell two files apart (two
  named the same, dropped together, with no path between them) they are
  numbered and the report says so.
- **Restart** — empties the viewer: no floor plan, no overlays, no filter, no
  selection, and a blank canvas. **Remove all** in the Overlays panel only ever
  cleared the overlays and left the floor plan drawn; this clears the lot. What
  survives is the workspace rather than its contents — panel widths and folds,
  and the folder held open in Files, so the next thing to load is one click
  away.
- **The load report** — every load says what each file did, in a chip at the
  right of the top bar (`✓ 3`, or `⚠ 1` when something did not go as asked).
  Click it for the report; a load with a problem opens it by itself. It sits
  in the top bar rather than a panel because a panel collapses, and the one
  thing saying a file did not arrive must not be collapsible. It names the
  quiet outcomes in particular: a file with no data lines in it, a file whose
  lines are not in `test target value` form, **two layouts at once** (a viewer
  holds one floor plan — it says which won and which were ignored), and a file
  re-read over itself (it replaces what it brought before, rather than
  counting its samples twice). Warnings inside a file name that file, so
  `line 5:` is never ambiguous between two.
- **Panels** — either side panel collapses to a slim rail (the `‹` / `›` in
  its heading, and the rail brings it back) and resizes by dragging the edge
  beside the canvas; double-click that edge to reset a width. Every section
  inside a panel — Files, Structure, About, Overlays, Networks, Inspector —
  folds to its heading when the heading is clicked. Widths, collapsed panels
  and folded sections are all remembered between visits. Sections are wired
  from the markup, keyed by their heading, so one added later folds with no
  extra code.
- **Zoom / pan** — wheel and drag (level-of-detail keeps hundreds of
  thousands of elements smooth). `f` fits selection, `0` fits all. Labels grow
  with the zoom — each is sized from its element's on-screen height — so
  zooming in to read something makes it bigger, up to a ceiling.
- **Reloading cannot empty it by accident.** A reload loses everything
  loaded, built, edited or set, so once you have done anything since the
  page opened, Ctrl+R (⌘R, F5) does nothing and a note says so, and the
  browser's reload button or closing the tab asks first. A page just as its
  URL loaded it, untouched, reloads as ever: the reload brings it all back.
  **Restart** empties the viewer too, so it asks first while there is a
  floor plan or a metric to lose, and says which.
- **Expand / collapse** — double-click any container, use the tree, or the
  Collapse racks/rows/rooms buttons. Collapsed containers show their
  aggregate results computed from all raw samples inside them.
- **Filter** — the top bar matches anything: bare words search ids, names,
  tags and attributes, and a bare glob (`*serv*`, `r76?`) searches exactly the
  same fields, plus the full path so `DH1/A/*` is a query; `+gpu` tags (globs
  too: `+stor*`, `+gp?`);
  `kind:rack`; `model=r76*` (`?` matches exactly one character);
  `net:storage`; result queries like `temp_c>70`, `burnin=FAIL`,
  `has:iperf_gbps` (a test name can be loaded from more than one file, and the
  element matches when any of them reads over the threshold); `!` negates, `|` ors, space ands. Matches keep their
  ancestors visible; "hide non-matching" prunes everything else, otherwise
  non-matches are dimmed.
  **Sections, and boolean logic.** `in:A01` is everything inside A01 (and A01
  itself) — by id or name, or by path, `in:H1/A01` for the A01 in hall H1.
  Values take ranges the way a layout writes them: `in:A[01..04]`,
  `in:R[01..08,12]`, `name=u[01..10]` — and so does a bare name:
  `ht11r01u[7..11]`, `ht11r[01,05,07]u[7,9]`, each bracket a range (`7..11`),
  a list (`1,5,9`) or both (`1..4,7`), spaced as you like (`[07, 09]`). A number in a range matches however
  it is padded, so `u[7..11]` finds `u07` to `u11`; a range matches whole
  names, so `u[5..6]` is not `u50`. Terms combine with `and`, `or` and
  `not` (any case) and parentheses: `not` binds tightest, then `and` (or a
  plain space), then `or`, so

  ```text
  role=server (in:A[01..04] or in:B07) not +gpu
  role=server in:H1/A01 or role=server in:H2/A[05..08]
  ```

  each pick out servers in several sections at once. A `|` still joins only
  the two terms beside it (`+gpu a | b` is `+gpu` and either), and a quoted
  `"or"` is a word to search for, not an operator. The load generator's
  **hosts** box reads the same language, and says how many hosts it takes as
  you type.
  **A metric with a space or a slash in its name has a second name** that the
  box can read: `iperf Mb/s (out)` answers to `iperf_mb_s_out`, and every
  overlay card prints its own under **filter as** — click it to filter by that
  metric, or type it into a comparison (`iperf_mb_s_out>18000`). The original
  name still works where it can be typed at all; `slug=` on a `!test` line
  sets a different one.
- **Overlays** — check any number of tests. With N enabled, every element is
  split into N side-by-side slices, each colored by its test with the test's
  short name and value printed on it (2, 3, 4, 5… all work). Per overlay you
  pick the aggregation, palette, and value range live.
  **Hide all** unticks every overlay without unloading it, and **Remove all**
  drops them and their samples — both at the top of the panel, since a
  results file can carry twenty-odd overlays. **standardize all** there
  standardizes every metric at once, overriding each card's own setting
  without overwriting it (see
  [Standardizing a metric](#standardizing-a-metric)). Overlays are **grouped by the
  file they came from**, and each group's header collapses it, so an mx export
  and an iperf export loaded together stay tellable apart; the header counts
  the group's overlays and how many are shown, and its **×** removes that
  file's overlays alone. **sort A–Z** orders the metrics alphabetically within
  each file; unticked they keep the order the file wrote them in, which the
  exports choose deliberately (mx and iperf write theirs in reading order).
  **Files never combine**: every overlay belongs to exactly one file, so two
  files that both carry a test called `temp_c` are two overlays with two sets
  of samples, two domains and two cards — one under each file's group. To
  accumulate a metric over time, concatenate the runs into *one* file, which
  is what the append-only format is for. Loading the same file again re-reads
  it, replacing what it brought before rather than counting its samples twice.
- **Networks** — toggle each named fabric, or **Show all** / **Hide all** them
  at once; opacity slider for dense views.
- **Measured flows** — `mx export --peers` and `iperf_orchestrator` write one
  sample per *flow*, tagged `peer=`. Those overlays offer **draw measured
  flows**, which paints each measured pair as a dashed curve coloured by its
  own value, and the inspector lists a host's flows per peer (worst first)
  rather than only the aggregate that hides them. `peer=wr01r01u02` in the
  filter bar selects the hosts that measured a flow to that host, globs
  included. **A flow is not a cable**: mx and iperf measure end to end, so a
  flow between two servers in one rack really crossed server → ToR → server,
  and neither tool can say how the traffic divided across the hops. **load
  the cables** models it — routes every flow over the fabric and draws each
  cable by what lands on it (see [Cable load](#cable-load-where-a-flow-metric-lands));
  measured per-cable numbers still need switch interface counters.
- **Cable load and capacity** — with a metric loaded onto the cables, the
  Networks panel holds what the cables show (model, counters, the two
  compared, or the change since taking something out), the colour key, the
  routing (hashed per connection with **Re-roll** and the connections a
  flow, polarized, adaptive, by capacity, or packet spraying), the spread for an
  estimate, the what-if, and the busiest cables or biggest changes; **width
  by capacity** draws each cable by its `gbps=` instead. See
  [Capacity](#capacity-what-a-cable-carries) and
  [Cable load](#cable-load-where-a-flow-metric-lands).
- **Inspector** — click an element for its tags (click a tag to filter by
  it), attributes, U-slot, link counts, and all its readings. Cables hang off
  the leaf devices, so a rack, row or room reports **links below** instead:
  every cable in its subtree per fabric, split into the ones that stay inside
  it and the ones that leave. **Show only this element's cables** then hides
  every cable that neither starts nor ends inside the selection, at any level
  — one server's two cables, or everything entering and leaving a room.
  The elements at either end of the surviving cables are marked with a faint
  wash and outline (a dot where they are too small to outline), so the far
  end of a hairline is visible without tracing it.

## Tests

```sh
node tests/run.mjs        # 1,003 assertions over the modules
node tests/browser.mjs    # 109 more, driving the page in Chromium
```

Both skip what they cannot run — `tests/run.mjs` needs python3 for the
`dcadd`/`dcimport` sections, `tests/browser.mjs` needs Playwright — so either
runs anywhere. **`--strict` turns every skip into a failure**, which is what CI
passes: a green run that quietly tested nothing is indistinguishable from a
green run that tested everything, and that is the one result a test suite must
never give.

GitHub Actions runs both on every push and pull request, plus a third job that
builds the Python package, installs the wheel into a clean environment and
fetches a page from the server it provides — the wheel's layout is not the
checkout's, so nothing else here would notice a package that installs and then
serves nothing. Three jobs, so a red mark says which layer broke.

## Repository layout

```
index.html, css/, js/     the viewer (ES modules, no dependencies)
examples/small.dc         starter layout   (~1,200 elements)
examples/small-results.tsv  two nightly runs of four tests
examples/mega.dc          scale test (~256k elements on one page)
examples/hostnames.dc     flat hostname naming (wr12r06u15 style)
examples/splice.dc        four servers spliced into each cable to the ToR
examples/dual-plane.dc    dual-homed servers, spliced to TOR a and TOR b,
                          spines per pod and two planes of superspines, with
                          every net's capacity
examples/dual-plane-flows.tsv  flows for it: one rack across the planes at 80%,
                          and an incast at 160% -- tick "load the cables"
examples/dual-plane-counters.tsv  switch port counters for the same run, as
                          hashing placed it: compare them with the model
examples/dual-plane-hosts.tsv  per-server totals, for the estimated load
examples/dual-plane-health.tsv  drops, soft lockups, an EDAC error and verdicts,
                          for the findings beside the flows
examples/iperf/          a floor plan using every layout feature, with a real
                         export-overlay run painted over it (see its README)
examples/hostnames-results.tsv  results addressed by flat name
examples/mx/              every layout construct, painted by a real mx run
examples/live/            three wide TSV tables in one folder: the format that
                          needs no layout file, and what "Load all" loads
js/traffic.js             cable load: flows routed over the fabric (shortest
                          routes, through +switch elements, hashed per
                          connection, adaptively, by capacity or evenly),
                          per-host totals estimated by
                          the gravity model, switch counters onto their
                          cables, comparisons, and what-if
js/tsv.js                 the wide-table reader: detection, the arrow that
                          makes a host a flow, tail -n, and the floor plan
                          read out of hostnames
js/version.js             the project's version; python/dcviz/__init__.py
                          carries the other half and tests/run.mjs pins them
python/dcviz/             the Python side: dcadd, dcimport and `dcviz serve`,
                          which is what `pip install` ships. Stdlib only.
tools/dcadd, tools/dcimport
                          two-line shims that run those from a clone with
                          nothing installed, so the copy that ships and the
                          copy that runs are one file
pyproject.toml            packaging (hatchling; no runtime dependencies)
CHANGELOG.md              what changed, per version
docs/releasing.md         how a release is cut, and the one manual PyPI step
docs/                     the results-format guide for converting data in
tests/fixtures/           real tool output, as the contract the suite checks
tests/run.mjs             headless test suite (node tests/run.mjs)
tests/browser.mjs         the same for the page itself (node tests/browser.mjs) —
                          drives Chromium through the interactions that module
                          tests cannot reach. Skips with a message where
                          Playwright is not installed; the viewer still has no
                          dependencies.
.github/workflows/        both suites plus a package build on every push and
                          pull request; release.yml on a v* tag
.github/release_notes.py  the release notes: every changelog section since
                          the last release
LICENSE                   GNU General Public License v3
```

## License

Copyright (C) 2026 Martin J. Gallagher

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program. If not, see <https://www.gnu.org/licenses/>.

Full text: the `LICENSE` file at the repository root, or
<https://www.gnu.org/licenses/gpl-3.0.html> — SPDX identifier
`GPL-3.0-or-later`. This repository follows the
[REUSE Specification](https://reuse.software/): source files carry SPDX
copyright and licence headers, and the licence text lives in `LICENSES/`.
