# P6 — capacity benchmark harness

**Phase-0 proof harness. NOT shipped code.** Nothing here is imported by the platform,
nothing here touches Supabase, production, or a real provider. It exists to answer one
question from [PLAN.md](/Users/armanisadeghi/code/common-docs/projects/persistent-cloud-browser/PLAN.md)
§*Capacity model and the "10 or 50?" answer* — **how many of each kind of browser session
does one real host actually hold** — with measurements instead of estimates.

It is deliberately built to be run **unattended by whoever owns the hosts**: stdlib-only
sampling (no psutil), one command, JSON per step plus a human summary, and the admission
arithmetic done for you. A benchmark nobody can re-run is worthless; a benchmark that
needs babysitting never gets run.

> ⚠️ **The numbers in `out/` from a laptop, a CI box, or a shared container are not
> capacity findings.** Only the live EC2 sandbox host and the hosted 8-core/32-GiB server
> produce results that belong in a report. The harness itself says so in every summary
> whose host looks containerised, headless, or smoke-mode.

---

## What it measures

PLAN.md's six workloads, in PLAN.md's order and numbering:

| # | id | what runs | capacity class |
|---|---|---|---|
| 1 | `w1_idle_tab` | idle **authenticated** tab after stabilization | browser_unstreamed |
| 2 | `w2_navigation` | ordinary Playwright navigation/form work | browser_unstreamed |
| 3 | `w3_heavy_app` | heavy modern web app, multiple tabs | browser_unstreamed |
| 4 | `w4_stream_1080p` / `w4_stream_1080p_hw` | one 1080p interactive stream, x264 then hardware | streamed_browser |
| 5 | `w5_sandbox_plus_stream` | full sandbox plus streamed browser | full_ui_sandbox |
| 6 | `w6_restore_storm` / `w6b_turn_relayed_stream` | restore/start storms; TURN-relayed streams | burst / streamed_browser |

Per step it records every metric PLAN.md names: p50/p95/p99 host CPU (plus **steal**),
resident memory, disk read/write and utilisation, browser start latency, restore latency,
automation action latency, encoder utilisation, bitrate, input latency, crash rate, and
kernel PSI. Noisy-neighbour impact is read directly off the ramp: each workload's own
level-1 numbers are the quiet baseline, and every higher level is the same work under
neighbours — that is what the `action_latency_vs_baseline` guardrail is.

Guardrails, all from PLAN.md: host CPU 75 %, memory 80 %, disk p95 saturation, encoder
80 %, p95 input latency 250 ms, p95 action latency ≥ 2× single-run baseline, any OOM/crash.
The ramp **stops at the first crossing** and reports the last passing level.

---

## Prerequisites

| Workload | Needs | If missing |
|---|---|---|
| 1, 2, 3, 6-storm | python3 ≥ 3.9, `playwright` (python) + a chromium build under `PLAYWRIGHT_BROWSERS_PATH`, `Xvfb` | skipped, with the install command in the summary |
| 4 (x264) | a Selkies image (`P6_SELKIES_IMAGE`) **or** `ffmpeg` + `Xvfb` for the encoder proxy | skipped |
| 4 (hardware) | NVIDIA GPU + driver + an ffmpeg with `h264_nvenc` | skipped |
| 5 | `docker` + `P6_SANDBOX_IMAGE` (the REAL sandbox image) + a stream plane | skipped |
| 6-TURN | Selkies + `P6_TURN_URL` + `--input-latency-cmd` | skipped |
| input-latency guardrail | `--input-latency-cmd` — a command printing one number (ms) per call | that guardrail stays UNMEASURED |
| CPU credit balance | burstable EC2 + `aws` CLI + CloudWatch `GetMetricStatistics` permission | steal% recorded as a labelled **proxy**, never as a balance |

Install the browser half on a bare host:

```bash
pip install playwright
PLAYWRIGHT_BROWSERS_PATH=/opt/pw-browsers playwright install chromium
apt-get install -y xvfb            # ffmpeg too, if you want the encoder proxy
```

---

## Running it

```bash
cd packages/matrx-scraper/proofs/cloud_browser/p6_capacity

python3 run_capacity.py --smoke                 # ~5 min: proves the harness works
python3 run_capacity.py                         # the real ramp (see timing below)
python3 run_capacity.py --soak-hours 6          # ramp, then the burstable soak
python3 run_capacity.py --only w3_heavy_app     # one workload
python3 run_capacity.py --max-concurrency 24    # cap the ramp
```

Useful flags: `--nodes N` (fleet size for the one-node-failure reserve), `--levels`,
`--stabilize` / `--measure`, `--headless` (last resort — **not** the production shape),
`--force-burstable` (IMDS blocked but you know it is a t-family host),
`--out <dir>`, `--input-latency-cmd '<cmd printing ms>'`.

Environment: `P6_SELKIES_IMAGE`, `P6_SANDBOX_IMAGE`, `P6_TURN_URL`,
`P6_DOCKER_EXTRA_ARGS`, `P6_HEAVY_ROWS`, `PLAYWRIGHT_BROWSERS_PATH`.

### Roughly how long

Each step is `stabilize (60 s) + measure (120 s) + teardown (~15 s)` ≈ **3¼ min**, and a
workload runs one step per level until a guardrail crosses.

| run | time |
|---|---|
| `--smoke` | ~5 min total |
| one browser workload to a ceiling of ~8 | ~20 min |
| all runnable workloads, full ramp, 4-vCPU host | **1–2 h** |
| `--soak-hours 6` | + 6 h (**required** on t2/t3/t4g before any admission number is trusted) |

Run it on an otherwise idle host: it measures the whole host, so a parallel build lands
in your results.

---

## Reading the output

```
out/<run-id>/
  host.json                     host shape, image + tool versions (goes into RESULTS-TEMPLATE.md)
  steps/<workload>__c<NNN>.json one file per concurrency step, INCLUDING every raw sample
  soak.json                     the soak step, when --soak-hours was used
  summary.json                  everything, machine-readable
  SUMMARY.md                    the human read, pre-shaped like RESULTS-TEMPLATE.md
```

Read `SUMMARY.md` top-down:

1. **Completeness** — `COMPLETE` or `INCOMPLETE` plus exactly what is missing. An
   `INCOMPLETE` run is a partial answer and must be reported as one; the harness will not
   let a skipped stream workload disappear into a green-looking page.
2. **Per-workload table** — last passing level, first failure level, which guardrail bound.
3. **Admission control** — the per-node limit already computed.

`summary.json → workloads[].steps[].guardrails[]` is the audit trail: every guardrail,
its observed statistic, its threshold, and `crossed: true | false | null`, where **null
means unmeasured** — never "fine".

### The admission number

PLAN.md: *70 % of the lowest measured failure/guardrail count, rounded down, and reserves
capacity for one worker/node failure.* The harness computes, per capacity class:

```
per_node_limit = floor(0.70 × lowest failure level across that class's workloads)
fleet_limit    = per_node_limit × (nodes − 1)        # one node held in reserve
```

It prints the alternative reading (70 % of the last *passing* level) beside it rather than
silently choosing — see ambiguity **A2** below. With `--nodes 1` the fleet limit is `0`,
which is not a bug: that is what reserving a node on a one-node fleet means.

---

### What is already in `out/` (harness evidence, NOT capacity findings)

Both directories were produced on the authoring **session container** (4 vCPU / 16 GiB,
no AWS, no GPU, no ffmpeg). They exist to show the harness runs and the JSON is well
formed. **Do not quote a number from either.**

- `out/smoke1/` — `--smoke`, all eight workload entries: 1, 2, 3 and the 6-storm half
  ran; 4 (x264 + hardware), 5 and 6b skipped with reasons.
- `out/ramp-check/` — workload 2 across 1 → 2 → 4 → 8: crossed `host_cpu_sustained` and
  `action_latency_vs_baseline` at 8, last passing 4, admission 5/node and 10 across
  `--nodes 3`. That is the ramp, the guardrails and the 70 % rule working end to end.

## Where the plan needed interpretation

Recorded here rather than resolved silently. If any of these is wrong, fix PLAN.md and
`harness/config.py` together.

- **A1 — "disk p95 saturation" has no number.** Pinned as p95 of device utilisation
  ≥ 95 % (`DISK_UTIL_SATURATED_PCT`), measured from `/proc/diskstats` `io_time`.
- **A2 — "70 % of the lowest measured failure/guardrail count" is ambiguous** between the
  level that failed and the last level that passed. The harness uses the failure level and
  prints both.
- **A3 — "sustained host CPU 75 %"** is implemented as the p95 of the measurement window,
  with the stabilization window discarded, so one spike cannot end a ramp.
- **A4 — input latency is not measurable by this harness.** It needs a real stream client;
  the `--input-latency-cmd` seam is where the operator's own probe plugs in. Without it
  workloads 4/5/6b cannot satisfy PLAN.md's 250 ms guardrail, and they say so.
- **A5 — CPU credit balance is not readable from inside an instance.** Only CloudWatch has
  it. Without the `aws` CLI the harness records steal% as an explicitly labelled *proxy*.
- **A6 — noisy-neighbour impact** is not a separate number; it is the ramp's
  action-latency-vs-baseline comparison. A separate isolated canary would need a reserved
  core to be meaningful.

## What this harness deliberately does NOT prove

- Nothing about WebRTC, TURN, or the auth gateway (that is P2/P5).
- Nothing about AES-GCM/KMS checkpoint cost — the storm restores a plain tar (P3 owns the
  envelope). Add its decrypt step before quoting restore SLAs.
- Nothing about a real Selkies image's behaviour unless you point it at one.
- Nothing about multi-host scheduling, only per-host capacity.
