# FEATURE — Persistent Cloud Browser worker runtime (WS-2)

The minimal **browser-worker deployment role**: one process serving exactly one
persistent Chromium profile for exactly one run, behind the **S2 worker command
protocol**. This is the server half of S2; the client half is the Browser Manager
(WS-5) — or, in tests, the in-process `StubControlPlane` shipped here.

**Cross-repo authority:** the protocol is
`common-docs/projects/persistent-cloud-browser/contracts/S2-worker-protocol.md`
and the architecture is that project's `PLAN.md`. This doc is the code-side contract;
where the two disagree, S2/PLAN win and the disagreement is filed in that project's
`DECISIONS.md`.

## What it is

`BrowserWorker` ([runtime.py](runtime.py)) exposes the eight S2 operations as async
methods taking the typed request models and returning the typed response models
(`bootstrap`, `heartbeat`, `command`, `observe`, `capture`, `controller_transition`,
`checkpoint`, `shutdown`), plus the reverse worker-event channel. A thin HTTP layer
would wrap these one-for-one (S2 §2 transport), but **the operations are the
contract** and are driven directly by the control plane — so the whole worker is
drivable and testable with **zero Browser Manager, zero DB, zero streaming**.

- **`launch_persistent_context`** against the orchestrator-mounted `user_data_dir`,
  with the **D-5 keyring-free cookie scheme** (`--password-store=basic` +
  `--use-mock-keychain`) so cookies use the basic obfuscated-file store, not a
  system keyring the worker cannot reach.
- **The egress guard is installed on the CONTEXT before the first page**, reusing
  `matrx_scraper.ai_browser.url_guard.install_egress_guard` verbatim — the same
  three-layer SSRF defence the stateful browser sessions use.
- **The ONE ordered command queue** (concurrency 1) with fencing-token + sequence +
  replay verification. Its 64-response replay window is persisted as sanitized
  receipts inside the profile directory, so an encrypted checkpoint carries the
  execute-once fence to a replacement host. A corrupt receipt fails bootstrap
  closed; command arguments are never written to that file.
- **Idle age is run-scoped.** Every accepted bootstrap resets `_last_activity`; a
  fixed-fleet process never carries the previous run's idle time into a new run.
- **Reset is a lock-release backstop.** `_reset_after_stopped_run()` idempotently
  releases the current `ProfileLock` before clearing it, so a partial exit cannot
  leak a flock for the lifetime of the reusable worker process.
- **Bootstrap is bounded below the control-plane deadline.** Chromium launch gets
  45 seconds and partial Playwright cleanup gets 5 seconds. A wedged external
  process therefore returns a typed `browser_crashed` refusal, releases the
  profile lock, and leaves the fixed-fleet HTTP worker available for health probes
  instead of timing out at the manager and losing the sole ECS task.
- **Two run modes:** `handoff_capable` (headed on the private Xvfb, encoder asleep)
  and `automation_only` (headless). The mode is fixed at bootstrap — Chromium cannot
  convert headless to headed in place (S2 §12.1).
- **Page/popup/dialog/download tracking** over `context.pages`, with reconciliation
  after a human episode (survivors, last-human-focused active-page selection, dialogs
  never auto-dismissed).
- **Local advisory profile lock** ([profile_lock.py](profile_lock.py)) — a
  `flock`-based fence; a second worker refuses to launch with `profile_locked_locally`.
- **The checkpoint hook** — closes Chromium cleanly, archives the closed profile,
  hashes it, encrypts with the manager-supplied DEK, and zeroizes. WS-3 owns the full
  crypto/verify/upload engine; the worker holds the plaintext DEK only for the archive
  operation (S2 §5.7 / OPEN item 3).
- **Presigned URLs are secret transport values.** Checkpoint upload failures log only
  the HTTP status, bounded S3 error code, and request ID; exception strings and URLs
  are forbidden because HTTP client errors include the full signed query.
- **Authenticated HTTP transport** — [http_app.py](http_app.py) exposes the eight
  operations one-for-one and delegates directly to this runtime. The public-key
  verifier in [auth.py](auth.py) checks audience, operation, expiry, and replay
  without giving the worker the signing key.
- **The worker image is opt-in.** Build
  `packages/matrx-scraper/Dockerfile --target browser-worker`; an untargeted
  build always resolves to the final `scraper-service` stage. The production
  scraper application relies on that default and must never boot this worker.

## The command vocabulary is DERIVED, not invented

[commands.py](commands.py) mirrors `matrx_scraper.ai_browser.actions` one-for-one and
**imports the result models** (`NavigateResult`, `ClickResult`, …) rather than
re-declaring them — a second declaration is a defect under the one-deserializer law.
`session_id` on every result is pinned to `run_id` (S2 §7.2). The six documented
divergences (D1–D6) are honored in `runtime._execute_command`. Four additions
(`activate_page`, `close_page`, `handle_dialog`, `wait_for_download`) express the
tab/popup/dialog/download world the single-page `actions.py` pointer cannot.

## The two failure planes (S2 §9.1) — load-bearing

- **Result-level:** the browser tried and the page said no (selector missing,
  timeout, SSRF gate). `ok=true`, `result.success=false`, `result.error_type` from the
  existing `actions.py` set. An AI loop recovers from these.
- **Envelope-level:** the protocol/state said no (stale token, wrong order, human in
  control). `ok=false` + a typed `WorkerError` from [errors.py](errors.py).

A worker that mixes the planes fails conformance.

## What the worker never holds (S2 §2.4)

No DB/KMS/Vault key, no broad storage credential, no `user_id`/`organization_id`/
credential/stream ticket. Human identity reaches it only as `controller_ref`. The
audit-facing facts pass through [sanitize.py](sanitize.py): query strings stripped,
labels capped, `HumanEpisodeSummary` carries counts + origins only — "not a keylogger
transcript."

## Auth seam

[auth.py](auth.py) is a thin `TokenVerifier` seam. Production hands the worker a JWT
verifier over `matrx_connect.ScopedTokenIssuer`'s public key (a host concern; NOT
imported here). `InMemoryTokenAuthority` is the stub's issuer+verifier — audience,
operation, expiry, and single-use `jti` replay, zero external crypto.

## Files

| File | Role |
|---|---|
| `runtime.py` | `BrowserWorker` — the eight operations, queue, fencing, controller machine, page tracking, events, checkpoint hook |
| `models.py` | S2 wire models (envelopes, state, page inventory, per-op request/response) |
| `commands.py` | `BrowserCommand` discriminated union + imported `actions.py` result models |
| `errors.py` | `WorkerError` catalogue + `WorkerProtocolError` |
| `sanitize.py` | the ONE `safe_url` / `origin_of` / `cap_label` privacy boundary |
| `profile_lock.py` | `flock` advisory single-host profile lock |
| `auth.py` | `TokenVerifier` seam + `InMemoryTokenAuthority` stub |
| `http_app.py` | Authenticated FastAPI transport for all eight worker operations |
| `stub_control_plane.py` | `StubControlPlane` — in-process Browser Manager stand-in |

## Tests

`packages/matrx-scraper/tests/test_cloud_browser_worker.py` — the WS-2 conformance
suite, driven entirely by `StubControlPlane`. Proves the Definition of Done: ordered
queue semantics, fencing rejection of stale tokens, both run-mode behaviours, page
reconciliation after a simulated human episode, and the `browser_controlled_by_human`
typed conflict — plus replay-once, the two failure planes, the command divergences,
bootstrap idempotency + advisory lock, replay across worker migration, inbound auth,
and access-revocation on heartbeat. Chromium-backed tests skip when the browser binary is absent;
`handoff_capable` tests additionally require Xvfb.

Run: `uv run --project packages/matrx-scraper --extra browser pytest
packages/matrx-scraper/tests/test_cloud_browser_worker.py`.

## Known limits (read before claiming a surface done)

- **Error responses carry op-specific success fields as defaults.** To construct a
  typed refusal of the correct response class, op-specific SUCCESS fields
  (`CommandResponse.result`, `active_page_id`, …) are Optional with defaults. The
  envelope is always complete; the success payload is meaningful only when `ok=True`.
  This is a deliberate, documented pragmatic reconciliation with S2's "each op returns
  its response type."
- **Checkpoint is a hook, not the WS-3 engine.** Archive + hash + AES-256-GCM (when
  `cryptography` is present) + zeroize; real KMS wrap/verify/upload and the manifest's
  full field set are WS-3.
- **`reopen_required` (automation_only → human-required) is modelled but not
  auto-triggered** — the worker detects human-required conditions through the
  `emit_handoff_requested` seam an adapter/heuristic drives; the automatic detector is
  WS-7/WS-10.
- **Human-episode pointer/keyboard activity buckets** are counted only when the human
  plane feeds them; the WS-4 streaming plane is where real input flows.
- **`OPEN` items from S2 §14** (bootstrap credential TTL, presigned-PUT vs relay,
  plaintext DEK to the worker, dialog text, nav-history commands, reopen run_id) are
  unresolved rulings — the worker implements the recommended defaults and defers.

## Change log

- **2026-08-27 — bounded launch failure:** production proved that clearing stale
  lock files was insufficient when Chromium bootstrap itself wedged. Startup and
  cleanup are now bounded below the manager HTTP deadline, with exact retry proof.

- **2026-08-23 — clean fixed-fleet reuse:** accepted bootstrap resets worker activity,
  and terminal reset releases any lingering profile lock before clearing state.
  A reused worker cannot immediately idle-stop or remain poisoned by a prior run.
- **2026-08-21 — fail-closed login detection without ordinary-form false positives:**
  MFA handoff now requires a visible field with an explicit one-time-code contract
  or exact OTP/verification/MFA identity. Numeric inputs and names merely containing
  `code` are not authentication evidence; hidden template fields are ignored.

- **2026-08-20 — checkpoint upload log boundary:** rejected S3 uploads preserve the
  useful status/error code/request ID without emitting presigned query credentials.
- **2026-08-20 — durable replacement state:** replacement bootstrap accepts the
  current fencing revision/token and restores the durable controller state, handoff,
  and human-input gate. It never falls back to revision 1 or agent control after a
  live takeover.

- **2026-08-20 — real-image P6 capacity mode:** the production browser-worker image now
  has an explicit `P6_BENCHMARK_MODE=1` entrypoint. It boots the real BrowserWorker,
  headed Chromium/Xvfb runtime, ordered command path, and production Selkies supervisor,
  emitting the P6 JSONL contract. Ordinary worker startup is untouched; there is no
  schedule or background benchmark.
- **2026-08-19 — explicit image role:** kept `browser-worker` target-only and
  restored `scraper-service` as the Dockerfile's default final stage.
- **2026-08-19 — fixed-fleet reuse:** after a clean shutdown releases the profile
  lock, the worker clears the prior run's identity, controller, replay, page, callback,
  and session state and accepts the next bootstrap. A second bootstrap while a run is
  still live remains refused.
- **2026-08-18 — network transport:** added the authenticated HTTP application and
  public-key verifier so a remote control plane can drive a worker while replayed
  calls remain rejected.
- **2026-08-18 — created (WS-2):** worker runtime + StubControlPlane + 14-case
  conformance suite. All green against chromium-1194 headless and Xvfb headed.
