# Direct canonical site crawler

The Matrix marketing crawler is executed by the standalone scraper service.
The browser calls that service directly; AI Dream is not a router, data API, or
persistence intermediary for this feature.

**This is the ONLY crawler.** The retired `scraper.crawl_*` / `scraper.sites` /
`scraper.site_pages` admin crawler is gone — never write to it, never treat it as
a fallback. Engine-level contracts (fetch, parse, browser, SSRF gates, tools)
live in [`../FEATURE.md`](../FEATURE.md).

Verified against live code 2026-08-09.

## Non-negotiable boundaries

- Stored data is read by the frontend directly from Supabase `web.*`.
- Python exposes new-work commands and their original live NDJSON response,
  plus exactly the DERIVED read shapes below. It exposes no site, page,
  snapshot, crawl-history, replay, or result ROW endpoint — "give me these rows
  back" is a Supabase read, and adding one here re-opens the DB-gateway path
  this crawler exists to avoid.
- Every command first validates the caller's access through the site's canonical
  RLS composition root.
- The detached worker writes canonical rows with the service database role. This
  is required because authenticated clients have SELECT-only grants on
  append-only `web.crawl_url` and `web.crawl_event`.
- New crawler output never writes or falls back to legacy `scraper.*` tables.
- The client disconnecting never cancels a crawl. Only the explicit cancel
  command requests cancellation.
- Every graceful worker cancellation marks its session failed before exiting.
  A loud stale-session reaper marks any queued/running row with no persisted
  activity for 30 minutes failed, so deployments and hard process exits cannot
  leave the UI reporting `running` forever. It runs at service startup AND in
  its own 10-minute loop (`_run_stale_session_reaper`), deliberately SEPARATE
  from the crash-resume loop: `resume_crashed_sessions` awaits each continued
  crawl to completion and a full-site crawl runs for hours, so while the two
  shared one loop the reaper was starved for exactly that long — the 30-minute
  promise silently became "reaped at the next deploy" (live: session `c5f5d8c6`
  sat `queued` for 76 minutes on 2026-08-15). The crash-resume loop still reaps
  immediately before reading its candidates, because a session is only resumable
  once it has been marked failed; that call is cheap and idempotent. Pinned by
  `tests/test_stale_session_reaper_independence.py`.

## Site lifecycle owns every descendant

`web.site.deleted_at` is the lifecycle root for every `web.*` row with a
`site_id` foreign key. A live child of a soft-deleted site is invalid.

- The database trigger `_cascade_soft_delete_descendants` copies the site's
  exact deletion timestamp to every still-live site descendant in the same
  transaction. Restoring the site restores only descendants carrying that
  exact cascade timestamp, so an independently deleted child stays deleted.
- Component writes reject an insert or restore that would leave
  `deleted_at IS NULL` beneath a soft-deleted site. This guard lives in the
  database because clients write canonical rows directly through Supabase;
  an aidream-only service step cannot cover every write path.
- Append-only crawl facts remain immutable. Their mutation guards allow only a
  `deleted_at` lifecycle transition; business data still cannot be rewritten.
- New `web.*` tables with a `site_id` foreign key and soft delete must be added
  to the cascade and parent-liveness guard. The ORM backfill script validates
  its inventory against generated `Site._inverse_foreign_keys` and fails loud
  on drift.

Historical repair is idempotent and dry-run by default:

```bash
uv run python scripts/backfill_soft_deleted_web_site_descendants.py
uv run python scripts/backfill_soft_deleted_web_site_descendants.py --apply
```

## Command endpoints

All endpoints require a Supabase user access token and are mounted below the
scraper service's `/api/scraper` prefix.

| Method | Path | Body | Meaning |
|---|---|---|---|
| `POST` | `/crawler/sites/{site_id}/sessions` | `CrawlStartRequest` | Create a canonical session, start crawling, stream events |
| `POST` | `/crawler/sessions/{session_id}/resume` | no body | Continue a CRASHED session from its durable frontier — rebuilds the persisted request (`max_pages` shrunk by pages already fetched), reattaches to the SAME batch execution's pending work items, streams events exactly like `/sessions`. `409` when a run of this session is already active anywhere (this broker, another process's live lease, or a lost lease compare-and-swap); `422` when the session is not resumable (finished, cancelled, short-run mode, resume-attempt cap) |
| `POST` | `/crawler/sites/{site_id}/bootstrap` | no body | One-page browser crawl that captures the site's homepage screenshot |
| `POST` | `/crawler/sites/{site_id}/pages/fetch` | `PageFetchRequest` (`url`, optional `capture_screenshot`) | On-demand capture of ONE page (existing or brand-new URL): snapshot + head_tags + seo_metrics + viewport screenshot; scope mode `page_fetch`; no crawl discovery |
| `POST` | `/crawler/sites/{site_id}/initialize` | no body | Idempotently initialize homepage facts, sitemaps, screenshots, and pending brand candidates |
| `POST` | `/crawler/sites/{site_id}/sitemaps/sync` | no body | Sync the sitemap graph into `web.sitemap` / `web.page` / `web.page_sitemap`, stream progress |
| `POST` | `/crawler/sites/{site_id}/gsc/sync` | no body | Sync the bound Google Search Console property into `web.gsc_page_stat` / `web.page`, stream progress |
| `POST` | `/crawler/sites/{site_id}/urls/reconcile` | no body | Recompute every observed URL alias, redirect, and declared canonical into one flattened page identity |
| `POST` | `/crawler/sites/{site_id}/links/resolve` | no body | Backfill `web.link_edge.target_page_id` for internal edges, stream progress |
| `POST` | `/crawler/sites/{site_id}/links/check` | no body | Populate `web.link_edge.http_status` (internal from crawl outcomes, external cache/live), stream progress |
| `POST` | `/crawler/sites/{site_id}/links/score` | no body | Recompute `web.page.link_score` (internal-link PageRank) for the site, stream progress |
| `POST` | `/crawler/sites/{site_id}/analyze` | no body | Run the deterministic page-analysis catalogue over stored evidence, writing `web.analysis_result` + `web.finding`, stream progress |
| `POST` | `/crawler/sessions/{session_id}/cancel` | no body | Request explicit cancellation |
| `GET` | `/crawler/user-agents` | — | The named `user_agent` choices a crawl-config UI renders (see below) |
| `POST` | `/crawler/sites/{site_id}/rescrape` | `RecrawlRequest` (optional `preset_id`) or no body | One-click "crawl this site again" — derives the config, then behaves exactly like `/sessions` |

## User-Agent override (`CrawlStartRequest.user_agent`)

One optional field on the start request sets the User-Agent for the **entire**
crawl. `None` (the default) means **no override**: every transport sends exactly
what it sent before the field existed, so an omitted `user_agent` is
byte-identical to the pre-override behaviour.

**Why it exists.** These are authorized FIRST-PARTY crawls of sites we control —
which is why `respect_robots` defaults to `false`. The override is not about
evading anyone; it serves three real jobs:

1. **Identify ourselves honestly** in the customer's server logs (`Our crawler`).
2. **Reproduce what a specific bot or device sees** — the page Google's mobile
   crawler reads is routinely not the page a desktop browser gets.
3. **Get through the customer's own WAF/CDN rule** that blocks unknown agents.

### The invariant: one crawl, ONE identity

The override is applied to **every** request the crawl makes:

| Where | How it lands |
|---|---|
| robots.txt matching + sitemap discovery | `SiteCrawler.user_agent` (override, else the bot default) |
| HTTP page fetch (curl_cffi **and** httpx) | replaces `User-Agent` in the rotating header profile; every other profile header is untouched |
| Browser page fetch (`browser_pool.fetch`) | Playwright **context** option `user_agent` |
| Browser capture (`fetch_with_capture` / `capture_url`) | replaces the device profile's UA, per navigation, for **every** device profile in the capture |

**A UA that reached only one of those would be worse than none** — the same crawl
would present two identities depending on whether a page happened to escalate to
the browser, and a customer's WAF allowlist would pass half of it. Pinned by
`tests/test_user_agent_override.py`, which asserts against the header a **real
local HTTP server received**, not against an intermediate kwarg.

Two deliberate consequences:

- **The override replaces the UA and NOTHING else.** A mobile capture taken "as
  Googlebot" keeps its mobile viewport, `is_mobile`, `has_touch`, and device
  pixel ratio — those are the device's geometry, not its identity
  (`DeviceProfile.context_kwargs(user_agent_override)`).
- **An overridden fetch bypasses the scrape cache**, read and write. The cache is
  keyed on URL alone; serving a Chrome-fetched body to a Googlebot request would
  silently defeat the whole point.

### Validation — one gate, `matrx_scraper/user_agents.py`

`normalize_user_agent()` is the ONE definition, shared by the request contract
and every transport. A UA the request accepts but a transport refuses would fail
per-URL, mid-crawl, deep inside httpx or Playwright.

- **Empty or whitespace-only → `None`**, meaning "use the default". It NEVER
  becomes an empty `User-Agent:` header — an empty UA is not neutral, it is its
  own frequently-blocked signal.
- **Control characters rejected** (checked on the raw value, so a trailing
  newline cannot hide behind the `strip()`). A bare CR/LF here is header
  injection.
- **Bounded at `MAX_USER_AGENT_LENGTH` (512).** Real agents top out near 200; the
  value is stored in `web.crawl_preset.config` forever.
- **latin-1 only** — header values are latin-1 on the wire.

There is **no env var** for this, by doctrine. The default is the absence of an
override; `MATRX_CRAWLER_USER_AGENT` is a CAPS constant in `user_agents.py`.

### Presets — what a non-technical user actually picks

The person configuring a crawl is a Subject Matter Expert, not a browser
engineer. They choose **"Googlebot"**, never a 90-character `Mozilla/5.0 …`
string. `GET /crawler/user-agents` serves the named list — `Default
(recommended)` · `Our crawler` · `Googlebot` · `Googlebot (mobile)` · `Bingbot` ·
`iPhone Safari` · `Chrome on Windows` — each with a plain-language description.

**A UI renders that list as the primary control; the raw string field is the
escape hatch, never the primary affordance.** The list is served rather than
hard-coded per client so the label a user reads and the value the API accepts
cannot drift. `Default` carries `value: null` — "do not override" is a real
choice and is NOT the same as sending our bot UA.

### Presets storage — free

`web.crawl_preset.config` **IS** a `CrawlStartRequest`, so the override is saved,
listed, and replayed by one-click rescrape with no preset-side work at all.

### Known limitation

The HTTP transport's curl_cffi TLS fingerprint still impersonates the header
profile's Chrome build, so a request claiming to be Googlebot presents a Chrome
TLS handshake. This is correct for the first-party use cases above (honest
identification, WAF allowlisting, device reproduction) and deliberately NOT
"fixed" — matching a search engine's network fingerprint is evasion, which is not
what this field is for.

## Crawl presets and one-click rescrape (`web_crawl/presets.py`)

A **preset** is a saved, named `CrawlStartRequest` bound to one site, stored in
`web.crawl_preset`. The stored config **IS** the request model — there is no
parallel "preset config" shape, so a preset can never encode a crawl the
crawler cannot run, and adding a knob to `CrawlStartRequest` adds it to presets
for free.

| Method | Path | Body | Meaning |
|---|---|---|---|
| `GET` | `/crawler/sites/{site_id}/presets` | — | The site's presets, most-recently-used first, plus the pinned `default_preset_id` |
| `POST` | `/crawler/sites/{site_id}/presets` | `CrawlPresetSaveRequest` | Create or replace by name (upsert on `(site_id, name)`) |
| `DELETE` | `/crawler/sites/{site_id}/presets/{preset_id}` | — | Soft-delete |
| `GET` | `/crawler/sites/{site_id}/recrawl-config` | `?preset_id=` | What a rescrape WOULD run, and where the config came from |

**Rescrape resolution order** — most specific first, and every answer reports
its `source`:

1. `preset` — an explicitly named `preset_id`,
2. `site_default_preset` — `web.site.settings.crawl.default_preset_id`,
3. `last_session` — the newest `full`/`list` session's persisted `scope["request"]`
   (the same blob resume rebuilds from, so a rescrape reruns what last ran),
4. `defaults` — `CrawlStartRequest()`.

Rules that exist because their absence is silent:

- **A named preset that does not resolve is an ERROR, never a downgrade to
  defaults.** Falling through would run a *different crawl* than the one the
  user asked for, and nothing would say so. A *pinned* default that was deleted
  does degrade — it must not brick the button — but screams in the log.
- **A stored config that no longer validates stays VISIBLE and flagged**
  (`config_error` + `raw_config`), never dropped from the list and never
  coerced. Dropping it makes the user's saved work vanish; coercing it runs a
  crawl nobody configured.
- **`use_count` is bumped only AFTER the crawl is accepted**, with an atomic
  `F` expression — a refused start is not a use, and two concurrent starts
  cannot lose an increment.
- **Authorization is the site's, always** — `viewer` to read, `editor` to
  write, enforced by RLS on `web.crawl_preset` inside the claim-bound session.
  There is no per-preset owner or `is_shared` flag: sharing a site shares its
  presets, through the one platform sharing kernel.

`web.crawl_preset` is a standard site component (`_rls_variant="component"`,
entity token `web_crawl_preset`, soft delete, versioned, org stamped from the
parent site). It replaced `scraper.crawl_presets` on 2026-08-09; the legacy
table's `base_url` / `user_agent` keys have no canonical equivalent —
`base_url` is the site's own `root_url`, and the canonical crawler has no
user-agent override at all.

## Recurring crawls — `web.crawl_schedule` (`web_crawl/schedules.py`)

A **schedule** is a site saying "crawl me again, on this cadence". It is
deliberately almost empty: `cadence`, `timezone`, and an optional `preset_id`.

**It carries NO crawl knobs.** What a scheduled crawl runs is resolved by the
same `derive_recrawl_config` a human clicking Rescrape goes through (preset →
site default preset → last full/list session's request → defaults), and it
starts through the same `WebCrawlService.prepare_rescrape` → `run_prepared`.
There is no second way to start a crawl and no second config resolution — which
is why the columns that once decomposed the config here (`scope`,
`respect_robots`, `screenshot_policy`, plus the unused `scheduler_task_id`) were
DROPPED on 2026-08-09 rather than wired up.

**Cadence** is a discriminated union stored in the `cadence` jsonb, and the
column has no default — a row inserted without one is rejected by
`crawl_schedule_cadence_kind`, because a schedule with no cadence is a row that
silently never fires:

| Shape | Meaning |
|---|---|
| `{"kind": "cron", "expression": "0 3 * * *"}` | Evaluated in the row's own `timezone`, so "3am daily" stays 3am across DST |
| `{"kind": "interval", "minutes": 1440}` | A fixed gap after the occurrence that just fired (never after "now", so a slow drain cannot make a schedule drift later every tick). Floor: `MIN_INTERVAL_MINUTES` (15) |

**Claim and lease.** `next_run_at` IS the queue; nothing about a due occurrence
lives in process memory. A dispatcher claims with ONE
`UPDATE … WHERE id IN (SELECT … FOR UPDATE SKIP LOCKED)` (`Model.claim_batch`)
that stamps `claim_token` + `claim_expires_at`, gated on `enabled`, not
soft-deleted, `claim_token IS NULL`, `next_run_at <= now()`, and `created_by IS
NOT NULL` (a schedule with no creator has no identity to crawl as). Two app
servers draining at the same instant therefore get disjoint rows and neither
blocks the other. Every drain first RELEASES leases whose `claim_expires_at` has
passed, so a dispatcher that died holding a claim strands nothing. Delivery is
**at-least-once** by construction — see the skip rule for why that is safe.

**Settlement is a compare-and-swap on `(id, claim_token)`** — a lease already
reclaimed by someone else is reported, never clobbered — and the three outcomes
are deliberately different things:

- **`started`** — a crawl session exists. Stamps `last_run_at` +
  `last_session_id`, clears `last_error`, resets `consecutive_failures`. This
  happens BEFORE the crawl detaches: the session is already durable, so
  stamping it is honest even if the process dies immediately, and it is what
  stops the occurrence from firing twice.
- **`skipped`** — `prepare_start` refused because the site is already crawling
  (the one-active-crawl rule, 409 "already active"). **Nothing is wrong**, so
  the failure streak is untouched and `last_run_at` is left alone — nothing ran.
  This is also what makes at-least-once delivery harmless: a re-fired
  occurrence lands on the still-active session and records a skip.
- **`failed`** — anything else. Increments `consecutive_failures` and DISABLES
  the schedule at `MAX_CONSECUTIVE_FAILURES` (10); a schedule that keeps failing
  needs a human, not another attempt every minute forever.

All three advance `next_run_at`. A cadence that cannot be advanced (invalid
cron, unknown timezone) disables the row with the reason in `last_error`,
because a schedule that cannot move would re-fire on the same instant forever.
An enabled row with a NULL `next_run_at` is seeded from its cadence at the top
of each drain — the same dead-on-arrival failure, caught one layer earlier.

**Crash safety after the start** is the crawler's own, already solved: the
session and the frontier are durable, so a process death mid-crawl is continued
by the crash-resume sweep. The schedule row is not involved.

**Entry point:** `dispatch_due_crawl_schedules()`. aidream's every-minute
`web_crawl_schedule_dispatch` system task (`sch_task`
`a7c1e2d3-…-000000000179`, renamed from `scraper_schedule_dispatch` when the
legacy `scraper.crawl_schedules` drain was deleted) is a thin caller of it.
Scheduled sessions carry `trigger='scheduled'`, so a scheduled crawl is
distinguishable from a human's in `web.crawl_session`. Tests:
`tests/test_crawl_schedules.py`.

## Derived read endpoints (`web_crawl/insights.py`)

Each is an aggregation (group-by / rank / downsample) a client cannot assemble
from row reads, over evidence that is already canonical. **No table was added
for any of them, and none may be.** (The run-over-run diff routes follow the
same rule and are documented in their own section below.)

| Method | Path | Query | Derived from |
|---|---|---|---|
| `GET` | `/crawler/sites/{site_id}/duplicate-clusters` | `max_clusters` (100), `max_pages_per_cluster` (50) | `web.snapshot.extracted.fingerprint` |
| `GET` | `/crawler/sites/{site_id}/link-graph` | `max_nodes` (1000), `max_edges` (5000) | `web.link_edge` + `web.page` |
| `GET` | `/crawler/sessions/{session_id}/progress-series` | `max_points` (500) | `web.crawl_event` rows of type `crawl_progress` |
| `GET` | `/crawler/sites/{site_id}/traffic-at-risk` | `max_pages` (200) | `web.gsc_page_stat` + `web.page.http_status_last` |

Access reuses the crawler's own gates — `site_identity` (a site the caller
cannot SELECT under RLS raises 404) and `assert_session_access` — and every
query runs inside `repo.rls()`, so Postgres stays the authority.

🚨 **Every cap reports what it dropped.** `web.link_edge` holds ~610k rows; an
uncapped graph melts the browser and a capped one that reports only what it
kept lies about the site. `*_total` / `*_returned` / `*_omitted` +
`scan_truncated` are a CONTRACT, pinned by
`tests/test_web_crawl_transport.py::test_every_capped_read_reports_what_its_cap_dropped`.

Contract details worth knowing before changing them:

- **Clusters** group on `(fingerprint_version, exact_sha256)` — the SAME key
  `analysis._check_duplicate_content_exact` uses, so a page flagged there
  always appears in exactly one cluster here. Version is part of the key: two
  extractor generations are not comparable. Anti-drift pin:
  `tests/test_crawl_insights.py::test_cluster_membership_matches_analysis_check`.
- **Graph** nodes are canonical pages ranked and sized by `page.link_score`
  (the internal PageRank `web_crawl/link_score.py` writes). That column is NULL
  until a COMPLETED full crawl has been scored, so in-degree over the WHOLE
  site graph ranks the unscored — never mixed with scores, since an unscored
  page sorts below every scored one rather than competing on a different scale.
  The response's `ranking` field names which key actually decided, and
  `nodes_with_link_score` says how many carry a score; the client must never
  guess. `rank_and_cap_nodes` is the one place ranking changes.
  Both edge endpoints are canonical by construction (`link_resolution` writes
  `target_page_id = page.canonical_page_id`, snapshots hang off canonical
  pages), so no alias collapse is needed. Edges are scoped to each page's
  CURRENT snapshot and returned only when BOTH endpoints survived `max_nodes`,
  so every edge is drawable; parallel edges collapse to one weighted edge
  (`link_count`).
- **Traffic at risk** joins "Google shows this" to "we cannot load this" — both
  halves were already canonical; the join is what did not exist, and for most
  GSC-declared URLs it could not be computed at all until the verification sweep
  filled `http_status_last`. Ranked by impressions, because the point is the
  traffic at stake. **Two exclusions are load-bearing, not tuning:** a `429` is
  a fact about OUR request rate and is reported as `throttled_not_assessed`
  rather than as a broken page (counting them told iopbm.com 99.18% of its
  traffic was at risk when nothing was wrong), and URLs with no recorded status
  are reported as `unverified_with_impressions` so the list never implies a
  completeness it lacks.
- **Progress series** is derived from the event ledger. The legacy
  `scraper.crawl_progress_snapshots` table was a second copy of that ledger and
  must never be recreated. First and last samples always survive downsampling,
  and `pages_per_second` is computed AFTER downsampling so each plotted value
  describes exactly the span it is drawn across. A RESUMED session restarts its
  counters, so a negative delta reports no rate rather than negative throughput.

## Run-over-run diff (`web_crawl/diff.py`)

"What changed since the last crawl" — the canonical replacement for legacy
`crawl_service.diff_runs` / `get_previous_run` and `site_service.list_site_diffs`
/ `get_site_diff`. **Derived on demand; `scraper.site_run_diffs` gets no
canonical twin.**

| Method | Path | Query | Returns |
|---|---|---|---|
| `GET` | `/crawler/sessions/{session_id}/previous` | — | `PreviousSessionResponse` — the baseline to diff against, `null` on a first crawl |
| `GET` | `/crawler/sessions/{base_session_id}/diff/{compare_session_id}` | — | `CrawlDiff` — full page-level diff of two named sessions |
| `GET` | `/crawler/sites/{site_id}/diffs` | `limit` (20, max 50) | `SiteDiffList` — counts-only diff of each recent crawl vs its predecessor |
| `GET` | `/crawler/sites/{site_id}/diffs/{session_id}` | — | `CrawlDiff` — that session against its own predecessor |

- **Substrate is `web.crawl_url`, NOT `web.snapshot`.** A snapshot exists only
  for a successful capture, so a page that went 200 → 404 writes none;
  diffing snapshots would report it as *removed* instead of *broken* and lose
  `pages_status_worse` entirely. `web.crawl_url` carries the terminal outcome
  of every attempted URL (`captured` / `redirected` / `failed`) plus its
  observed status, and points at the snapshot when there is one. Title, meta
  description, content hash, and word count come from that snapshot.
- **Identity is `web.page.canonical_page_id`**, so a URL crawled under two
  spellings is ONE page on both sides and never diffs as added + removed. This
  is the standing answer to what the legacy plpgsql recomputed with
  `scraper.canonical_url(...)` on every call — and the reason no cache table is
  needed: one diff is four indexed reads, and the per-site LIST is the same
  four reads because the whole window loads in one `session_id__in` query and
  the consecutive pairs are compared in memory.
- **Five compared fields:** `http_status`, `title`, `meta_description`,
  `content_hash`, `word_count`. Content hash is a superset of the legacy DB
  function, which compared only status/title/word_count and so scored a
  rewritten page of the same length and title as UNCHANGED.
- **`pages_returned` ⊂ `pages_added`** — a URL missing from the base run whose
  `web.page.first_seen` predates that run: the site knew it before, so it came
  back rather than appearing. It is always `0` when there is no base run —
  nothing can return from a run that does not exist, and dating it from the
  compare session's own start instead would label every page a prior sitemap
  sync created "returned" (41 of 41 on a real site).
  `pages_status_worse` / `pages_status_better`
  ⊂ `pages_changed`; a 200 → 301 is a change with no direction claimed, and a
  status appearing out of `NULL` never claims a direction either.
- **Only `full` / `list` sessions are diff BASELINES** (`DIFFABLE_CRAWL_MODES`,
  mirroring `service._EXCLUSIVE_CRAWL_MODES`). A `page_fetch`, `bootstrap`,
  `analysis`, or `*_sync` session touches one page or none; choosing one as
  "the previous run" would report the entire site as removed. An explicit
  two-session diff compares whatever the caller names — only *resolution* of an
  implicit baseline filters by mode.
- **Counts are always complete; only the arrays are capped** (`MAX_LIST_ITEMS`,
  500 — the legacy cap). A session over `MAX_PAGES_PER_SESSION` (20k fetched
  URLs) sets `truncated` rather than silently comparing a partial set; that is
  the signal to revisit the no-cache-table decision, not a reason to pre-cache.
- Access reuses `assert_session_access` / `site_root` and every query runs
  inside `repo.rls()`, exactly like the insights reads. Two sessions from
  different sites are a 422, not a diff of two unrelated page sets.

The four start responses include `X-Crawl-Session-Id` and `X-Site-Id`. Events also
carry `site_id`, canonical `session_id`, and a monotonically increasing
`sequence`. `run_id` is retained as a compatibility alias and is always equal to
`session_id`.

The normal start body supports crawl limits, include/exclude patterns, list mode,
render mode, screenshots, sitemap seeding, host rate limits, and an explicit
`respect_robots` switch. The first-party product default is `false`.

Every `include_patterns` / `exclude_patterns` entry must be a valid regex —
`CrawlStartRequest` compiles each one and an invalid pattern is a 422 naming
the bad pattern, BEFORE any session is created. The crawler keeps a defensive
skip for non-contract callers, but any skip it performs emits a durable
`crawl_warning` ("the crawl runs WIDER than requested") — never a silent
log line.

Host-scope judgments use the real Public Suffix List (tldextract with its
bundled offline snapshot — never a runtime PSL fetch — including private
registry suffixes): `evil.co.uk` and `example.co.uk`, or two `github.io`
project sites, are DIFFERENT registrable domains and `follow_subdomains` can
never bridge them. www/apex equivalence (below) is unchanged.

## Homepage bootstrap contract

After `web.create_site` succeeds, the frontend calls:

```http
POST /api/scraper/crawler/sites/<site_id>/bootstrap
Authorization: Bearer <supabase access token>
```

The service creates a normal canonical `web.crawl_session` with
`scope.mode = "homepage"`, crawls only `web.site.root_url` through Playwright,
stores the body and one desktop viewport PNG, creates
`web.screenshot(kind = "homepage")`, and sets
`web.site.homepage_screenshot_id`. Failure to capture or persist the screenshot
fails the session; it does not silently produce a screenshot-less success.

## Site initialization contract

`POST /crawler/sites/{site_id}/initialize` uses the same caller JWT, site-editor
authorization, detached work lifecycle, and NDJSON transport as bootstrap. It is
safe to re-run. **Value lands in seconds, not after a minute:** the homepage is
fetched FIRST and observed identity is written to `web.site` immediately; the
remaining steps then run CONCURRENTLY, each persisting its own results the
moment it completes. A slow step never blocks an already-finished step.

1. **`identity` (first, sequential).** Fetch and persist the homepage through
   the standard snapshot pipeline (redirect chain + the canonical `/` page
   identity), extract homepage candidates once from the captured HTML +
   `PageSummary` (no extra fetch), and IMMEDIATELY fill
   `web.site.description` (meta description), `favicon_url` (best favicon
   candidate), `og_image_url` (og:image), and `logo_url` (schema.org logo
   first, else a high-confidence header logo; below the 0.8 confidence bar it
   stays null — the discovery inbox still receives every candidate). These are
   OBSERVED fields: **only null/empty columns are filled — a value the user
   may have set is never overwritten**
   (`WebCrawlRepository.update_site_identity`; the per-run
   `{written, skipped_existing}` result lands in `initialization.identity`).
   Homepage fetch failure fails the session.
2. **`sitemaps`, `screenshots`, `discovered` — concurrent.**
   - `sitemaps`: the full sitemap sync (below).
   - `screenshots`: capture `desktop_full`, `desktop_fold`, `mobile_full`,
     `mobile_fold`; PNGs go through `matrx-files` to private S3 and each
     `web.screenshot` stores only the resulting `files.files.id`. Retention
     (below) prunes superseded captures after each batch.
   - `discovered`: persist the candidates already extracted in step 1. One
     pending `web.discovered_item` per candidate in ONE idempotent statement;
     extraction preserves specific review kinds for Schema.org and visible
     homepage evidence, including `fax` (Schema.org `faxNumber` or labeled
     Fax/Facsimile text), `title`, `description`, `site_name`, `og_image`, and
     `twitter_image`. A labeled fax is excluded from generic phone matching so
     it does not enter the inbox twice under conflicting kinds.
     Favicons require observed `<link rel="…icon…">` evidence; the extractor
     never invents `/favicon.ico`. Direct `<video>` sources are media, while an
     iframe is a video only when `video_embed_provider` recognizes a known
     media provider. The provider is preserved in candidate context. Analytics,
     tag-manager, CAPTCHA, ad, and arbitrary application iframes therefore
     never enter the brand inbox as imagery. The arbiter is the DB dedup index
     `(brand_id, category, guessed_kind,
     url, value_hash) NULLS NOT DISTINCT` — `value_hash` is a STORED
     GENERATED `md5(value::text)` column that is never inserted but IS named
     in the ON CONFLICT target, so two different url-less facts can never
     collide. The batch self-dedupes on `(category, guessed_kind, url,
     canonical-json-of-value)` before the statement (ON CONFLICT cannot skip
     a duplicate arriving twice in one INSERT). IDs stay deterministic
     (uuid5 of the full identity including value). No confirmed brand/site
     field is written by candidate persistence.
3. **`url_reconciliation` (final).** Recompute canonical URL identity after
   the concurrent sitemap step and the homepage crawl have both contributed
   evidence. The durable `site_initialization_progress` stream reports this
   step; it is deliberately not a new `initialize_step` frontend-refetch name.
4. Save the step summary to `web.site.initialization` (checkpointed under a
   lock after every step) and stamp `initialized_at` after the final
   checkpoint.

### Initialize stream contract (the frontend refetches per-step on these)

Each step transition emits one NDJSON `initialize_step` data event — names and
shape are a CONTRACT (`matrx_scraper.web_crawl.contracts.InitializeStepEvent`):

```json
{
  "event_type": "initialize_step",
  "site_id": "…",
  "session_id": "…",
  "step": "identity" | "screenshots" | "sitemaps" | "discovered",
  "status": "started" | "complete" | "failed" | "skipped",
  "message": "…",
  "counts": {},
  "error": "TypeName: message — only when status=failed",
  "ts": "ISO-8601"
}
```

`counts` on `complete`, per step: `identity` → `{written, skipped_existing,
candidates}`; `screenshots` → `{captured, superseded, pruned}` (plus
`prune_failed` when retention hit errors); `sitemaps` → `{found, urls,
pages_upserted}`; `discovered` → `{media, facts, socials, links, identity}`.
The `identity` complete event is the "identity is persisted" announcement and
always precedes every concurrent step's events. The terminal summary save
still emits the durable `site_initialization_progress` event with
`step="site_update"` (status `ok`/`failed`).

A step failure is appended to `initialization.errors`, emitted as a failed
`initialize_step` event, and does not roll back completed work or block the
other concurrent steps. A dependent step that cannot run is emitted as
`skipped` with the prerequisite named in its message; it is not appended as a
second error. The root homepage failure preserves the crawler's exact terminal
cause (for example `RateLimited: HTTP 429 …`) in the durable initialization
summary rather than replacing it with a generic missing-snapshot message.
Non-blocking maintenance or partial-success notices go in
`initialization.warnings`: screenshot-history pruning after successful capture,
and individual sitemap failures when at least one sitemap document succeeded.
Warnings stay visible but never turn a completed step or the whole
initialization red.

### Screenshot retention

Screenshots keep a bounded history per `(site_id, page_id, kind)` (site-level
captures key on `page_id IS NULL`): the current capture plus at most **3
priors**. `prune_screenshot_history` (`matrx_scraper.web_crawl.persistence`)
runs after every capture batch — initialize re-runs and full crawls alike —
and soft-deletes anything older: the `web.screenshot` row gets `deleted_at`
AND its `files.files` row is soft-deleted through the canonical cascade-aware
primitive. Never a hard delete and never an S3 purge here — storage cleanup
belongs to the platform media machinery. `{superseded, pruned}` counts are
reported loudly in the step summary; per-row prune failures are counted
(`prune_failed`) and logged, never allowed to kill a capture that already
persisted. Initialize captures are **append-only** `web.screenshot` rows
(newest live row per kind wins — that is what the frontend hero resolves);
the previous update-in-place behaviour left every superseded immutable file
row floating with no referencing screenshot row.

## Sitemap sync contract

One importable operation — `matrx_scraper.web_crawl.sitemap_sync.sync_site_sitemaps`
— runs identically as the standalone `POST /crawler/sites/{site_id}/sitemaps/sync`
command (NDJSON `sitemap_sync_progress` events, its own `web.crawl_session` with
`scope.mode = "sitemap_sync"`) and as the initialize-site sitemaps step. Routers
and step wrappers hold no sync logic, so the same function can back a workflow
node later.

- Discovery: `robots.txt` `Sitemap:` lines plus `/sitemap.xml` and
  `/sitemap_index.xml`; sitemap indexes expand recursively with hard bounds —
  depth 3, 500 sitemap documents, 50,000 urlset URLs per run. Tripping a bound
  sets `truncated` in the summary and records the reason as an error; never a
  silent stop. (`matrx_scraper/sitemaps.py` owns the bounded fetch + parser.)
- Documents are fetched breadth-first in CONCURRENT waves (per-host limit 8,
  `DEFAULT_SITEMAP_FETCH_CONCURRENCY`); responses are processed sequentially
  in wave order so bounds stay deterministic and parents always precede
  children. This is what turned a 30-doc sequential walk (~30-60s) into a
  few seconds.
- One `web.sitemap` row per fetched document, upserted on the
  `(site_id, url)` arbiter with `kind`, `parent_sitemap_id` (index children),
  `status_code`, `url_count`, `child_count`, `last_fetched_at`, `last_seen`.
  A fetch failure records `fetch_error` + `status_code`/`last_fetched_at` only
  — it never clobbers what a prior successful fetch learned; a successful
  re-fetch clears `fetch_error`. A sitemap that disappears is NOT deleted; its
  `last_seen` simply stops advancing (404/410 candidates are absent, not
  errors).
- Every urlset URL goes through
  `web_crawl.url_identity.upsert_observed_page_urls`, the same matcher used by
  crawls and search-provider observations. The exact observed URL remains a
  durable alias row; membership is written against its resolved
  `canonical_page_id`. NEW rows are born with `provenance='sitemap'`;
  provenance, status, and user-owned intent fields are never overwritten.
- Membership lands in `web.page_sitemap` on the `(page_id, sitemap_id)`
  arbiter with `lastmod` / `changefreq` / `priority` / `last_seen`. Malformed
  `lastmod`/`priority` values degrade to NULL (cosmetic fields never kill a
  run). Every `<loc>` goes through ONE gate — `normalize_sitemap_loc`
  (`validate_and_correct_url` then `normalize_url`): a recoverable scheme-less
  loc (`www.example.com/page`, a routinely-seen malformation) is corrected to
  https instead of mangled or lost, while non-http(s) schemes, hostless
  values, and localhost/internal targets are skipped and counted loudly.
- All writes are batched (hundreds of rows per statement via matrx-orm
  `bulk_upsert`) and idempotent — the whole operation is safe to re-run.
- 🚨 **Every batch is de-duplicated on its own conflict key before the
  statement runs** — `dedupe_upsert_rows(rows, on_conflict=..., context=...)`
  in [`upsert_rows.py`](upsert_rows.py), last occurrence wins, loud warning
  whenever rows actually collapse. Postgres rejects an
  `INSERT ... ON CONFLICT DO UPDATE` whose VALUES list targets one key twice
  (`CardinalityViolationError`: *cannot affect row a second time*), and every
  key here is DERIVED — two distinct sitemap URLs legitimately resolve to one
  `canonical_page_id`, so `(digest, sitemap_id)` being unique does NOT make
  `(page_id, sitemap_id)` unique. This killed the whole datadestruction.com
  sync (191 URLs, 9 sitemaps) after the pages had already been written.
  Reconciliation (`_merge_sitemap_memberships`, `_merge_gsc_stats`) groups by
  the arbiter first and GSC daily stats merge duplicates numerically — a new
  `bulk_upsert` caller must do one of the three, never assume its input is
  already unique. Pinned by
  `tests/test_sitemap_sync.py::test_two_urls_resolving_to_one_page_upsert_once`.
- **Dismissal memory — upserts that mean "this thing exists again" REVIVE
  soft-deleted rows, and the revive is REMEMBERED.** Arman's ruling
  (2026-08-08): *the crawler represents reality — a user "delete" of a
  crawler-observed row is a dismissal: hide it, but if a later crawl/sync
  re-observes it, it comes back visibly AND permanently carries the fact
  that the user dismissed it before; never silently ignored, never
  forgotten.* If never re-observed, it stays buried. Mechanics:
  - The ORM upsert cannot SET a column to NULL (None values are dropped),
    so every revive is an explicit follow-up write: a successfully
    re-fetched `web.sitemap` row, a re-observed `web.page` (via
    `upsert_observed_page_urls` — observed alias AND its canonical), and
    every re-synced or reconciled `web.page_sitemap` membership.
  - User-facing rows (`web.page`, `web.sitemap`) revive with a durable
    marker appended to `metadata.dismissals`:
    `{dismissed_at: <old deleted_at>, revived_at, revive_reason:
    "reobserved", revived_by_session}` — a LIST, so repeated dismiss/revive
    cycles are all preserved (`append_dismissal_marker` in
    `url_identity.py`). Junction rows (`web.page_sitemap`) revive without a
    marker — they are not user-facing.
  - Never silent: crawl-path revives surface as a durable `crawl_warning`
    on the session ("Re-observed a page you previously dismissed…",
    `context.reason = "revived_after_dismissal"`, via
    `PersistResult.warnings`); sitemap-sync revives ride the run's durable
    errors/summary lines and `SitemapSyncResult.revived_after_dismissal`;
    every revive also logs at WARNING.
  - GSC daily stat upserts are pure measurements and deliberately do not
    revive anything. Tests: `tests/test_dismissal_memory.py`.
- The summary `{found, urls, pages_upserted, truncated}` is written to
  `web.site.initialization.sitemaps` when run via initialize, and emitted as
  the standalone stream's final `sitemap_sync_progress` (`status="ok"`) data
  event; the standalone session also stores it in `crawl_session.stats`.

## Google Search Console sync contract

One importable operation — `matrx_scraper.web_crawl.gsc_sync.sync_site_gsc` —
runs as the standalone `POST /crawler/sites/{site_id}/gsc/sync` command
(NDJSON `gsc_sync_progress` events, its own `web.crawl_session` with
`scope.mode = "gsc_sync"`, `X-Crawl-Session-Id` header) and is workflow-node
ready. The router and service wrappers hold no sync logic.

- Binding: `web.site.integrations.marketing.providers.google_search_console`
  = `{enabled, credential_ref, resource_ref}`. `credential_ref` is a
  `users.integration_connections` id — safe metadata plus a canonical-vault
  reference. This package does not own the refresh token, so resolving it
  is an **injection seam**: a host registers
  `configure_ext(google_credential_resolver=<async callable>)` and that is
  what runs. aidream wires its in-process `resolve_connection_credential`
  (same ownership + WS-6 resource-binding checks as the bridge route, one
  less network hop — before this seam an in-process crawl HTTP-called
  aidream from inside aidream); matrx-local or a customer wires their own
  Google OAuth store. **With no resolver registered** the module falls back
  to the platform bridge the Matrx standalone microservice is deployed
  alongside — `GET {AIDREAM_URL}/api/google-integrations/internal/credential`
  (service-token bridge auth: `Authorization: Bearer $AIDREAM_SERVICE_TOKEN`
  — the same value as aidream's `AIDREAM_SANDBOX_SERVICE_TOKEN` — plus
  `X-Matrx-User-Id`) — because that deployment has no vault and no host
  process to inject into. That fallback needs NO shared encryption env
  and NO Google OAuth client env: the bridge re-authorizes ownership
  (requesting user or the site's organization) and returns
  `refresh_token + client_id + client_secret` (the OAuth client the token
  was minted for). `tests/test_host_independence.py` pins that no OTHER
  module reaches for an `AIDREAM_*` variable. Decryption never happens in
  this package; the old
  AES-256-GCM path and the hand-authored `db/models_users.py` were deleted
  (2026-07-20). Service-account fallback is deliberately NOT used for user
  credentials.
- Query: Search Analytics `dimensions=["date","page"]`, `dataState=final`,
  last 28 daily buckets ending at today−2 (GSC's data lag), fully paginated
  at 25,000 rows per request.
- `fetch_search_analytics_page` is the public raw-page primitive used by the
  extended matrx-seo collector. It executes through the same `GscClient` access
  token cache/refresh path and returns exact JSON, headers, and HTTP status.
  The SEO collector adds bounded property/query/country/device/search-appearance
  histories in `seo.*`; this dashboard sync remains the canonical
  `web.gsc_page_stat` writer and its behavior is unchanged.
- Rows outside the site's host (subdomains count as in-scope — `sc-domain:`
  properties report them) are counted in `skipped_out_of_scope` and recorded
  as a summary error line, never silently dropped.
- Unknown in-scope URLs go through the one canonical URL matcher. The exact
  GSC URL is retained as an alias, while daily statistics are written to the
  resolved canonical page. NEW rows are born `provenance='gsc'`;
  provenance/status/intent are never overwritten.
- `upsert_observed_pages_for_urls` is the shared external-observation page
  primitive. GSC delegates with `provenance='gsc'`; the durable GA4 SEO
  collector delegates with `provenance='ga4'`; Bing Webmaster delegates with
  `provenance='bing_webmaster'`. All use identical conflict behavior, so no
  provider can invent a second page registry or overwrite
  user-owned page fields.
- Daily stats upsert `web.gsc_page_stat` on the `(page_id, date)` arbiter
  (`clicks`, `impressions`, `ctr`, `position`), batched.
- The property's submitted-sitemaps listing is fetched and stored in the run
  summary (`submitted_sitemaps`) for diffing against `web.sitemap`; a listing
  failure is a loud summary error, not a run failure.
- The run summary `{property, days, pages, stats_rows, submitted_sitemaps,
  skipped_out_of_scope, errors[]}` is ALWAYS written to `web.site.gsc_sync`
  — including on failure (with the error appended) before the exception
  re-raises. `web.site.gsc_synced_at` advances only on success.

## Canonical URL identity and batch reconciliation

`matrx_scraper.web_crawl.url_identity` is the single URL-identity authority for
crawl responses, sitemaps, GSC, GA4, and Bing Webmaster. No producer chooses a
page id from its own URL hash anymore.

- Every `web.page` has a non-null `canonical_page_id`. A canonical row points
  to itself; an alias points directly to the canonical row—never through an
  alias chain. A database trigger defaults new rows to self and rejects
  cross-site pointers. Exact observed URLs remain durable registry rows so
  later evidence can reverse a stale decision.
- **CMS publication adopts intent without creating a second identity.** The
  public `adopt_published_page_url` seam checks the plan-derived URL first and
  the actual live URL second. When a `planned` row exists under a different
  origin (for example a custom-domain plan published at
  `mymatrx.com/c/{site}`), it moves that SAME row to the live URL and advances
  it to `active`. With no plan candidate, creation still delegates to
  `ensure_planned_page_urls`; this module remains the one
  `normalize_url`/`url_hash` writer. A conflicting live row or an observed
  plan candidate raises instead of guessing or duplicating.
- Matching first groups scheme/www variants of the same normalized path and
  query. Crawl facts then contribute time-aware directed evidence:
  requested/final URLs, every redirect hop, and an in-scope declared
  `rel=canonical`. The newest observation for a source wins; at equal time,
  declared canonical outranks redirect, which outranks a direct response.
- Redirect chains flatten to their final sink. Cycles are collapsed
  deterministically and counted/logged as errors. A newer direct response can
  reverse an older redirect, which is why reconciliation is repeatable rather
  than a one-time merge.
- `reconcile_site_urls` reloads all page, crawl URL, and snapshot evidence for
  one site and recomputes the entire mapping in bounded batches. It runs after
  every full/page crawl — including failed and cancelled runs, because positive
  redirect/canonical facts stay valid even when coverage is incomplete — plus
  standalone sitemap sync, standalone GSC sync, and site initialization.
  Failure-path reconciliation never infers absence; negative presence remains
  gated separately by `coverage_qualified`. Operators and workflows can also call
  `POST /crawler/sites/{site_id}/urls/reconcile`
  (`scope.mode="url_reconciliation"`, NDJSON
  `url_reconciliation_progress`).
- Batch reconciliation consumes historical facts with
  `fresh_observation=False`: it may create a missing identity and may dismiss a
  resolved alias, but it never revives a user-dismissed canonical row, appends
  dismissal memory, or advances `last_seen`. Only a genuinely new crawl,
  sitemap, or provider observation can do those things.
- Reconciliation moves mutable evidence to the winner: GSC daily rows are
  aggregated by day (clicks/impressions summed, position impression-weighted),
  sitemap memberships are deduplicated, and empty canonical intent fields are
  filled from aliases. Conflicting user intent is never guessed: the canonical
  value wins and the conflict is counted/logged. Alias page rows are
  soft-deleted from normal listings only after evidence moves; immutable crawl
  facts remain attached to their observed alias and are reached through
  `canonical_page_id`.

## Link reconciliation contract

The crawl write path resolves every internal link's target URL to a canonical
page id at write time (including a matched alias's `canonical_page_id`) but NEVER
creates pages from links — an unknown target leaves
`web.link_edge.target_page_id` NULL. The backfill
`matrx_scraper.web_crawl.link_resolution.resolve_site_link_targets` (standalone
`POST /crawler/sites/{site_id}/links/resolve`, NDJSON
`link_resolution_progress` events, `scope.mode = "link_resolution"`) walks a
site's unresolved internal edges in id-keyset batches, resolves what the
registry now knows via one set-based `bulk_update_by_pk` per batch, and loudly
reports `{scanned, resolved, unresolved}`. It creates no pages either, and is
idempotent — each run only touches edges still NULL.

## Link status contract

`web.link_edge.http_status` was NEVER populated — the crawler recorded edges
but checked no target, so broken-link detection had no data anywhere in the
product (matrx-frontend `FOUND_DEFECTS.md` D74; live audit found all 10,676
rows NULL). `matrx_scraper.web_crawl.link_check.check_site_links` (standalone
`POST /crawler/sites/{site_id}/links/check`, NDJSON `link_check_progress`
events, `scope.mode = "link_check"`) fills it in two passes:

- **Internal — no network, same-crawl truth first.** Every edge reads the target
  URL's `web.crawl_url.http_status` from the SAME session as its source
  snapshot, even when the failed target correctly has no `web.page` row and
  `target_page_id` is NULL. This is load-bearing: a 404/500 earns no accepted
  snapshot, so `page.latest_snapshot_id` alone hides the broken target behind
  an older 200. Historical resolved edges fall back to that latest snapshot
  only when their session ledger has no status. Truly unfetched targets stay
  NULL and are counted — never guessed.
- **External — bounded, polite, cached by existing evidence.** A status for the
  same site + exact target URL verified within 24 hours is copied from the
  existing `link_edge` evidence (no cache table). Remaining distinct targets
  use HEAD first, GET fallback, redirect following, final-status recording,
  global concurrency 10, per-host serialization + 0.5s spacing, and a 10s
  timeout. A network-level failure records **`0` = no response**. One round is
  capped at 500 targets; automatic completion drains as many bounded rounds as
  needed, persisting every round before starting the next.

**Successful full/list crawl completion now has a link-evidence barrier.** The
sink runs idempotent target resolution + status verification BEFORE persisting
`CrawlCompletedEvent(status="completed")`, scoped to that session's snapshots.
If a deploy cancels the worker, the success event never lands, the existing boot
sweep resumes the crawl, and already-written statuses are skipped. The checker
runs before post-crawl analysis, so `broken_internal_links` and
`broken_external_links` write normal `web.analysis_result` / `web.finding` rows
in that same run. Old crawl sessions are deliberately NOT auto-backfilled; they
remain explicit pre-evidence sessions until re-crawled (the standalone command
still supports a deliberate historical backfill).

Idempotent: each run only touches edges still NULL (`recheck=True` re-verifies).
Non-http(s) targets (mailto:, tel:) are skipped and counted.

**There is ONE checker, and `check_urls(urls, …)` is how a non-crawl caller
reaches it.** `check_site_links` owns `web.link_edge` rows; outreach broken-link
prospecting (aidream `services/seo/broken_link_prospecting.py`) checks the
outbound links of pages we do NOT own and therefore has no edge rows at all.
Both go through the same `_PoliteChecker`, so the three things that would
silently drift between two implementations cannot: HEAD→GET fallback, redirects
followed to a final status, and per-host spacing under a global semaphore. It
returns `{url: status}` keyed by the distinct http(s) URLs actually checked,
with the same `0 = no response` rule, and awaits an optional `on_result(url,
status)` per completion for callers that stream or persist as they go. **Do not
write a second URL checker** — the crawl path itself now consumes this one.

**2026-07-23 (WS-15) — this pipeline was silently a no-op since the day it shipped.**
matrx-frontend's `web.reject_immutable_fact_mutation()` trigger (applied 2026-07-20,
one day before this module) blanket-rejects every UPDATE on `web.link_edge` — every
`http_status` backfill write here threw `ObjectNotInPrerequisiteStateError`, live-verified
on `allgreenrecycling.com` (120,240 edges, 100% NULL). Fixed at the DB, not in this
module: `db/migrations/0240_web_link_edge_http_status_mutable.sql` narrows the trigger
function to allow derived `http_status`. On 2026-07-27 the same narrow exception
was extended to derived `target_page_id`, making `/links/resolve` operational.
Every content column, every DELETE, and the function's other two callers
(`web.snapshot`, `web.analysis_result`) stay exactly as immutable as before. Any
future column meant to be filled after insert needs an explicit narrow exception,
never a blanket one.

## URL verification contract (`web_crawl/url_verify.py`)

`web.page` is an anchor registry: sitemap sync and GSC sync DECLARE URLs into it
without fetching them. On 2026-08-11 that left **7,423 of 10,424 live rows**
(4,511 sitemap-declared, 3,607 GSC-declared) with both `http_status_last` and
`content_type_last` NULL — the registry knew those URLs existed and nothing else.

Two consequences, both fixed by supplying evidence rather than by new logic:

1. `matrx_utils.web_page_class` has two signals — recorded content type and URL
   shape — and the first was silent for 80% of the registry.
2. **`sitemap_health` (weight 2.0) has ALWAYS promised the band ">10% of sitemap
   URLs are non-200, redirected, noindexed or robots-blocked", and
   `site_analysis._load_sitemap_membership` has ALWAYS classified junk off
   `page.http_status_last`.** With the column NULL those entries fell into
   `undiscovered` ("we haven't looked"), so the check scored only the small
   fetched subset and called a sitemap full of 404s clean.

**A verification is not a capture.** HEAD first, falling back to a **ranged**
GET (`Range: bytes=0-0`) for the many servers that answer HEAD with 403/405 —
`HEAD_FALLBACK_STATUSES` is that set, and a 404 is deliberately NOT in it
(a 404 is a terminal answer about the URL, not a complaint about the method).
No body is read, nothing is parsed, and **no `web.snapshot` row is written** —
snapshots are immutable append-only facts about a CAPTURE
(`web.reject_immutable_fact_mutation`), and minting one from a header-only probe
would be a lie in the fact table.

- **Content type comes from `scraper.content_type_from_header`** — the ONE
  header→token mapping, which `detect_response_content_type` also delegates to.
  Both writers of `content_type_last` must speak one vocabulary. A header the
  platform does not recognise yields **`unknown`**, never a guess: `unknown` is
  in `web_page_class.UNKNOWN_CONTENT_TYPES` and is treated as NO signal, so
  classification falls through to URL shape. NULL still means "nobody ever
  looked"; the two are not the same fact and must not collapse.
- **Redirects are followed and `http_status_last` records the FINAL status** —
  identical to what the crawler records, because one column may not mean two
  things depending on which writer touched it. The hop lives in
  `page.metadata['url_verification']`, and `_load_sitemap_membership` reads
  `redirected` from there for the `redirecting` junk class.
- **No response at all records `0`** (`NO_RESPONSE_STATUS`), matching
  `link_check` — a dead host reads as broken, never as unchecked.
- **Writes never clobber a crawl.** Every persist is a compare-and-swap guarded
  on `http_status_last IS NULL`; a crawl that landed mid-probe always wins, and
  the decline is counted (`lost_to_crawl`), not hidden.
- **Politeness matches the crawler's own** — 4 rps per host
  (`HostRateLimiter.default_rps`) for a probe far cheaper than a page fetch, and
  `validate_public_http_url` runs immediately before EVERY request, not once at
  seed time, so a DNS rebind cannot reach an internal address.
- **robots.txt is respected** and an unreadable robots.txt means "no rules
  known", never an invented block (`RobotsDocument.is_allowed` fail-open).

**Durable by construction** (`common-docs/policies/durable-work-queue-standard.md`):
the frontier is `runtime.work_item` rows scoped to one batch `global_execution`
linked as `web_site_url_verification`, claimed `FOR UPDATE SKIP LOCKED` under a
120s lease, identity = the page id (so a re-seed is a storage-layer no-op).
Seeding is **prioritized**: sitemap∩GSC-impressions first, then GSC impressions,
then sitemap, then the rest — the sweep is bounded, so it spends its budget where
a wrong status actually costs traffic. Process death is a non-event: the reaper
returns expired leases and `resume_unfinished_verifications()` (host side)
restarts the job. Proven by `tests/test_url_verify.py::test_chaos_a_killed_worker_
loses_nothing_and_duplicates_nothing`, which kills a worker mid-run and asserts
zero dropped and zero duplicated items.

Host wiring, execution lifecycle, and the boot resume live in
`aidream/services/seo/url_verification.py`; the operational entry point is
`scripts/verify_registry_urls.py` (dry-run by default).

## Link score contract (internal PageRank)

`web.page.link_score` is a 0..100 PageRank over the site's own internal link
graph — the single best at-a-glance signal of which pages matter, and the
ranking + node-size key for the link graph. The pure algorithm is
`matrx_scraper.pagerank.compute_link_scores` (damping 0.85, dangling-mass
teleport, rescaled so the most-linked page is 100); `web_crawl/link_score.py`
is only the `web.*` plumbing around it. Importable
(`score_site_links(site_id=...)`, workflow-node ready), standalone via
`POST /crawler/sites/{site_id}/links/score` (NDJSON `link_score_progress`,
`scope.mode = "link_score"`), and run automatically at the end of a crawl.

- **The graph is the site's CURRENT state.** Nodes are live pages that have
  actually been captured (`page.latest_snapshot_id` set); edges are internal
  `web.link_edge` rows belonging to those latest snapshots, selected with the
  same sub-select `insights.load_link_graph` uses. A sitemap-only page is not a
  node — it has no outbound links, and treating it as one dilutes every real
  page's share — but its URL still resolves, so links pointing at it are not
  dropped.
- **Both ends collapse to `canonical_page_id` before scoring**, so a link to a
  redirected URL credits the page it resolves to. Target URLs resolve through
  the same `url_hash(_normalise_url(...))` discipline every other writer here
  uses — which also means scoring does NOT depend on `/links/resolve` having
  run. Every page in a canonical group is written the group's score, so an
  alias row is never left NULL. Targets outside the page registry are counted
  (`edges_unresolved`), never guessed.
- **Recompute happens only for a COMPLETED `full` crawl** (plus explicit
  on-demand runs). A `list` crawl captures hand-picked URLs and a homepage
  bootstrap is one page: most links point at pages that were never fetched, so
  every page would look artificially isolated. The step hangs off the success
  path of `run_prepared`, so a failed, cancelled, or still-running session
  never scores either. **A stale-but-coherent score beats a fresh incoherent
  one** — `page.link_score_computed_at` is how a consumer tells which it has.
  Pinned by `tests/test_link_score.py`.
- **Wholesale and idempotent** — every captured page is rescored from the
  current graph on each run; there is no incremental path to drift.
- **Existing-site backfill** — `scripts/backfill_link_score.py` inventories
  every active, non-deleted `web.site`, defaults to `--dry-run`, and `--apply`
  calls the same `score_site_links` function for each site with live
  `web.link_edge` rows. Sites with no edges are reported as needing a re-crawl;
  the script never invents scores for a missing graph. It prints total,
  captured, scored, edge, and captured-inbound coverage per site and is safely
  re-runnable because the canonical scorer is wholesale and idempotent.
- Post-crawl failure degrades to a durable crawl WARNING
  (`post_crawl_link_scoring_failed`), never a failed crawl: the capture already
  succeeded, and `/links/score` is the retry path.

The score lives on `web.page` rather than on `web.snapshot` (immutable by
trigger — a derived, recomputable metric does not justify narrowing that guard)
or `web.analysis_result` (pass/warn/fail check rows that drive findings; a
neutral metric does not belong there). It sits with the other crawl-derived
"latest known" denormalizations the registry already carries —
`http_status_last`, `content_type_last`, `latest_snapshot_id` — and one indexed
column (`page_link_score_idx` on `(site_id, link_score desc)`) is what makes
"the site's strongest pages" a single sorted read.

## Page analysis contract (the `system_rules` provider)

When embedded in aidream, `run_prepared`, standalone analysis, and post-crawl
analysis call the package's optional `growth_stage_tracker` host seam. The
package stays independent when that seam is absent. Crawl points at its durable
`web.crawl_session`; analysis points at one immutable `web.analysis_result`
(or its analysis `crawl_session` when a valid sweep writes zero result rows).

`matrx_scraper.web_crawl.analysis.analyze_site_pages` is the deterministic
per-page audit worker commissioned 2026-08-08 (`web.analysis_result` had been
empty platform-wide since design). It is importable (workflow-node ready), runs
standalone via `POST /crawler/sites/{site_id}/analyze` (NDJSON
`analysis_progress` events, own `web.crawl_session` with
`scope.mode = "analysis"`), and runs automatically as a post-crawl step after
every full/list crawl's URL reconciliation — a failure there degrades to a
durable `crawl_warning` (`reason=post_crawl_analysis_failed`), never a failed
crawl and never a silent log line.

- **🚨 The run is LIVE the whole time, and every death is explained (2026-08-12).**
  `prepare_analysis` claims the same run lease crawls hold; `run_analysis`
  heartbeats it every `RUN_LEASE_HEARTBEAT_EVERY` (which also touches the
  session row's `updated_at`, the exact signal `fail_stale_sessions` reads —
  before this, every analysis longer than 30 minutes was reaped mid-work).
  Every death path persists a terminal error: a crash stores the exception, a
  cancellation (shutdown / stolen lease) stores `WORKER_STOPPED_ERROR`. Pinned
  by `tests/test_analysis_session_liveness.py`.
- **Results flush incrementally** (every `_RESULT_INSERT_BATCH` rows) so a
  killed run leaves its finished pages' observations behind, per the
  durable-work-queue baseline; finding reconciliation still runs once at the
  end over all written pairs.
- **`summary.timings` is the built-in profiler.** Every evidence loader and
  phase records seconds into the summary (streamed in progress events, stored
  in session stats) so a slow run names its hot phase itself.
- **🚨 Never page an aggregate over the query universe.** The GSC
  cannibalization loader is TWO grouped passes (totals, then IN-list-bounded
  page splits for significant queries only). Its previous 100-names-per-batch
  shape re-scanned the whole 28-day window twice per batch (~1.8s each,
  measured live) — 20+ minutes alone on a 19k-query site, which is what made
  every large-site analysis run die at the reaper's 30-minute mark.

- **🚨 This module implements NO per-page check.** Every per-page verdict lives
  ONCE in [`seo_audit.PAGE_CHECKS`](../seo_audit.py) — the same functions a live
  one-shot audit runs — and every threshold is a named CAPS constant there
  (`CONTENT_*_WORDS`, `IMAGE_ALT_FAIL_*`, `REDIRECT_CHAIN_MAX_HOPS`,
  `LARGE_PAGE_BYTES`, `TTFB_GOOD_MS` / `TTFB_POOR_MS`; the SERP length limits
  stay in `meta_metrics`, which owns them as a TypeScript mirror). `PageFacts`
  EXTENDS `seo_audit.PageEvidence` so a persisted snapshot is checkable without
  any translation. Adding a per-page check here re-creates the three-way drift
  killed on 2026-08-09 — pinned by `tests/test_seo_checks_single_source.py`.
- **🚨 A row may not describe more than the code measures.** The catalogue row
  IS the published contract; a row promising a richer measurement than the
  check performs misleads every reader of the catalogue and quietly withholds
  the bands it advertises. Two rows were narrowed/upgraded on 2026-08-09 when
  exactly that was found:
  - **`page_weight` grades the HTML DOCUMENT, not the whole page.** Subresource
    transfer sizes are never captured — `snapshot.extracted.resources` is a
    deduplicated URL inventory from the markup (kind / url / tag /
    source_attribute / attrs), with no byte counts, because the crawl never
    fetches a subresource. The row now says document weight and publishes the
    one `LARGE_PAGE_BYTES` ceiling the code applies. Widening it back to total
    weight means fetching every subresource during the crawl — a crawl-traffic
    decision, not a scoring tweak.
  - **`ttfb_server_response` grades TRUE TTFB** (`perf.ttfb_ms`) on the row's
    800 / 1800 ms bands — **never `perf.response_time_ms`**, which is total
    elapsed and includes the body download. TTFB comes from the transport:
    curl's `STARTTRANSFER_TIME_T` summed across redirect hops, or the httpx
    streamed-headers timestamp. The Playwright transport cannot report it, and
    no snapshot written before 2026-08-09 has the key — both answer `n_a` with
    a re-capture remediation, and the check may NEVER fall back to the total.
    All three bands score by FLOOR division so the matrx-frontend TypeScript
    mirror (`features/marketing/seo/audit/checks.ts`) cannot disagree on a
    midpoint: Python `round()` is half-to-even, `Math.round` is half-up.
- **What lives here is cross-page.** `CROSS_PAGE_CHECKS` — duplicate titles,
  duplicate meta descriptions, exact duplicate content, the broken /
  redirecting internal-link rollups, and the whole internal-linking /
  architecture set (below) — each needs the whole crawled site and therefore
  cannot live in a per-page primitive.
- **`url_design_quality` is the canonical pure URL-shape verdict.** It reads
  only `web.page.url` through `PageFacts`: start at 100; deduct once per
  category for length over 115 (15), more than two query parameters (15),
  uppercase in path/query (10), underscores in path/query (10), non-ASCII
  anywhere (10), and a normalized session parameter name
  (`aspsessionid`/`jsessionid`/`phpsessid`/`session`/`sessionid`/`sid`, 25),
  then clamp at 1. Any deduction warns; 100 passes. The implementation lives
  once in `seo_audit.PAGE_CHECKS`; the catalogue sweep only adapts it, and the
  Python↔TypeScript fixture pins exact parity.
- **🔗 Internal linking + architecture read the canonical graph evidence.** Seven
  items are implemented: `excessive_outlinks`, `nofollow_internal_links`,
  `anchor_text_descriptiveness` (deterministic ONLY — the row forbids a model
  call), `crawl_depth`, `internal_inlink_coverage`, and `orphan_pages` (the one
  `crawlability_indexability` member of the set), plus `internal_link_equity`.
  The first six read `web.link_edge`; `internal_link_equity` reads the canonical
  `web.page.link_score` that the completed-crawl PageRank job already wrote.
  Rules:
  - **ONE pass over the edges builds everything.** `_load_link_stats` folds
    every current-snapshot edge into `PageLinkStats` (outbound volume, internal
    nofollows, anchor descriptiveness, verified statuses) AND the site-level
    `SiteAggregates.inlinks` / `depth_by_page` / `OrphanCensus` at the same
    time. Snapshots are chunked by `_SNAPSHOT_BATCH_SIZE` and edges are
    keyset-paginated inside each chunk by `_EDGE_BATCH_SIZE` — a 20k-page
    site's edge table never lands in memory at once.
  - **Link targets resolve through `link_score.build_site_graph`**, the ONE
    place that knows how a target URL becomes a canonical page id. A link to a
    redirecting alias therefore credits the page it resolves to, exactly as
    link scoring does — never a second resolution rule.
  - **`internal_link_equity` never builds a second PageRank.** The page census
    already loads `page.link_score` and `target_keyword`; `index_page_facts`
    converts all non-NULL scores into one inclusive site percentile map. Ties
    share a rank and top ties receive 100. The page's score is that percentile;
    exactly as the live catalogue promises, a prioritized page whose
    `target_keyword` is set and whose percentile is below 25 is capped at 40.
    A NULL link score is `n_a` with the full-crawl action, never an invented
    zero. No graph reload and no N+1 query is involved.
  - **Optional page-ranking fields normalize at the evidence boundary.** A
    narrow/legacy page projection may omit `link_score` or `target_keyword`
    entirely; `_page_link_facts` treats omission exactly like database NULL.
    Both snapshot-backed and transport-only `PageFacts` use that one helper,
    so missing optional evidence remains `n_a` instead of crashing the sweep.
  - **Structural counts admit EVERY edge; verified counts still require a
    non-NULL `http_status`.** How many links a page emits and how they are
    worded are facts of its own markup and need no link check to be true;
    "is this target broken" does.
  - **`crawl_depth` is a BFS from the homepage** (`site.root_url` resolved
    through the same graph), so the answer is the SHORTEST click path. A page
    the BFS never reaches is `n_a`, NOT a fail — the row hands unreachable
    pages to `orphan_pages`, and two items scoring one defect double-counts it.
    `internal_inlink_coverage` defers the same way at zero inlinks.
  - **🚨 A HALF-CRAWLED SITE MANUFACTURES ORPHANS — the graph checks refuse to
    judge one.** `LINK_GRAPH_MIN_CAPTURE_RATIO` (0.8) requires that most known
    live pages were actually captured before an ABSENCE of links means
    anything; below it, `orphan_pages` and `internal_inlink_coverage` answer
    `n_a` with a "finish a full crawl" remediation, and `crawl_depth` withholds
    only its PENALTIES (a shallow pass survives — more coverage can only
    shorten a path). Measured, not theorised: on titaniummarketing.com a crawl
    that died with 119 pages fetched and 181 still queued made **81 of 124
    captured pages look orphaned**, every one of them a lie — the pages linking
    to them simply had not been fetched. This is the same rule `link_score`
    already applies by only scoring after a COMPLETED full crawl.
  - **🚨 `orphan_pages` keeps its two populations APART.** A page the crawler
    captured that nothing links to is a confirmed in-graph dead end. A URL
    known only from a sitemap, GSC, or a manual import was never captured at
    all — orphaned in the same graph, but it may not resolve at all, and
    reporting it as "a crawled page with no inlinks" sends the user hunting for
    a linking problem when the URL itself may be the problem. Both count toward
    the site ratio the catalogue's formula scores (`OrphanCensus.site_score`,
    floor included); only the captured class carries the "crawled successfully"
    verdict, and the uncaptured one gets its own reasoning plus
    `evidence.captured=false` and a re-capture remediation. The site root is
    never an orphan. A truncated run sets `OrphanCensus.complete=False` and the
    check answers `n_a` — a partial registry cannot honestly divide by "known
    live pages".
- **⚡ `performance` scores PAGESPEED, not the crawl.** Five items landed
  2026-08-09 — `cwv_lcp` (2.5), `cwv_inp_tbt` (2.0), `cwv_cls` (2.0),
  `asset_delivery` (1.5), `caching_policy` (0.75) — all in
  `seo_audit.PAGE_CHECKS`, all pure. Rules:
  - **No HTTP fetch can produce a Core Web Vital.** LCP / TBT / CLS need a real
    browser render, which is PageSpeed Insights' job. The ONE store is
    `seo.page_performance`, written ONLY by
    `run_collection("pagespeed_insights", …)` (matrx-seo owns the vertical); the
    sweep READS it through the narrow read-only mirror
    [`db/models_seo_host.py`](../db/models_seo_host.py) — a package dependency
    on matrx-seo would cost the whole graph for one table, and matrx-seo already
    mirrors `web.page` in the opposite direction for the same reason.
  - **🚨 A check NEVER fetches.** Collection is a separate step; putting a
    minute-long paid remote render inside a per-page sweep over 20k pages is the
    failure this rule exists to prevent. `_load_lab_performance` does ONE query
    per 200 pages, `DISTINCT ON (page_id, strategy)`, so at most two rows per
    page reach memory however long the sampling history is. **Mobile wins** when
    both strategies exist: Google's CWV thresholds and the catalogue bands are
    the mobile ones, and mobile is the indexing crawler.
  - **Never measured ⇒ `n_a` + `COLLECT_PAGESPEED`, never a pass.** Same for a
    measurement older than `LAB_PERFORMANCE_MAX_AGE_DAYS` (90) — a year-old
    render is not today's truth. Telling a user their slow page is fine is worse
    than telling them nothing.
  - **🚨 Lighthouse 13 renamed the entire opportunity family.**
    `render-blocking-resources`, `uses-text-compression` and
    `uses-long-cache-ttl` NO LONGER EXIST; the successors are the `*-insight`
    audits, and every one reports `scoreDisplayMode: "metricSavings"` — a mode
    matrx-seo's capture filter did not keep, so PSI's whole delivery fix-list
    was discarded before it reached the DB. Fixed at the source
    (`matrx_seo.providers.pagespeed::_delivery_facts`, which projects a BOUNDED
    `lighthouse.delivery` block), pinned by
    `matrx-seo tests/test_pagespeed_lighthouse13.py` against a REAL Lighthouse
    13.4.1 response. Rows written before that fix answer `n_a` for
    `asset_delivery` / `caching_policy` — never a fabricated zero.
  - **Savings are the LARGEST single-metric win, never a sum.** An audit
    reporting `{LCP: 300, FCP: 300}` cost 300ms of work, not 600.
    `cache-insight` is deliberately excluded from `asset_delivery` — caching has
    its own catalogue item, and counting it twice charges one defect to two
    scores. `caching_policy` is BYTE-weighted (the row's formula), and the HTML
    document is excluded from the denominator: it is normally uncacheable by
    design, and counting it would penalise every dynamic site for doing the
    right thing.
  - **`mobile_usability_lab` is NOT implemented, on purpose.** Its published
    formula names `tap_targets`, `font_size` and `content_width`; Lighthouse 13
    deleted all three audits and `font-size` has no replacement anywhere in the
    payload. Scoring 40% of a formula and calling it the whole thing is a
    fabricated verdict. Blocked on an Arman decision — see `FOUND_DEFECTS.md`.
- **🔍 `search_signals` reads GOOGLE'S OWN numbers, not our inferences.** Two
  page items landed 2026-08-09 on `web.gsc_page_stat` (written by `gsc_sync`,
  dimensions date × page): `gsc_ctr_opportunity` and `gsc_performance_decay`.
  They are registered in `CROSS_PAGE_CHECKS` for a structural reason — their
  evidence is a database load, and `seo_audit` never touches the DB. Rules:
  - **FOUR queries, whatever the size of the site.** `_load_gsc_stats` asks for
    the synced date span, then one `group_by("page_id")` rollup per comparison
    window. The per-day rows are NEVER loaded: a 20k-page site with a year of
    history is millions of rows, and the checks only ever read per-page totals.
  - **Windows are anchored to the freshest SYNCED day, never to "today".** GSC
    lags reality by ~2 days, so a sweep run on a Monday would otherwise read
    the not-yet-delivered weekend as a collapse. `_gsc_windows` returns the
    current / prior / quarter-ago windows, each `GSC_PERIOD_DAYS` long,
    contiguous between current and prior.
  - **CTR is recomputed from window totals**, never averaged from the daily
    `ctr` column — averaging rates weights a 3-impression day the same as a
    3,000-impression one.
  - **The position→CTR curve is ours.** The `score_contract` says "expected-
    for-position" without supplying the curve, so
    `GSC_EXPECTED_CTR_BY_POSITION` (+ the page-two and deep constants) is its
    ONE declaration.
  - **No property bound ⇒ `n_a` naming that fact, on every page, forever.** The
    binding is parsed by `gsc_sync.parse_gsc_binding` — the ONE parser of that
    shape — so "connected" here can never mean something different from what
    the sync accepts. A bound-but-never-synced site gets the `SYNC_GSC`
    one-click remediation. The default `SiteAggregates()` is unbound, so a
    caller that forgets the loader gets `n_a`, never a fabricated pass. A load
    failure is caught, recorded in `summary.errors`, and degrades to `n_a` —
    Google being unreachable never fails the whole sweep.
  - **`gsc_performance_decay` needs `GSC_DECAY_MIN_HISTORY_DAYS` (118) of
    synced history** and answers `n_a` below it. The sync's default is 28 days,
    so decay is `n_a` on every site synced with defaults — honest, and the
    reason it says so in the verdict. Deeper history comes from calling the
    sync with a larger `days`.
  - **Its top two bands require the drop to hold across BOTH comparisons**
    (prior period AND a quarter ago), so they read the smaller of the two. Zero
  clicks a quarter ago is not missing data — the history gate already proved
  that window is synced — so it means the page grew from nothing and cannot
  be "in decline" against it.
  - **`gsc_keyword_cannibalization` is a SITE verdict over canonical
    `seo.search_performance_daily` query×page evidence.** It never infers from
    `web.gsc_page_stat`. The read-only `SeoSearchPerformanceDaily` mirror
    exposes only the fields this check consumes. The newest collection run per
    date wins (the same dedup contract as the live GSC insight RPC), the window
    is the 28 days ending at the freshest stored day, and only
    `provider='gsc'` / `dimension_profile='query_page'` rows qualify. Significant
    queries (>=200 impressions) are read in batches of 100; each batch is
    grouped query×page in Postgres, so the entire raw daily history and query
    universe never sit in process memory. A query is cannibalized when at least
    two pages each hold >=20% of its impressions. The live formula is
    `round(100 * (1 - cannibalized/significant))`, or 100 when collected data
    contains no significant queries. No query×page collection at all is `n_a`,
    never a clean pass. Evidence stores bounded query samples plus every
    competing page identity in each sample.
  - **`gsc_index_coverage` remains UNIMPLEMENTED for want of evidence.** It
    needs the URL Inspection / Page Indexing API; search-performance presence
    is not proof that an indexable URL is indexed. It remains filed in
    [`FOUND_DEFECTS.md`](../../../../FOUND_DEFECTS.md) and may not be faked from
    page totals.
- **The catalogue is the contract.** Every check is a NAMED builtin
  `web.analysis_item` row; `PAGE_CHECKS` maps item key → check function
  (**57 checks live** — 41 per-page + 16 cross-page; the live count is
  `len(analysis.PAGE_CHECKS)`: title/meta presence + length + duplication, H1
  presence, thin content, image alt presence, meta-robots conflicts,
  canonical presence + conflicts, broken + redirecting internal links,
  broken external links, exact duplicate content, the transport +
  pagination set — `broken_page_4xx`, `server_error_5xx`, `redirect_chain`,
  `redirect_loop`, `mixed_content`, `page_weight`, `ttfb_server_response`,
  `pagination_markup` — and the head-metadata set the capture had always
  stored but nothing had ever judged: `viewport_meta`, `html_lang_validity`,
  `og_image_validity`, `social_meta_completeness`, `meta_refresh_redirect`,
  plus the internal-linking / architecture set: `excessive_outlinks`,
  `nofollow_internal_links`, `anchor_text_descriptiveness`, `crawl_depth`,
  `internal_inlink_coverage`, `orphan_pages`, plus the outline/depth/fake-200
  set: `heading_hierarchy`, `text_html_ratio`, `content_depth`,
  `soft_404_detection`, `temporary_redirect_usage`).
  A registered check whose catalogue row is missing
  FAILS the run loudly — never invents an ad-hoc key. The reverse gap is loud
  too: a canonical `seo_audit` check with no catalogue row is named in
  `summary.errors` and `logger.error` on EVERY run until the row is seeded —
  computing evidence and discarding it is never silent. That list is
  **currently empty** (live-verified 2026-08-09: all 24 registered checks have
  builtin catalogue rows). The last eight transport/pagination checks became
  recordable when aidream's `IssueDetector` was retired into this module;
  `ANALYZER_VERSION` moved to `1.1` because the score population changed.
  Severity comes ONLY from the item's `severity_map` bands applied to the score.
- **🖼️ The `images_media` set (5 checks, seeded 2026-08-09) reads
  `snapshot.images.items`** — the per-`<img>` inventory (responsive sources,
  alt, dimension attributes, `loading`, featured marking), capped at
  `IMAGE_INVENTORY_LIMIT` while `images.count` stays the true total. A non-zero
  count with an empty `items` means the snapshot predates the inventory and
  every one of the five answers `n_a`.
  Before the snapshot is persisted, the crawler enriches this SAME inventory
  in one bounded pass — never one pass/check and never a second store. It GETs
  at most 20 distinct selected sources per page with concurrency 4 and a 2 MiB
  body ceiling, deduplicates repeated image URLs, revalidates every redirect
  target against the public-network guard, and records `capture_status`,
  `http_status`, response `bytes`, `content_type` / `actual_format`, and decoded
  `natural_width` / `natural_height`. A `<picture>` source is preferred over
  its legacy fallback. Every unmeasured item says why (`skipped_limit`,
  `missing_url`, `network_error`, `blocked_url`, `redirect_limit`, or
  `too_large`); missing/partial evidence produces `n_a` + recapture remediation,
  never a fabricated green verdict.
  - `image_dimension_attrs` — share of images declaring width AND height.
    A CSS `aspect-ratio` reserves space equally well but is invisible to an
    HTML-only audit; that is the documented limit of the evidence.
  - `image_lazy_loading` — both directions. Viewport geometry is not captured,
    so "above the fold" is DOM order (`IMAGE_ABOVE_FOLD_DOM_COUNT`) **plus the
    page's own featured/OG image**, which IS captured and is the usual LCP
    element. A lazy hero is the fail band; eager below-fold images warn.
  - `image_modern_format` — BYTE-weighted WebP/AVIF share from the measured
    response type and transfer bytes. Vector/unclassifiable responses are
    excluded; no classifiable measured bytes ⇒ `n_a`.
  - `image_oversized` — decoded natural width vs declared display width plus
    the 300 KiB / 1 MiB transfer bands. It reports estimated avoidable bytes;
    2x width remains valid retina headroom. Missing any required measurement ⇒
    `n_a`, never the old srcset-only approximation.
  - `broken_images` — fails on 4xx/5xx responses and on a non-image payload
    returned with HTTP 200. A partial status capture cannot pass.
- **📐 Outline, depth, and fake-200 detection (5 checks, 2026-08-09).**
  `heading_hierarchy`, `text_html_ratio`, `content_depth`, `soft_404_detection`,
  and `temporary_redirect_usage` — all five score evidence the capture already
  stored (`headings.all`, `extracted.text_bytes`, `structured_data.schema_types`,
  `head_tags.og`, `extracted.redirect_chain` hop statuses). The rules that keep
  them from double-charging a defect someone else already owns:
  - **`content_depth` is NOT a second `thin_content`.** Thin content asks an
    absolute question with a fixed floor ("is there enough here for any query
    at all?"); content depth asks a relative one against the page's OWN declared
    type ("is there enough here for an *article*?"). A 400-word page is a
    shallow article and a perfectly normal product page. When the page declares
    no type (`declared_page_type` finds no schema.org `@type` and no `og:type`)
    there is no expectation to measure, so it answers `n_a` — never a restated
    floor. Utility types (contact, about, checkout, search) are exempt by the
    row's own rule.
  - **`heading_hierarchy` never charges a missing H1.** A skip is a jump of more
    than one level between two CONSECUTIVE headings; the FIRST heading is never
    a skip, because a page whose outline starts at h2 is `h1_presence`'s verdict.
    `headings=None` (a snapshot predating `headings.all`) is `n_a`; `[]` is the
    real answer "this page has no headings", which only warns once the page
    carries more than `CONTENT_OK_WORDS` of text.
  - **`text_html_ratio` needs `extracted.text_bytes`**, the UTF-8 length of the
    same visible text `word_count` counts, written at capture beside it. A
    snapshot older than that key is `n_a` — the ratio is never estimated from
    word count. Only the sub-3% band warns; the row calls itself advisory.
  - 🚨 **`soft_404_detection` scores PAGE-LOCAL signals only.** Not-found
    phrasing in the title plus a near-empty body. The row also describes a hash
    match against the SITE's own 404 template — a site-wide comparison this
    per-page primitive structurally cannot make — so that tier is NOT evaluated
    and is filed in [`FOUND_DEFECTS.md`](../../../../FOUND_DEFECTS.md). An
    honest 404/410 passes: it is not pretending to be anything.
  - 🚨 **`temporary_redirect_usage` is SINGLE-SESSION ONLY.** The row scores a
    302/307 that survives ≥3 crawl sessions harder (40) than one seen once (65),
    because persistence is what proves it was never temporary. This sweep reads
    only the LATEST accepted snapshot per page, so it records the single-session
    band (65) and never claims the stricter one — also filed in `FOUND_DEFECTS.md`.
    A chain captured without hop statuses is `n_a`, never a pass. Meta-refresh
    redirects belong to `meta_refresh_redirect`, not here.
- **🔒 The `security` category (4 checks, 2026-08-09) — and the evidence it
  forced into capture.** `mixed_content` was the only implemented row; the other
  four were the reason the category was worth doing (`https_enforcement` is
  weight **3.0**, the joint-highest in the catalogue). Their subjects differ,
  and so do their homes:
  - **`https_enforcement` is PER-PAGE** (`seo_audit.PAGE_CHECKS`) because a
    single `http://` page on an otherwise-HTTPS site IS the defect, and that
    fact is free — the URL's own scheme (`fail`, score 5). The other three
    bands need the URL's `http://` variant probed, which a check may NEVER do
    (checks are pure). **Since 2026-08-13 that probe is per PAGE.** The site
    probe samples real page paths and fetches each one's `http://` twin
    (`site_probe.page_http_variants`, HEAD-first, `PAGE_HTTP_VARIANT_PROBE_LIMIT`
    paths); `analysis.stamp_http_variant_evidence` gives each page its OWN
    verdict (`evidence.scope="page"`) and falls back to the site's `http://`
    ORIGIN result (`scope="origin"`) only for a page outside the sample. Bands:
    200 ⇒ live insecure duplicate (30), a redirect that stays on `http` ⇒ 30,
    302/303/307 ⇒ 70, 301/308 ⇒ 100, unreachable ⇒ pass; no probe at all ⇒
    `n_a`, never a pass on the redirect nobody checked.
    🚨 **The origin was NOT good enough, which is why this changed.** A server
    that redirects its root while happily answering `http://host/deep/path`
    scored `pass` on every page of the site — the origin probe cannot see a
    path. A sweep where NOT ONE page carries page-level evidence is loud
    (`summary.errors` + a WARNING naming the degradation) and still scores.
  - **`hsts_policy` + `security_headers` are SITE subjects**
    (`analysis._site_check_*`) and read response headers **sampled** across the
    site (`SECURITY_HEADER_SAMPLE_LIMIT` pages). A baseline header counts as
    present only when EVERY sampled page sends it — partial coverage is the
    defect sampling exists to find, and the reasoning says which headers are
    inconsistent. `x-frame-options` is satisfied by a CSP `frame-ancestors`.
    `hsts_policy` bands on the WEAKEST sampled `max-age`
    (`HSTS_MIN_MAX_AGE_SECONDS`, the row's 6 months); `includeSubDomains` is
    advice in the reasoning, never a band. An HTTP-only site is `n_a` — that is
    `https_enforcement`'s failure to report, not HSTS's.
  - **🚨 The headers did not exist before this.** `web.snapshot` stored none, so
    both checks would have been permanently `n_a`. Capture was ADDED at the
    source: `seo_audit.security_response_headers()` filters the response to an
    **allowlist** (`SECURITY_RESPONSE_HEADERS`) at `_build_result_from_response`,
    rides `ScrapeResult.security_headers` → `PageSummary.response_headers` →
    `snapshot.extracted.response_headers`. The allowlist is the point: the full
    header set carries `set-cookie` and credential echoes that must never reach
    a persisted snapshot. `None`/absent (a snapshot older than the capture) ⇒
    `n_a`; `{}` ⇒ the server really sent none.
  - **🚨 `tls_certificate` (weight 2.5) is `n_a` on EVERY site today.** The crawl
    completes the TLS handshake and keeps nothing from it — no issuer, expiry,
    hostname set, or chain verdict — so the check names the missing evidence and
    refuses to score. `TlsFacts` is the shape a capture must fill; the expiry
    bands (`TLS_EXPIRY_CRITICAL_DAYS` / `TLS_EXPIRY_WARN_DAYS`, plus
    expired/untrusted/hostname-mismatch ⇒ score 1) are implemented and tested
    and light up the moment it does. The gap is filed in
    [`FOUND_DEFECTS.md`](../../../../FOUND_DEFECTS.md). A certificate nobody
    looked at is not a healthy certificate.
- **Check granularity follows the CATALOGUE, not convenience.** Status and
  redirects are four checks, not two, because the catalogue models them as
  four items with different weights (a 5xx outage outranks a 4xx dead end; a
  loop outranks a long chain). A check that answers two catalogue items at
  once can only ever be recorded under one of them — the other verdict is
  computed and thrown away, which is the exact failure the loud reverse-gap
  check exists to prevent.
- **Evidence in, zero tokens.** Checks read only persisted evidence: the
  latest accepted snapshot (`seo_metrics`, `head_tags` — including `lang`,
  `og`, `twitter`, and `meta` (`{viewport, refresh}`) — `headings`, `images`,
  `audit_metrics.indexability`, `extracted.fingerprint`, `extracted.pagination`,
  `extracted.redirect_chain`, `extracted.mixed_content`, `extracted.text_bytes`,
  `structured_data.schema_types`, `perf.bytes`,
  `perf.ttfb_ms`, `word_count`) plus current-snapshot `link_edge`
  rows — **internal AND external** (every edge for the structural
  internal-linking counts; only edges with a verified `http_status` for the
  broken/redirecting rollups).
  Missing evidence is an `n_a` result whose reasoning names the fix
  ("re-crawl", "run the link check") — never a silent pass. `head_tags.meta`
  is the newest of those keys (added 2026-08-09 with the viewport /
  meta-refresh checks): a snapshot captured before it, or by a transport that
  parsed no HTML, stores `null` there, and `viewport_meta` /
  `meta_refresh_redirect` answer `n_a` rather than reporting a tag as missing
  on a page nobody ever read it from. `lang` / `og` / `twitter` were always
  captured, so a null there IS the page's answer, not a gap. Canonical pages
  only (aliases skipped); non-HTML pages skipped and counted.
- **🚨 A URL with NO snapshot is still audited — from `web.crawl_url`.** A
  page that 404s, 500s, times out, or loops through redirects usually earns no
  snapshot at all, so a snapshot-only sweep silently excluded exactly the URLs
  most likely to carry a transport defect: `broken_page_4xx`,
  `server_error_5xx`, `redirect_chain`, and `redirect_loop` could never fire
  for their own subjects (verified 2026-08-09). The evidence already existed
  and nothing read it — `web.crawl_url.http_status` plus
  `metadata.redirect_chain`, written at insert by
  `persistence.crawl_url_fetch_metadata` for exactly this case.
  `_load_transport_only_facts` walks the site's snapshot-less `crawl_url` rows
  in id-keyset batches (`_PAGE_BATCH_SIZE`, inside the same
  `_MAX_PAGES_PER_RUN` ceiling — no second ceiling exists), keeps the NEWEST
  attempt per page (`completed_at`, then session `sequence`), and builds a
  `PageFacts` carrying ONLY those two transport fields. Rules:
  - **Every content field stays unset.** Nothing was parsed, so the checks
    that can say "not measured" answer `n_a` — a transport-only page must
    never contribute a content *pass*. The presence checks (title, meta
    description, canonical) still `fail`/`warn`, which is the honest reading of
    a URL that serves no document; the transport `fail` sits beside them and
    names the real cause. Pinned by `tests/test_transport_only_analysis.py`.
  - **A snapshot always wins.** A page analyzed from its snapshot is never
    re-analyzed from the poorer crawl_url evidence.
  - **No page row, no verdict.** `web.analysis_result.subject_id` must be a
    page, so a `crawl_url` with a NULL `page_id` (or one whose page is
    deleted / an alias / non-HTML) is skipped and counted in
    `summary.crawl_urls_skipped_no_page` — never a failed run, never silent.
    Recovered pages move from `pages_skipped_no_snapshot` to
    `pages_transport_only`.
  - Transport-only facts carry no `latest_snapshot_id`, so they join no
    link-edge rollup and no duplicate-content/title group — those checks stay
    `n_a` for them by construction.
- **🌐 Structured data + international (4 checks, 2026-08-09) read a capture
  that was already RICHER than any check.** `snapshot.structured_data` has
  always retained every parsed JSON-LD document, every ORIGINAL script string —
  malformed ones included — beside the parse error it produced, plus normalized
  blocks, microdata, RDFa and microformats; `head_tags.hreflang` has always
  held the annotations. Nothing here re-parses a page: a second JSON-LD parser
  would be a second source of truth for the same bytes.
  - **`structured_data_validity`** (per-page) reports, in the row's order:
    parse errors first (an unreadable script voids every rich result the page
    was marked up for, so it outranks any property problem), then properties
    Google REQUIRES for a rich-result type actually in use, then the merely
    recommended ones. The required/recommended tables and the accepted alias
    spellings (`openingHoursSpecification` IS opening hours;
    `aggregateRating`/`review` satisfy Product's offer requirement) are named
    constants in `seo_audit` — `RICH_RESULT_REQUIRED_PROPERTIES`,
    `RICH_RESULT_RECOMMENDED_PROPERTIES`, `SCHEMA_PROPERTY_ALIASES`. A page
    with no markup is `n_a` (absence is the coverage item's question), a page
    with no *capture* is `n_a` **with a re-capture remediation** — different
    facts, never the same answer.
  - **`hreflang_validity`** (per-page) validates codes, absolute URLs, the
    self-reference, canonical agreement, and x-default. `x-default` is the ONE
    non-language value a `hreflang` may hold; every other tag goes through
    `_BCP47_RE`, the single language-tag shape shared with
    `html_lang_validity`.
  - **`hreflang_reciprocity`** (CROSS-page, `analysis.py`) is return-tag
    checking, and it is the one check whose evidence often lives on a site we
    do not have. A target on ANOTHER registrable domain (`example.de` from
    `example.com`'s crawl) or on a same-site page this crawl never captured
    **cannot be confirmed or denied — it is `n_a` with the unverified targets
    named, never a fail.** The score is the share of the *verifiable* targets
    that return-link, so off-site versions never dilute a healthy on-site
    cluster. `SiteAggregates.hreflang_targets` registers EVERY crawled page,
    including pages declaring no hreflang — otherwise a page that merely forgot
    its return tag would read as "never crawled" and escape the check.
  - **`local_business_markup`** (CROSS-page) is site-level by construction: its
    row asks whether business markup exists "anywhere on site" and whether NAP
    values agree "across own pages". The verdict is computed once from
    `SiteAggregates.business` and recorded on every page subject. The most
    complete declaration speaks for the site (a stub `Organization` on a blog
    page must not condemn the full `LocalBusiness` on the homepage), and values
    are compared through `flatten_schema_value`, so a `PostalAddress` object
    and the same address written differently are one value, not a conflict.
  - **One indexer.** `index_page_facts` builds every cross-page grouping —
    titles, descriptions, fingerprints, the business entity, the hreflang graph
    — in one pure pass that `_load_page_facts` calls and the tests call
    directly, so a cross-page test exercises the REAL aggregation instead of a
    hand-built lookalike that can silently disagree with it. Tests:
    `tests/test_seo_structured_and_international.py`.
- **Write contract (DB-enforced by `web.validate_cross_pointers` + the
  status/score CHECKs).** `web.analysis_result` is insert-only immutable:
  pass/warn/fail carry score 1–100, `n_a`/`error` carry NULL; every row stamps
  `metadata.reasoning` (the sentence an expert would say — canvas doctrine
  rung 3), `provider_version = ANALYZER_VERSION` (bump on ANY scoring-rule
  change), and `run_id` = the producing session. `web.finding` is the mutable
  lifecycle register: a warn/fail result opens (or refreshes) the ONE open
  finding per (site, subject, item) — `finding_open_uniq` — a pass resolves
  it; resolved rows stay as history and a recurrence opens a NEW finding.
  Suppression is user state and is never touched. `web.v_page_score` /
  `web.v_site_score` / `web.v_priority_queue` populate from these rows with
  no further writes. Tests: `tests/test_page_analysis.py`,
  `tests/test_seo_checks_single_source.py`.

## Site-subject crawlability checks (`web_crawl/site_analysis.py`)

The catalogue's crawlability + URL-architecture items describe the SITE, not any
page, and carry the heaviest weights in the whole catalogue. Four are live
(2026-08-09): **`robots_txt_health` (3.0 — joint-highest)**, **`sitemap_health`
(2.0)**, **`sitemap_coverage` (1.5)**, **`host_protocol_consistency` (1.5,
category `url_architecture`)**.

- **Where they live.** `web_crawl/site_analysis.py` holds the four checks, their
  thresholds, and `load_site_evidence`. They are merged into
  `analysis.SITE_CHECKS` — the ONE site-subject registry, shared with the
  TLS/HSTS/header and near-duplicate checks — by
  `analysis._adapt_crawlability`, which is a pure dispatch and never logic. One
  sweep, one catalogue resolution, one finding
  reconciliation. Pinned both ways by `tests/test_seo_checks_single_source.py`
  (`test_site_analysis_module_defines_only_site_checks` and
  `test_every_crawlability_site_check_is_reachable_from_the_sweep_registry`,
  which asserts the registry closes over the SAME function object).
- **Subject shape (DB-enforced).** `subject_type='site'` requires
  `subject_id == site_id` AND `page_id IS NULL` (`web.validate_cross_pointers`);
  same status/score contract as a page row.
- **`host_protocol_consistency` scores THREE duplicate families, not two.**
  Its description always named www/non-www, http/https, AND trailing-slash;
  until 2026-08-13 nothing captured the slash pair and the contract had no band
  for it. It now does (`HOST_SLASH_DUPLICATE_SCORE` 65, between the soft-redirect
  60 and mixed-internal-links 70 bands, first-match-top-down), scored from
  `site_probe.slash_pairs` — and the LIVE `web.analysis_item.score_contract` was
  updated in the same change, because the row is the spec.
- **Every threshold is a named constant with ONE home**, in
  `site_analysis.py`, listed in the test's `CRAWLABILITY_SITE_THRESHOLDS`. The
  status boundaries (`SITE_PASS_MIN_SCORE` 85 / `SITE_WARN_MIN_SCORE` 50) are the
  seam that makes each contract's bands land on the status it describes — 85
  ("minor issues only") passes, 55 ("invalid directives") warns, everything the
  contracts call blocking or duplicating fails.

### 🚨 The evidence these score did not exist — `web_crawl/site_probe.py`

Three site-level facts were **captured nowhere**: the crawler read robots.txt
into an in-memory `urllib.robotparser` cache and threw the FILE away; "this site
has no sitemap" was indistinguishable from "sitemap sync never ran" (both are
zero `web.sitemap` rows); and nothing had ever fetched the www/non-www ×
http/https variants of the root. **A pass on unverified evidence is a lie**, so
the capture was built rather than the verdict guessed.

- `capture_site_probe(site_id)` is the ONE entry point. It fetches robots.txt,
  the four host variants, and up to `SITEMAP_LOCATION_PROBE_LIMIT` sitemap
  locations (robots' own `Sitemap:` directives first, then `/sitemap.xml` and
  `/sitemap_index.xml`), through the same two-part SSRF gate `preview.py` uses.
- **Per-PAGE probes (2026-08-13) — the root cannot answer for a path.** The
  same entry point samples the site's own page URLs
  (`load_page_probe_sample` → canonical, 2xx, https, non-root, one per path,
  ranked newest-seen) and probes two things per sampled path, HEAD-first with
  the `HEAD_FALLBACK_STATUSES` GET retry `url_verify` already defines:
  `page_http_variants` (the path's `http://` twin — `https_enforcement`) and
  `slash_pairs` (`<path>` vs `<path>/` — `host_protocol_consistency`'s third
  duplicate family). Caps are CAPS constants:
  `PAGE_HTTP_VARIANT_PROBE_LIMIT` (20), `SLASH_PAIR_PROBE_LIMIT` (10),
  `PAGE_SAMPLE_QUERY_LIMIT` (300), `PAGE_PROBE_CONCURRENCY` (4) — a diagnostic
  probe, never a second crawl. A slash pair whose two forms land on the SAME
  final URL is consolidation working, not a duplicate; a pair that never
  answered scores nothing. Both lists are ADDITIVE on the stored shape (no
  `format_version` bump): a probe captured before they existed carries none and
  the checks fall back instead of being invalidated.
- **Storage is `web.site.metadata["site_probe"]`, NOT `crawl_session.stats`.**
  That column is REPLACED wholesale by every `CrawlProgressEvent` and by the
  terminal `CrawlCompletedEvent` (`persistence.py::_stats_from_progress`), so
  anything written beside those keys is silently erased mid-crawl. The site row
  merges, one probe per site, `captured_at` stamped — a stale probe is visible,
  never missing. A probe whose `format_version` we don't know is IGNORED, not
  misread.
- **Failure is evidence, and the two kinds stay apart.** `http_status` set means
  the site answered (a 5xx robots.txt is a scoreable catastrophe);
  `fetch_error` set means it never did (that is `n_a`, never a score).
- **Called on BOTH paths into the sweep** — `_run_post_crawl_analysis` and the
  standalone `run_analysis` — via `WebCrawlService._refresh_site_probe`, so a
  site analyzed standalone scores the same evidence a post-crawl one does. A
  failed probe is logged loudly and is never fatal; the checks fall back to
  their `n_a`.
- **robots.txt parsing is one primitive**, `matrx_scraper/robots_txt.py`:
  Google's longest-match precedence with `*`/`$` (which `urllib.robotparser`
  does not implement), the `Sitemap:` directives, the addressed agents, and a
  per-line syntax-error list. It NEVER raises (the broken file is the thing
  being measured) and fails OPEN (robots.txt is a deny list; inventing a block
  would report a catastrophe that isn't happening).

### What each check reads, and when it refuses to answer

| Check | Evidence | `n_a` when |
|---|---|---|
| `robots_txt_health` | the probe's robots capture + the site's advertised URLs | no probe, or the site never answered |
| `sitemap_health` | `web.sitemap` rows + `web.page_sitemap` memberships, cross-checked against page status / noindex / canonical / robots | sitemap sync has never run (and the probe can't prove absence) |
| `sitemap_coverage` | the sitemap set vs the crawled indexable page census | sync never ran, no indexable pages captured, or the page census was truncated |
| `host_protocol_consistency` | the four variant probes + the sampled `<path>` / `<path>/` pair probes + internal-link host forms from `web.link_edge` | no probe, or no variant answered |

A sitemap proven ABSENT by the probe is scored, not skipped — "no sitemap"
is a finding, "we never looked" is not. Loads are batched
(`_SITEMAP_MEMBERSHIP_BATCH`, `_PAGE_LOOKUP_BATCH`), the page census is reused
rather than re-queried, and the internal-link host scan stops the instant it
finds a second form (a truncated clean answer says so in its reasoning).
Tests: `tests/test_site_analysis.py`, `tests/test_site_probe.py`,
`tests/test_robots_txt.py`.

## Remediation bindings — an `n_a` check NEVER instructs, it offers a button

🚨 **No check may tell a user to run a command in prose.** The reader is a
brilliant, completely non-technical subject-matter expert; "run the link check
command to populate them" (the literal wording shipped until 2026-08-09) is a
dead end for them, and the NO DEAD ENDS doctrine
(`common-docs/policies/no-dead-ends.md`) requires every problem the system can
detect to ship with its one-click fix.

The split is: **the sentence carries the meaning, the binding carries the
action.**

- `seo_audit.Remediation` — a frozen `{command, scope, label, explainer}`.
  `command` must appear in `seo_audit.REMEDIATION_COMMANDS`, which maps each
  one to the endpoint that ALREADY performs it. Five exist: `links_check` →
  `POST …/links/check`, `page_fetch` → `POST …/pages/fetch`, `site_recrawl` →
  `POST …/rescrape`, `sitemaps_sync` → `POST …/sitemaps/sync`, and
  `pagespeed_collect` → `POST /seo/pages/{page_id}/pagespeed/sync` (the ONE
  on-demand PSI funnel; note the `/seo` prefix — it is an SEO-collection route,
  not a crawler one).
  **Never invent a command with no endpoint** — a binding the UI cannot
  dispatch is the same dead end wearing a button. Five module constants are the
  whole vocabulary: `CHECK_SITE_LINKS`, `RECAPTURE_PAGE`, `RECRAWL_SITE`,
  `SYNC_SITEMAPS`, `COLLECT_PAGESPEED`.
- `CheckOutcome.remediation` carries it, and `analyze_site_pages` persists it
  as `web.analysis_result.metadata.remediation` (`asdict`, verbatim). No column,
  no table, no migration — `metadata` is already the row's jsonb.
- **A check answering `n_a` because EVIDENCE IS MISSING must attach one.** A
  check answering `n_a` because it is genuinely not applicable ("this page has
  no images", "this page is not part of a next/previous series") attaches none
  — that is the signal the frontend uses to tell "blocked" from "nothing to
  do", and it is why the reasoning for those cases states a fact rather than a
  waiting state.

**Why NOT `platform.assists`.** An assist is a personal, deduped, addressed
item. These are per-page evidence rows: one crawled site emits thousands of
`n_a` results (pages × blocked checks) all pointing at the same two or three
site-wide commands. Minting assist rows would flood the ledger and break the
assists doctrine's "loud, never nagging" law — and the button belongs ON the
finding, not in a dock. So the row carries the binding, and matrx-frontend's
`features/marketing/crawler/remediation.ts` is its ONE runner, dispatching to
the same `direct-client` crawler commands every other surface uses. There is no
second mechanism. (System of record for assists:
`common-docs/systems/assists/FEATURE.md`.)

**Frontend consumption.** `listPageBlockedChecks` reads the page's NEWEST
`computed_at` and returns only that run's `n_a` rows carrying a remediation —
an older blocked check whose evidence has since arrived must never show a
button for a solved problem. `PageBlockedChecksCard` groups them by command
(several checks usually wait on one action) and renders one verb-labeled button
per action with the `explainer` shown before it runs.

**Parity.** `seo_audit`'s browser mirror
(`matrx-frontend features/marketing/seo/audit/checks.ts`) carries the same
`Remediation` type, constants, and strings, and `checks.parity.test.ts`
compares `remediation` alongside status/score/issue_count/evidence/reasoning.
Change a reasoning string or a binding on one side and you change it on the
other in the same unit of work, then regenerate the fixture:
`.venv/bin/python packages/matrx-scraper/scripts/generate_page_checks_parity_fixture.py`.

## Catalogue coverage — how much of the contract is actually implemented

> The catalogue is the SEO scoring CONTRACT: one weighted builtin
> `web.analysis_item` row per check. **A row nothing implements is a promise the
> platform silently does not keep** — it never scores, never opens a finding, and
> never appears in a site's weighted score, while reading exactly like a
> capability we ship.

An audit on **2026-08-09** found **83 builtin rows against 24 implementing
checks** — 59 unimplemented, including four of the heaviest in the catalogue
(`robots_txt_health` 3.0, `https_enforcement` 3.0, `tls_certificate` 2.5,
`gsc_index_coverage` 2.5). Nothing in the codebase said so; it was found by
hand-diffing the live DB against the code. The OPPOSITE direction was already
guarded (`_uncatalogued_page_checks`, above); the direction that mattered was
not guarded at all.

**The guard is [`scripts/check_seo_catalogue_coverage.py`](../../../../scripts/check_seo_catalogue_coverage.py)**
(in aidream — it needs the host's ORM pool, which a package must never assume).
It is a "compare TRUTH to code" guard in the `db/schema_analysis/` shape:
pluggable checks, tiered output, LOUD, **never blocking**. `scripts/release.sh`
runs `--summary` on every release; the count is a **roadmap, not a failure**.

```bash
python scripts/check_seo_catalogue_coverage.py            # full weight-ranked report
python scripts/check_seo_catalogue_coverage.py --summary  # what release.sh prints
python scripts/check_seo_catalogue_coverage.py --json     # machine-readable
```

What it reports, in tiers:

1. **Catalogue rows with NO implementing check** — the roadmap, ranked by
   `weight` descending, headlined by the one number that matters
   (`N OF M BUILTIN SEO CHECKS ARE NOT IMPLEMENTED`).
2. **Implemented checks the catalogue cannot record** — folds in
   `_uncatalogued_page_checks` so there is **ONE reporting surface, not two**,
   and additionally catches a `seo_audit.PAGE_CHECKS` entry that the sweep never
   registers in `CATALOGUED_PAGE_CHECKS` (implemented, but it never runs).
3. **`score_contract` over-promises** — a row documenting a measurement its
   check does not perform. Not statically detectable (prose + rule bands vs
   Python), so known instances are RECORDED in `KNOWN_CONTRACT_MISMATCHES` with
   a fingerprint of the contract as filed; the guard re-checks them every run and
   says so when a recorded contract is edited, instead of repeating a stale
   claim. Two are recorded today (`page_weight`, `ttfb_server_response` — see
   `FOUND_DEFECTS.md`, 2026-08-09). Add one with
   `--fingerprints` to get the current hash.

**Shrinking tier 1 is the only thing that lowers the number**: implement the
check in [`seo_audit.PAGE_CHECKS`](../seo_audit.py) (per-page) or
`analysis.CROSS_PAGE_CHECKS` (needs the whole site), and name a per-page one in
`CATALOGUED_PAGE_CHECKS`. Extending the guard is one `@register_check("name")`
function in the script.

## Persistence

- `web.page` remains URL plus user metadata. It never receives content.
- Every successful capture appends `web.snapshot`, writes the original response
  body with its real format/MIME (HTML, XML/KML, JSON, text, PDF, or image),
  optional markdown, and every requested PNG through canonical `matrx-files` to private
  S3, appends its `web.link_edge` rows, and moves only the page's
  `latest_snapshot_id` pointer. The database stores only `files.files` UUIDs:
  required `body_file_id`, optional `markdown_file_id`, and required
  `web.screenshot.file_id`.
- HTML snapshots retain the complete machine-readable page evidence, not a
  summary-only projection. `structured_data` stores every parsed JSON-LD
  document, every original JSON-LD script string (including malformed scripts
  alongside parse errors), normalized entity blocks, microdata, RDFa, and
  microformats while preserving the legacy `schema_org` representative and
  `schema_types`. `images.items` stores every `<img>` with responsive sources,
  alt/dimension/loading evidence, and featured-image marking.
  `extracted.resources` stores the deduplicated DOM- and
  structured-metadata-declared inventory for images and responsive variants,
  video/audio, embeds, scripts, stylesheets, fonts, icons, manifests, tracks,
  and linked documents; its counts remain complete when the defensive
  5,000-item detail ceiling fires. `extracted.page_identity` stores the
  featured image/source, CMS/generator, application/site name, author,
  publication/modification dates, page schema types, locale, theme color,
  related API/shortlink URLs, and safe WordPress/Shopify identifiers. Secret
  Shopify checkout-token values are never retained. All of these are immutable
  facts from the same captured HTML and require no extra fetch or table.
- Every snapshot carries capture-time **content fingerprints** for duplicate
  detection: `content_hash` (sha256 of the raw response bytes) plus
  `extracted.fingerprint` = `{version, exact_sha256, simhash64, shingle_size,
  token_count}` computed by `parser/hashing.compute_text_fingerprint` over the
  SAME visible text the audit counts words on (fallback captures use their
  extracted text; empty/non-text captures store `null`). `exact_sha256` is the
  hash of whitespace-normalized lowercased text (exact-duplicate grouping);
  `simhash64` is a 64-bit weighted simhash over 3-word shingles as fixed-width
  hex (near-duplicate clustering via XOR popcount — 90% similarity ==
  hamming ≤ 6). Pure-Python sha256+md5 on purpose: values are compared across
  crawls and deployments, so no optional library whose feature hashing could
  drift. `near_duplicate_content` consumes that stored SimHash in
  the ONE `analysis.SITE_CHECKS` pass: it compares only the same fingerprint
  version, counts indexable pages in connected ≥90%-similar clusters, excludes
  pairs already consolidated under a shared canonical, and applies the live
  catalogue formula `round(100 * (1 - near_dup_pages/indexable_pages))`. A
  missing fingerprint or unknown indexability makes the site verdict `n_a`,
  never a fabricated pass. Result evidence includes bounded cluster page URLs,
  matching pairs, Hamming distance and similarity percentage, so the issue UI
  has direct doors to every implicated page. Pure clustering lives in
  `web_crawl/near_duplicate.py`; tests:
  `tests/test_near_duplicate_site_check.py`. Any algorithm change MUST bump
  `FINGERPRINT_VERSION`, and consumers
  compare equal versions only. `extracted.text_hash` mirrors `exact_sha256`.
  The frontend content report (`features/marketing` crawl reports) clusters on
  these; no backfill worker exists — a re-crawl populates them naturally.
  Tests: `tests/test_content_fingerprint.py`.
- **Page vs machine resource is ONE rule, in `matrx_utils.web_page_class`.** A row
  is a resource when EITHER the crawler's `web.page.content_type_last` verdict is
  non-`html`, OR the URL's shape proves it (asset extension / known machine
  endpoint path). HTML-only SERP/social/heading/indexability metrics are stamped
  only for pages; resources stay visible crawl evidence without being mislabeled
  as broken web pages. Consumed by the analysis gate, the insights gate, and the
  capture-time audit gate; mirrored by `web.is_machine_resource_url` +
  `web.v_page_list.is_resource` and matrx-frontend's
  `features/marketing/lib/page-content-class.ts` — **change one, change all
  three.** It lives in matrx-utils (the dependency root) so matrx-seo's PageSpeed
  budget planner shares the SAME definition — a `/wp-json/` URL has no file
  extension, so its suffix-only test used to spend paid PSI requests on
  WordPress REST APIs. Tests: `matrx-utils/tests/test_web_page_class.py`.
  - Response-driven classification alone was the rule until 2026-08-11, on the
    reasoning that a path blacklist would mislabel an unusual endpoint serving
    real HTML. It could not hold: `content_type_last` is NULL for 8.7k of 10.8k
    registry rows (every sitemap-/GSC-declared URL never fetched), so the verdict
    is silent for most of the registry. `datadestruction.com`'s audit ranked 717
    `/wp-json/...` endpoints as its worst pages. The shape set is therefore kept
    deliberately narrow — extensionless, `.php`, `.html` and `.aspx` are NOT
    matched, because that ambiguity is exactly what the verdict is for.
- Crawl discovery follows the canonical audit's real `<a href>` navigation
  graph. Generic parser resources (`src`, stylesheets, oEmbed/feed alternates,
  scripts, images) remain link/resource evidence and are never promoted into
  pages merely because they have same-site URLs.
- Browser capture is only attempted for renderable page URLs. Known
  non-renderable resources (`.xml`, `.kml`, `.json`, `.txt`, PDFs, and images)
  use the normal fetch path even when a crawl requests screenshots. Browser
  responses are classified from their actual response headers; Playwright does
  not imply HTML.
- Screenshot capture is an optional artifact boundary for full crawls. A
  screenshot timeout or incomplete requested set emits a durable warning with
  the missing kinds and underlying errors, persists every successful image,
  and retains the fetched page body, snapshot, parsing, and link discovery.
  Each image attempt has a fixed five-second ceiling so broken font loading
  cannot consume Playwright's 30-second default for every requested image.
  Homepage bootstrap remains strict because its explicit product outcome is a
  homepage screenshot.
- A proxy-indicative response (`407` or `5xx`, including nonstandard `520`) or
  proxy transport exception is not trusted as the origin-page result. Both
  HTTP and browser capture paths retry that URL once without a proxy, emit a
  durable warning with `proxy_status` or `proxy_error_class` plus
  `fallback=direct`, open a run-scoped proxy circuit breaker, and classify the
  page only from the direct response. Once open, the rest of that crawl goes
  direct instead of repeating the same failed proxy round-trip per URL.
  Origin-semantic `4xx` responses such as `404` and `429` keep their normal
  page/rate-limit handling.
- A fetch, render, or parse defect is terminal for that URL only: it emits and
  persists one `page_failed` fact, disqualifies coverage reconciliation, and the
  remaining queue continues. **Canonical page presence changes ONLY on
  authoritative negative HTTP evidence** (`failed_fetch_disposition` in
  `persistence.py`): `410` soft-deletes immediately; `404` debounces through
  the same consecutive-miss counter reconciliation uses (`missing` at once,
  `gone` + soft delete only at `GONE_AFTER_CONSECUTIVE_MISSES`); every other
  failure (network/timeout/render/5xx/429) records its `crawl_url` outcome
  and evidence `last_checked_at` but NEVER touches an existing page's status,
  presence verdict, or miss counter — a transient outage cannot corrupt state
  a prior successful crawl established. A brand-new URL that has never
  succeeded is still created `missing`. Any successful capture resets the
  consecutive-miss counter to 0. Tests: `tests/test_failed_fetch_isolation.py`. Canonical database/storage persistence failure is
  also isolated to that URL: the page is never reported as parsed or captured,
  a durable warning identifies `failure_scope=page` and
  `crawl_continued=true`, coverage is disqualified, and the remaining queue
  continues. The session therefore cannot report complete coverage while one
  bad artifact or row prevents the rest of the site from being captured.
- `low_text_content` is a quality signal, not a transport or parse failure. A
  successful response carrying that signal emits a durable warning with its
  status/byte count, then preserves and parses the page normally. This prevents
  JavaScript-heavy, index, and intentionally concise pages from becoming false
  crawl failures merely because boilerplate-stripped text is short.
- `web.crawl_url` gets one immutable terminal outcome per attempted URL with its
  own monotonic `(session_id, sequence)`. Its `metadata.redirect_chain`
  (written at insert via `crawl_url_fetch_metadata`) records the full fetch hop
  chain — `{status, url}` per hop, oldest first, final URL last, length 1 = no
  redirect — on BOTH the success and failure paths, so redirect-to-404/410
  evidence survives for URLs that never earn a snapshot. Every transport
  captures real hops (curl_cffi manual walk, httpx `response.history`,
  Playwright `redirected_from`); a missing transport chain synthesizes
  `status: null` hops, NEVER a fabricated status code. Presence of the
  `redirect_chain` key — even as a 1-hop chain — is the capability marker the
  frontend crawl reports use to distinguish "no redirect" from "crawled before
  hop capture existed" (sessions before 2026-08-08 lack it). The hop shape is
  load-bearing for `url_identity._relations_from_crawl_facts` and for
  matrx-frontend `features/marketing` report parsers — change it in lockstep
  or not at all. Tests: `tests/test_redirect_chain_evidence.py`.
- `web.page_evidence(source_type = "crawl")` is refreshed independently of
  first-creation provenance so other sources (manual, sitemap, GSC, GA4, CMS)
  can continue supporting the same canonical URL.
- `web.crawl_event` gets every emitted event before it is published live. It has
  a separate monotonic `(session_id, sequence)` and levels constrained to
  `debug | info | warning | error`.
- `web.crawl_event` is the durable history and gap source. The browser reads it
  directly from Supabase; Python never replays stored events to the frontend.
- **Re-crawl content-hash dedupe (approved 2026-08-08).** When the exact
  bytes to store are byte-identical to the page's PREVIOUS capture, the new
  `web.snapshot` row is still appended (observation history stays truthful;
  `latest_snapshot_id` still moves) but it POINTS AT the previously stored
  `files.files` objects instead of writing fresh copies. Contract:
  - Body html and markdown hash separately and dedupe independently: body
    compares against the prior snapshot's `content_hash` column; markdown
    against `metadata.artifact_hashes.markdown_sha256` (every new snapshot
    stores `metadata.artifact_hashes = {body_sha256, markdown_sha256}` —
    legacy snapshots lack the markdown hash, so markdown never dedupes
    against them). Hashes are of the exact bytes stored.
  - Comparison is ONLY against the page's CURRENT capture
    (`latest_snapshot_id`, non-deleted) — never an arbitrary older snapshot.
  - A deduped snapshot stamps `metadata.artifact_reuse =
    {body, markdown, reused_from_snapshot_id}`; the DB guard
    (`web.validate_snapshot_artifact_files`, extended by aidream migration
    `0306_web_snapshot_artifact_reuse.sql`, applied live 2026-08-08)
    validates reused references under
    `web.assert_crawl_artifact_file_reused` — every invariant kept
    (org/site, live, internal-only, mime, immutable web_crawl metadata)
    except the session binding, which instead requires the file's recorded
    `crawl_session_id` to name a real session of the SAME site.
  - **Shared-file safety (THE correctness risk):** reused files are never
    added to the compensation list, so a failed persist purges only freshly
    written artifacts — never a file an older live snapshot references
    (`tests/test_content_hash_dedupe.py` locks this). Screenshot retention
    prunes ONLY `web.screenshot` rows + their PNGs (screenshots never
    dedupe), and NO retention path deletes snapshot body/markdown files —
    any future snapshot-file retention MUST refcount `artifact_reuse`
    chains before touching storage.
  - Stats: session stats + progress carry `pages_unchanged` (captures whose
    body bytes matched the previous capture). `bytes_downloaded` is
    untouched — it counts network download, which dedupe does not change.
- Otherwise every capture gets a fresh immutable file row and S3 object even
  when bytes match another capture. Files carry `system_artifact`, `system_immutable`,
  `artifact_domain = web_crawl`, `web_site_id`, and `crawl_session_id` metadata.
  The artifact FK is the complete data relationship. File authorization derives
  from that FK to the snapshot or screenshot and then from its `site_id`; no
  association row is created.
- The standalone scraper installs the same authoritative database-backed file
  access checker used by the main host (`files.has_access_for` for files,
  `iam.has_access_for` otherwise). Readiness fails closed when that checker is
  absent, so contextual crawl artifacts never fall into the standalone
  owner/public subset.
- If row persistence fails after an upload, compensation purges each newly
  written orphan through the exact file-ID/storage-URI hard-delete primitive.
  It does not impersonate the historical file owner. Database deletion happens
  before object deletion, so a referenced artifact's FK stops cleanup before
  its S3 bytes can be touched.
- Byte-identical artifacts within one capture remain distinct immutable file
  rows. `force_new_copy` validates their equivalence target by canonical root,
  owner, organization, and checksum without requiring a domain edge that
  cannot exist until persistence completes. The capture writer never treats an
  access-denied checksum match as an orphan and never deletes it before retry.
- Crawler artifacts must use S3-backed `files.files` identities with
  `visibility='internal'` — ORG-INTERNAL, NEVER `personal` (Arman, 2026-07-29:
  "Absolutely never personal."). `personal` hid the entire crawl corpus from
  every org member except the crawler; the writer and the DB guard both reject
  it going forward (the guard accepts `personal` transitionally until the
  creation-fix release is deployed, then tightens to internal-only). A raw
  bucket name or storage path in `web.*` is a contract violation.
- A canonical write exception, missing file identity, or non-`s3://` result is
  fatal to that persistence step. Scraper persisters never log-and-continue
  with a null media pointer and never fall back to another storage backend.
- Initialization screenshots go through the SAME immutable crawl-artifact
  writer as every other capture: each re-run writes fresh `files.files` rows
  (session-scoped path, `force_new_copy`) carrying the full
  `system_immutable` / `artifact_domain = web_crawl` / `web_site_id` /
  `crawl_session_id` metadata, then APPENDS a new `web.screenshot` row per
  kind (newest live row wins; bounded by the retention prune above). The DB
  guard `web.assert_crawl_artifact_file` (trigger on `web.screenshot` insert
  AND update) enforces exactly that contract — a managed/versioned write does
  not satisfy it and was the production CheckViolationError this replaced.
  All screenshot rows are append-only; supersession is expressed through the
  retention soft-delete, never by mutating `file_id` in place.

## Reconciliation

Only unconstrained full crawls that also drain the runtime queue with no failed
fetches are coverage-qualified. Hitting `max_pages`, cancellation, or any fetch
failure disqualifies negative reconciliation. Qualified successful page hashes
are compared with the canonical page list at completion. Known pages not seen
become `missing`; they are not deleted. HTTP 410 immediately marks the canonical
page `gone` and soft-deletes it. Constrained/list/homepage crawls never produce
negative reconciliation evidence.

## One active site-wide crawl per site

`prepare_start` refuses to create a new `full`/`list` session while a live
one exists for the site: it first runs the stale-session reaper (a crashed
run must never block a start), then 409s ("already active", naming the
session id — the same contract resume uses) when a non-deleted
queued/running full/list session is still fresh: `running` judged by
`run_lease_is_live`, `queued` by reaper-freshness (`updated_at` within
`STALE_SESSION_AFTER`, failing closed on an unreadable timestamp). Race
backstop: two starts that both pass the pre-check re-list AFTER creating
their sessions and the racer with the larger `(created_at, id)` yields —
its just-created session is terminated (`abandon_duplicate_session`) so it
cannot itself block later starts. Bootstrap / initialization / page_fetch /
sitemap / GSC / reconciliation runs stay unrestricted. Tests:
`tests/test_single_active_crawl.py`.

## The frontier read cannot lie about being empty

`RuntimeWorkQueueBackend.counts()` decides when a crawl stops — the crawler's
terminal condition is `pending == 0 and in_flight == 0`. `work_item_counts`
builds its result from a GROUP BY and fills every missing state with 0, so **a
query that returns NO ROWS is indistinguishable from a drained queue**: no
exception, no log, the crawl just stops and reports success.

That is not hypothetical. `runtime.work_item` carries an RLS SELECT policy
(`iam.has_access('work_item', id)`) that evaluates **FALSE for every row today**
— so ANY read of that table as role `authenticated` returns zero rows, silently.
The durable frontier is worker-owned infrastructure and must never be read that
way, but "must never" is not a guarantee, and being wrong costs a truncated
crawl recorded as a success.

So the lie is made impossible: **work items are never deleted, only
transitioned**, so the TOTAL row count for an execution can only ever grow. If
the backend has seen N > 0 rows and now sees zero, the frontier did not empty —
the read did. It re-reads once (reconcile before refusing), then raises
`FrontierReadError`, which fails the run LOUDLY and keeps it resumable.

Tests: `tests/test_frontier_read_honesty.py` — including the two cases that must
NOT raise (an always-empty queue, and a queue that genuinely drains, where
terminal items still count toward the total).

## A truncated run is NEVER recorded as a success (the truncation gate)

`SiteCrawler.run()` refuses to report `completed` when it ends with work still on
the frontier. If the run wasn't cancelled and didn't hit `max_pages`, and either
`in_flight_count()` or `queue_depth()` is non-zero, the run was **truncated**:

- status becomes **`failed`** (→ session `status='failed'`), never `complete`;
- `error_message` names the exact shortfall (items in flight, items queued, pages
  fetched of pages discovered);
- a `crawl_warning` with `context.truncated=true` is emitted and an ERROR is
  logged — it screams, it doesn't just flip a field;
- because the session lands `failed`, it becomes eligible for the existing
  crash-resume sweep instead of being buried as a success.

**Why this exists:** a crawl that lost its client stopped mid-run and wrote
`status='complete'` with `pages_fetched: 0`, leaving its claimed work item in
flight forever. In the UI and in `web.crawl_session` that is indistinguishable
from a real crawl of a site with nothing on it — the most expensive kind of lie a
pipeline can tell, because nobody goes looking inside a success. **This gate is
the detection layer, not the cure** — it makes the failure loud and resumable
wherever it comes from.

**The dominant cause it kept catching was found and fixed 2026-08-13** (176
failed `page_fetch` sessions in two days, all "<1s, fetched 0 of 1, 1 in
flight"): the run loop's page-budget stop condition ignored *reserved* page
slots, so on a durable frontier it could fire in the window between a worker's
slot reservation and its claim's DB commit becoming visible to `counts()` —
tearing the only worker down mid-item with nothing settling the claim. Two
layers now prevent it, each pinned by
`tests/test_page_fetch_frontier_settle.py`:

1. no stop condition fires while `_pages_reserved > 0` — a reserved slot means
   work may exist that the frontier read cannot see yet;
2. a worker cancelled mid-item requeues its claim (`will_retry=True`, shielded)
   before the `CancelledError` propagates, so even a torn-down worker strands
   nothing.

Three endings legitimately leave work behind and are NOT flagged: a user cancel,
hitting the page budget (`limit_reached`), and an empty frontier. Tests:
`tests/test_crawl_truncation_gate.py` (including both non-truncation cases —
a gate that cries wolf gets deleted by the next person).

**Dead-letters break `coverage_complete`.** A dead-lettered work item (the
durable store's attempt budget firing — see
`runtime_queue.DEFAULT_ITEM_MAX_ATTEMPTS`) was neither fetched nor counted
failed, so `SiteCrawler.run()` reads
`RuntimeWorkQueueBackend.dead_letter_count()` at the end (duck-typed; the
in-memory frontier never dead-letters) and a run with any dead-letters emits a
loud `crawl_warning` and reports `coverage_complete=false`.

## Run ownership — one live run per session, fleet-wide

A crawl session may have exactly ONE live run across every process and
container. Ownership is the **run lease**: `metadata.run_lease` =
`{owner, epoch, acquired_at, heartbeat_at, host}`.

- **Claim** — `WebCrawlRepository.claim_run_lease(session_id, is_resume=)` is a
  compare-and-swap on the row's `version` (bumped by the platform `_touch_row`
  trigger on every UPDATE, so no application code maintains it). Exactly one
  claimant can win; a loser raises `RuntimeError("already active…")` → **409**.
  Both `prepare_start` and `prepare_resume` claim; the resume claim also clears
  the stale terminal marks and any leftover `cancel_request`, and bumps
  `metadata.resume.attempts`.
- **Hold** — `_watch_for_cancel` heartbeats every `RUN_LEASE_HEARTBEAT_EVERY`
  (30s) alongside its cancel poll. Losing the lease **cancels this run**, so a
  run declared dead stops instead of fighting its successor.
- **Live vs crashed** — `run_lease_is_live(session)`: status `running` and a
  signal (heartbeat, else the row's `updated_at`) inside `RUN_LEASE_TTL`. The
  TTL is deliberately `STALE_SESSION_AFTER` — "lease still live" and "the
  reaper would not have reaped it" must never disagree. A session with no lease
  at all (written by an older build) falls back to `updated_at` and fails
  CLOSED. `_assert_session_resumable` refuses a live session.
- **Write gate** — every session status/stats/terminal write carries
  `lease_token` and filters on `metadata @> {"run_lease":{"owner":<token>}}`:
  `mark_session_running`, `complete_session`, `fail_session`, and the
  `crawl_started` / `crawl_progress` / `crawl_completed` branches of
  `persist_event`. A suppressed write logs an **error** naming the session —
  it means two runs of one session existed at once. This is the write that made
  the cross-process bug destructive: the LOSER of a race stamped `failed` over
  the WINNER's live session.
- Event rows themselves are never lease-gated — `crawl_event` is a log, and a
  losing process's own events stay in the history.

Event-sequence collisions are the SECOND, independent layer.
`crawl_event_session_sequence_unique` stays, but a duplicate
`(session_id, sequence)` is no longer run-fatal: `DurableCrawlEventSink`
re-reads `MAX(crawl_event.sequence)`, re-sequences the event past it, and
retries (bounded by `EVENT_SEQUENCE_MAX_RETRIES`, loud on every firing).
Only that one constraint is recoverable — every other integrity error still
fails the run.

The five non-crawl session flows (sitemap/GSC sync, URL reconciliation, link
resolution, link check) mint a fresh session nobody else can be running and
deliberately hold no lease; their write gates degrade to unconditional.

## Deployment requirements

The scraper service requires the existing database, authentication, and
canonical Files/S3 configuration:

- `SUPABASE_MATRIX_HOST`, `PORT`, `DATABASE_NAME`, `USER`, `PASSWORD`
- `SUPABASE_MATRIX_URL`
- `SUPABASE_MATRIX_SECRET_KEY`
- `SUPABASE_MATRIX_JWT_SECRET` during HS256 compatibility (JWKS is derived from
  the Supabase URL for asymmetric tokens)
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`
- `AWS_S3_DEFAULT_BUCKET`

Readiness requires the canonical web and files ORM registrations, a canonical
`FileManager`, and a successful bounded S3 bucket probe in addition to the
browser pool. Startup fails closed before a crawl session is created when any
of them is unavailable. The engine's `scraper.*` cache/domain-config capability
binds to the SAME database through the same `SUPABASE_MATRIX_*` values (one
resolver — there is no separate scraper database), and its bootstrap is likewise
fatal; only the derived cache/domain-config components are reported separately
without blocking the canonical crawler. Startup never probes retired
`scraper.crawl_runs` tables.

## Host scope: www/apex equivalence

`www.<domain>` and the apex domain are UNCONDITIONALLY the same site for
every host-scope judgment (crawl scope classification, seed validation, GSC
in-scope checks), in both directions — a site registered as
`https://example.com` whose sitemap serves `https://www.example.com/...`
must never classify its own URLs as `outside_site_scope`. This is distinct
from `follow_subdomains`, which continues to gate every OTHER subdomain.
Exact observed page URLs and hashes stay untouched, but scheme/www variants now
converge through `web.page.canonical_page_id`.

## Change log

- **2026-08-17** — CMS publish gained canonical planned-row adoption. A
  cross-origin `/c/{site}` live URL now reuses and activates the existing plan
  identity instead of inserting a second `web.page`; regression coverage proves
  the insert seam is never called in that case.

- **2026-08-15** — **D74 closed at the crawl boundary.** Successful full/list
  crawls now resolve and verify their own link edges before the durable terminal
  event, making the work deploy-resumable and ensuring deterministic analysis
  sees broken-link truth. Internal status prefers an accepted same-session
  snapshot, then uses same-session `crawl_url` outcomes for snapshot-less
  4xx/5xx; this ordering prevents a later duplicate retry from overwriting a
  successful capture. External checks reuse 24-hour URL evidence, then run
  bounded HEAD/GET checks with redirects, timeout, concurrency, and per-host
  pacing. Historical sessions stay honestly NULL. The first Search Quake proof
  exposed and fixed the accepted-snapshot precedence case. Final production
  proof session `8168bcba-932d-4424-b1a7-5cab7eda53b4` populated 3,761 of
  3,802 Cosmetic Injectables edges (the 41 targets outside the bounded crawl
  stayed honestly NULL), persisted 85 external 404 edge occurrences, opened 20
  findings, and produced zero false broken edges back to the accepted homepage.

- **2026-08-12** — **Analysis runs are live, durable, and profiled.** The
  page-analysis session now claims + heartbeats the run lease (no more
  30-minute reaper kills mid-work), every death path stores a terminal error
  (`WORKER_STOPPED_ERROR` on cancellation), results flush incrementally, and
  `summary.timings` names each phase's cost. The GSC cannibalization loader
  was rewritten from a paged double-aggregate (~2 queries per 100 query names,
  20+ minutes on a 19k-query site) to two bounded grouped passes. Fixes the
  three silent datadestruction deaths of 2026-08-11.

- **2026-08-10** — 🚨 **ONE CRAWL WORLD — the dual-crawler gap is closed
  (growth-loop `G-CRAWL-DUAL`).** The legacy `scraper.crawl_*` store had already
  been moved to `graveyard` on 2026-08-09 (17 tables, all data intact, nothing
  dropped), but the retirement was **unrecorded** — no `public._schema_migrations`
  row and no `platform.deprecated_relations` entry, so a grep for
  `scraper.crawl_runs` found nothing and the move was invisible to the next
  agent. Migration `0327_one_crawl_world_record_and_guard.sql` is that record
  and is a guard: it re-asserts the move, refuses if a table reappears in
  `scraper`, maps each retired table to its canonical replacement, and pins
  both the surviving 7 `scrape_*` engine tables and the canonical `web.*` crawl
  tables. Deleted in the same change: the four dead `crawl_start` /
  `crawl_status` / `crawl_pages` / `crawl_cancel` AI-tool descriptors, which
  still sat in `ALL_TOOLS` (and so in the MCP server's advertised list)
  addressing `scraper.crawl_runs` through host `_ext` handlers nobody had wired
  since `aidream/services/scraper/` was deleted — every call raised. Second
  guard layer: `tests/test_standalone_db_bootstrap.py::test_no_legacy_crawl_tools_remain`,
  beside the existing model-level pin.

  **There is exactly ONE crawler: this module, over `web.*`.** Entry points are
  `api/crawl_router.py` (HTTP) and `web_crawl/schedules.py::dispatch_due_crawl_schedules`
  (the every-minute `web_crawl_schedule_dispatch` system task). There is no
  `_ext` seam for starting a crawl, no `aidream/services/scraper/`, and no
  second crawl schema. `graveyard.*` is a read-only archive; reversal is
  `ALTER TABLE graveyard.<t> SET SCHEMA scraper` per table.

Compressed 2026-08-09 — everything below shipped and is described in the
contract sections above; this list exists to date a behaviour, not to explain
it. Full detail is in git history.

- **2026-08-11** — URL identity reconciliation now also runs after failed and
  cancelled crawls. A partial run's positive redirect/canonical observations
  are safe and durable; only negative presence inference remains coverage-
  qualified. This closes the path that left already-resolved aliases live in
  the page registry whenever a later crawl ended early.

- **2026-08-09** — Recurring crawls closed canonically (`web_crawl/schedules.py`
  over `web.crawl_schedule`). Recurring crawls had been DEAD ON ARRIVAL: the
  system task drained `scraper.crawl_schedules`, a table with zero rows and no
  writer. The dispatcher now lives beside the crawler it drives, claims with
  `FOR UPDATE SKIP LOCKED` + a lease, resolves config through
  `derive_recrawl_config`, and records an already-crawling site as a SKIP.
  `matrx_scraper/scheduling.py` + `aidream/services/scraper/schedule_dispatch.py`
  + the legacy schedule CRUD/routes deleted; `web.crawl_schedule` gained
  `preset_id` / lease / outcome columns and lost its config-knob columns.
- **2026-08-09** — `CrawlStartRequest.user_agent`: one per-crawl User-Agent
  override applied to BOTH fetch paths (HTTP + browser) and to robots/sitemap
  discovery, with named presets at `GET /crawler/user-agents`. Closes the last
  legacy preset-config field that had no canonical equivalent. Omitting it is
  byte-identical to prior behaviour; validation lives once in
  `matrx_scraper/user_agents.py`.
- **2026-08-09** — Internal-link PageRank landed canonically: `web.page.link_score`
  + `link_score_computed_at` (indexed), `web_crawl/link_score.py`, the post-crawl
  step, and `POST .../links/score`. Closes the last legacy-only crawl feature
  (`aidream/services/scraper/pagerank.py` → `scraper.crawl_pages.link_score`);
  the pure algorithm already lived in this package and was reused, not rewritten.
  `scripts/backfill_link_score.py` applies that same scorer to every existing
  active site with edge data and reports no-edge sites as requiring a re-crawl.
- **2026-08-09** — Crawl presets + one-click rescrape closed canonically
  (`web.crawl_preset`, `web_crawl/presets.py`, four routes). Replaces
  `scraper.crawl_presets` and the fork's `derive_recrawl_config`; presets are
  site components under the site's own RLS (no per-preset owner or
  `is_shared`), and the stored config is a `CrawlStartRequest`, not a dict.
  `matrx_scraper/db/preset_queries.py` deleted — it was bound to the legacy
  table and had no consumers.
- **2026-08-09** — Run-over-run diff closed canonically (`web_crawl/diff.py`,
  four GET routes): added / removed / returned / changed pages between two
  crawl sessions, derived from `web.crawl_url` + `web.page` + `web.snapshot`.
  No `site_run_diffs`-style cache table, and none may be added.
- **2026-08-09** — The legacy crawler's three read/aggregation gaps closed
  canonically: duplicate-content clusters, the internal link graph, and a run's
  progress series (`web_crawl/insights.py`, three capped GET routes). Derived
  entirely from `web.snapshot` / `web.link_edge` / `web.crawl_event` — no new
  table, and `scraper.crawl_progress_snapshots` gets no canonical twin.
- **2026-08-08** — Per-page analysis workers commissioned (`web.analysis_result`
  had been empty platform-wide since design) · full redirect hop-chain evidence
  on every transport · capture-time content fingerprints
  (`extracted.fingerprint`) · re-crawl content-hash artifact dedupe · dismissal
  memory on revived rows · one active site-wide crawl per site · transient fetch
  failures isolated from canonical page presence · deploy-cancelled crawls
  auto-resume · invalid include/exclude regex is a 422 at the boundary · real
  Public Suffix List for registrable-domain judgments.
- **2026-08-04** — Cross-process resume closed by a durable run lease (CAS on
  `version`, heartbeated, gating every session write) plus recoverable event-
  sequence collisions. Durable resume is PROVEN by a chaos test that kills a
  live crawler mid-run and drains the same batch from a second process
  (`tests/test_crawl_chaos_resume.py`) — **change the frontier → extend that
  test.** Bot-protection challenges now escalate to the browser at any status.
  `scraper.py::detect_challenge_reasons` is the **one classifier** for both HTTP
  and browser captures; challenge evidence outranks rate-limit retry handling,
  while healthy pages containing embedded Turnstile widgets remain successful.
  Regression coverage: `tests/test_challenge_escalation.py`.
- **2026-07-30** — Crash-resume shipped (`prepare_resume` + the host's
  fail-then-resume reconciliation loop, 3-attempt cap); durable frontier wired
  live via `work_queue_factory`; `http_first` escalates blocked/transport-failed
  HTTP fetches to browser rendering.
- **2026-07-27/28** — One canonical URL-identity layer across crawl, sitemap,
  GSC, GA4, and Bing observations (`web.page.canonical_page_id` + repeatable
  batch reconciliation); response-driven page/resource classification; snapshot
  evidence became complete (all JSON-LD/microdata/RDFa, per-image detail, full
  resource inventory, page identity).
- **2026-07-25** — Canonical persistence failures made page-scoped rather than
  session-fatal; proxy-indicative responses retry direct once behind a run-scoped
  circuit breaker; `low_text_content` demoted to a warning.
- **2026-07-20/24** — All crawler output routed through immutable canonical
  `files.files` UUIDs on private S3 with fail-closed readiness; sitemap sync,
  GSC sync, link resolution, and the re-runnable site initializer shipped;
  www/apex host equivalence; screenshot retention (current + 3 priors).
- **2026-07-21/23** — Link STATUS verification (`/links/check`); the trigger that
  had made it a silent no-op was narrowed at the DB (migration 0240), later
  extended to `target_page_id` so `/links/resolve` works. Any future
  fill-after-insert column needs its own narrow exception, never a blanket one.
