# matrx-scraper engine — fetch, parse, browser, tools

> Cross-repo plan: `/Users/armanisadeghi/code/common-docs/projects/persistent-cloud-browser/PLAN.md` — read it before touching persistent Cloud Browser profiles, Browser Manager lifecycle, WebRTC takeover, or browser-worker execution in any repository.

The durable contract for everything in `matrx_scraper/` **except** the canonical
site crawler, which has its own contract:
[`web_crawl/FEATURE.md`](web_crawl/FEATURE.md).

Verified against live code 2026-08-09.

---

## Entry points

| Call | What it does |
|---|---|
| `scrape(url, **opts)` / `scrape_many(urls)` / `scrape_many_stream(urls)` | Fetch + full parse → `ScrapeResult`. Prefer the streaming variant; it yields per URL as each finishes. |
| `parse_html(html, **opts)` | The parse pipeline alone, no network. |
| `audit_html(html, url)` | The ONE capture-time page-evidence primitive (below). |
| `quick_preview(url)` | robots + homepage SEO audit **+ verdicts** + desktop screenshot. |
| `crawl_site(base_url)` / `SiteCrawler` | Generic BFS traversal. The product crawler is `web_crawl`. |
| `BraveSearchClient` · `PsiClient` · `GscClient` | Pure HTTP clients — return typed values, persist nothing. |

`ScrapeResult` reports outcome as **`success: bool` + `failure_reason: str | None`**.
There is no `status` string and no `error` field — a consumer still reading those
is reading a contract retired in 2026-04.

The parse pipeline is 8 stages: normalize → `NoiseRemover` → `ScrapeFilter` →
`ElementExtractor` → `LinkExtractor` → metadata (extruct) → hashing
(MinHash/SimHash) → markdownify. Parsing is CPU-bound blocking work: `parse_html`
and `scrape` both dispatch the whole pipeline to a worker thread. **Keep those
async boundaries** — including first-use public-suffix/domain discovery — or a
research batch stalls the server event loop.

### Links — two fields, one parse, and why they are not merged

`LinkExtractor` answers two different questions and `ScrapeResult` carries both:

| Field | Shape | Question |
|---|---|---|
| `links` | `{internal, external, images, documents, audio, videos, archives, others}` → `list[str]` | *which URLs does this page point at* |
| `link_records` | `list[dict]`, one per `<a href>` | *how does this page LABEL what it points at* |

A record is `{target_url, anchor_text, text_source, rel, nofollow, link_type,
region}` — key names deliberately match `seo_audit.LinkItem` and the
`crawl_links` rows so every link surface in the package speaks one vocabulary.
`text_source` is `"anchor"`, `"image_alt"` (an image-only link, labelled by its
alt text) or `""` (no label at all). `region` is the nearest landmark ancestor —
`nav | header | footer | aside | main | article | body`. Anchor text is truncated
at `ANCHOR_TEXT_MAX_CHARS` (500) and records are capped at `ANCHOR_RECORD_LIMIT`
(2000), the same limits the SEO audit uses.

**`links` did not change and will not.** Its bucket-of-bare-URL-strings shape is
persisted and read outside this package — matrx-local's
`scrape_store.STORED_FIELDS`, the frontend scraper pages
(`links.internal?.length`, `Record<string, string[]>`), and `crawler.py`'s
no-raw-HTML fallback, which turns the media buckets into `resources` rows.
Widening those entries to dicts would silently corrupt stored rows and break
every consumer doing string work on them, so the richer per-link evidence went
into a NEW parallel field instead. Both are produced by the same parse: anchor
text is the strongest human-authored label a link carries, and once the parse is
discarded it cannot be recovered without refetching the page.

Duplicate `(url, anchor text)` pairs collapse — a nav link repeated in the footer
is one fact. The same URL with *different* anchor text is kept as separate
records; that variation is the signal (internal anchor distribution is a core SEO
input, and `pagerank.py`'s link graph can be weighted by it). Both fields are
governed by the single `ScrapeOptions.get_links` flag and both are skipped by
`fast=True` / `skip_links=True`.

---

## Complete page evidence

`seo_audit.audit_html` returns, from ONE parse of the captured HTML, the normal
SEO metrics plus:

- every JSON-LD script as both original text and parsed JSON, malformed-script
  errors retained rather than discarded;
- flattened entity blocks and all microdata, RDFa, and microformat evidence;
- every image with responsive sources, alt/dimensions/loading attributes, and
  featured-image marking;
- the deduplicated DOM- and structured-metadata-declared resource inventory
  (responsive media, embeds, scripts, styles, fonts, icons, manifests, tracks,
  documents);
- CMS/generator, application/site name, author, publish/modified dates, page
  schema types, theme color, featured-image provenance, related API/shortlink
  URLs, and safe WordPress/Shopify identifiers (checkout tokens are **never**
  retained).

`structured_data.extract_structured_payload` owns the structured-data contract;
the older `extract_structured_data` helper delegates to it. Canonical crawls
persist this evidence inside existing snapshot JSONB columns — **there is no
parallel fetch, table, or parser.** The `[server]` install includes `[metadata]`;
production must never silently degrade microdata/RDFa/microformat capture
because only the core package was installed.

---

## The per-page checks — ONE implementation, in `seo_audit`

`audit_html` says what a page IS; `seo_audit.PAGE_CHECKS` decides whether that
is GOOD. **Both halves live in `seo_audit.py`, and the second half is the ONLY
implementation of a per-page SEO verdict anywhere in the platform.** Until
2026-08-09 the same rules existed three times — here, in
`web_crawl/analysis.py`, and in aidream's `crawl_service.IssueDetector` — with
thresholds that had already drifted apart (thin content fired at 100 words in
one and 300/200/100 in another; canonical mismatch compared raw strings in one
and normalized URLs in another). Consumers now call these functions:

- `PageEvidence` — the checkable struct. Build it from a live audit with
  `evidence_from_audit(audit, http_status=…, redirect_chain=…, …)`, or extend
  it (`web_crawl/analysis.py::PageFacts`) to check a persisted snapshot.
- `PAGE_CHECKS` / `run_page_checks(evidence)` — every check, keyed by name.
- `CheckOutcome` — `status` (`pass|warn|fail|n_a`), `score` (1–100, or `None`
  for `n_a`, matching the `web.analysis_result` constraint), `reasoning`
  (the sentence an expert would say), `issue_count`, `evidence`.

**Every threshold is a named CAPS constant declared exactly once**:
`CONTENT_OK_WORDS` / `CONTENT_WARN_WORDS` / `CONTENT_FAIL_WORDS`,
`IMAGE_ALT_FAIL_RATIO` / `IMAGE_ALT_FAIL_COUNT`, `REDIRECT_CHAIN_MAX_HOPS`,
`LARGE_PAGE_BYTES`, `TTFB_GOOD_MS` / `TTFB_POOR_MS`. The SERP length limits are NOT
redeclared here — `meta_metrics` owns them as a byte-identical mirror of the
frontend TypeScript, and the indexability verdict comes from `audit_metrics`.
Missing evidence is always `n_a`, never a silent pass.

Cross-page work (duplicate titles across a crawl, site rollups) is a different
job and lives in `web_crawl/analysis.py`. Pinned by
`tests/test_seo_checks_single_source.py`, which fails if a check is
re-implemented outside `seo_audit` or a threshold gains a second home.

### Who consumes the verdicts — and the wire layer they use

**Evidence never ships alone.** `title_length: 74` is not a product: the site
owner reading it is a non-technical expert, and a model reading it guesses at a
threshold it will get wrong. Every surface that returns a page audit returns the
verdicts with it. Three consumers, one implementation:

| Consumer | Shape it returns |
|---|---|
| `web_crawl/analysis.py` (site-wide sweep) | `CheckOutcome` per page, rolled up into `web.analysis_result`. |
| `preview.py::quick_preview` (dashboard New Crawl card) | `homepage.checks` = `PageCheckReport` — every verdict, worst first. |
| `ai_tools/specs.py::scraper_audit_html` (agent tool) | `{audit, checks}` = `PageAuditSummary` + `PageCheckDigest`. |

The wire layer lives beside the checks in `seo_audit.py` and decides nothing —
it only projects `run_page_checks`:

- `PageCheckResult` / `PageCheckTally` — one verdict, and the counts.
- `build_page_check_report(evidence)` → **`PageCheckReport`**: all checks, worst
  first. For a surface that renders every row.
- `build_page_check_digest(evidence)` → **`PageCheckDigest`**: the fail/warn
  verdicts in full with their reasoning, passes and `n_a` as bare names. For a
  model — eighteen verbose "this is fine" objects re-enter the prompt on every
  loop iteration and buy nothing.
- `summarize_audit(audit)` → **`PageAuditSummary`**: the evidence, bounded. The
  full `to_dict()` (structured-data blob, resource inventory, every link row) is
  right for persistence and wrong for a tool result; `scraper_audit_html` is
  bounded by construction rather than by the host's generic truncation, which
  would destroy the structure (`matrx-ai`'s `TOOL_RESULT_SIZE_GATE.md`). The
  package cannot set `ToolResult.output_self_capped` itself — it never builds a
  `ToolResult` — so the bound is structural, and pinned by a test.

**Transport facts must be REAL.** `evidence_from_audit(audit, http_status=…,
redirect_chain=…, response_bytes=…, response_time_ms=…, ttfb_ms=…)` takes what the caller
actually captured; `quick_preview` passes its own fetch's status, hop chain,
byte count and latency, and `scraper_audit_html` passes only what the caller
supplies. A `None` yields `n_a` — never a silent pass on evidence nobody had.

Both surfaces are pinned by `tests/test_seo_verdicts_on_live_surfaces.py`: a
known-bad fixture must come back with the expected failing keys, non-empty
reasoning, and exactly the set `run_page_checks` produces.

---

## SSRF — every caller-supplied URL is gated, at the action layer

Every surface that fetches or renders a caller-supplied URL runs the **same two
address gates** — plus, on the two JS-executing surfaces, a third **egress**
gate — because each one is an arbitrary-URL fetch from INSIDE the scraper
network:

1. **Pre-gate** — `validate_public_http_url` before the fetch, and the
   **CORRECTED url it returns** is what gets fetched. Validating one string and
   fetching another is how these gates get walked around.
2. **Post-gate** — re-validate the **final** address after redirects (and, for
   sessions, after any action that can navigate), and **withhold the body** when
   it isn't public. By then the bytes have been read; discarding them is the
   only safe move.

Rejections **never echo the resolver's reason** to a caller — it names the
resolved address, which turns any authenticated caller into an internal-network
resolution oracle. Log it server-side instead; a gate that fires silently
teaches nobody it fired.

| Surface | Gate |
|---|---|
| `crawler` / `web_crawl` / `sitemaps` / `/scrape` | `validate_public_http_url` inline |
| `POST /browser-fetch` | inline in `api/scrape_router.py` |
| `POST /browser-inspect` | inline in `api/scrape_router.py` |
| `POST /browser/*` sessions | **`ai_browser/url_guard.py`**, applied in `ai_browser/actions.py` |
| `POST /preview` | inline in `preview.py::quick_preview` |

The session gate is in the **actions**, not the router, because four consumers
drive those functions — the HTTP router, the MCP server, matrx-ai's `browser_*`
tools, and direct Python callers. A router-only gate protects one of the four.
`guard_proxy` sits in `BrowserSessionManager.create` for the same reason:
`POST /browser/sessions` passes no url at all, only a proxy, which is the same
reach one hop removed.

For sessions the post-gate must cover more than redirects: a `click` on an
internal link and a `type_text` + Enter form submit both navigate somewhere the
pre-gate never saw. So `guard_landing` runs after every navigating action AND
independently inside every reader (`get_html`, `get_text`, `screenshot`,
`get_element`, `query_selectors`, `eval_js`, `wait_for`, `scroll`), and parks the
page at `about:blank` on a miss — two layers, either sufficient alone. A blocked
action returns the normal `success=False` result with `error_type="blocked"`,
which greps out on its own; `about:blank` is reported as `"validation"` / "no
page loaded", never as a security error.

### The third gate — EGRESS (`install_egress_guard`)

Both gates above key on the page's **address**, so neither can see a request the
page **issues**. On a legitimately-public page, `eval_js` with
`fetch('http://169.254.169.254/…')` passed every check and returned internal
content — and so did a public page's own scripts pulling internal content into
the DOM, where `get_text` returns it. `allow_eval_js` is a caller-supplied
request field, so it is an opt-in, not a gate.

`install_egress_guard(context)` (`ai_browser/url_guard.py`) closes it with ONE
context-level `route("**/*")` that applies the **same** `validate_public_http_url`
to every request the page makes — document, subresource, XHR, fetch — whatever
issued it. Blocked requests are aborted with `blockedbyclient`.

- **Cost is bounded by a per-context host memo:** one DNS resolution per unique
  `(scheme, host, port)` for the life of the context, not one per request. The
  memo is part of the contract — a test asserts it, because losing it puts DNS in
  front of every subresource.
- **Fails CLOSED.** A request whose verdict can't be established — including a
  bug in our own handler — is aborted and screamed about, never allowed.
- **Installed on exactly two contexts:** `BrowserSessionManager.create` (the
  `eval_js` surface) and `preview.py::_take_homepage_screenshot` (renders an
  arbitrary caller URL). **Deliberately NOT the crawler's browser pool** — it
  runs no caller-supplied JS, and a Python callback in front of every subresource
  of every page of every crawl is a real cost on the hottest path we have.
  Widening it is a measured decision, not a reflex.

Tests: `tests/test_browser_session_ssrf.py` (public-page-fetches-internal, the
memo, fail-closed, non-network schemes, and a structural pin on both install
sites — the guard only works where it is installed, and each is one easily
deleted line).

---

## Proxy and extraction safety

`fetch_normally_with_proxy` means proxy-only: it requires the comma-separated
`DATACENTER_PROXIES` pool, rotates to a different proxy after a retryable
failure, and **never falls back to a direct request**. A missing pool raises
`ProxyConfigurationError` instead of silently exposing the host IP. (The retired
`PROXY_DATACENTER_URL` / `PROXY_RESIDENTIAL_URL` fields were never read by the
fetch engine.)

CPU-heavy PDF/image extraction has explicit async entry points
(`extract_text_from_pdf_bytes_async` / `extract_text_from_image_bytes_async`)
which run the synchronous OCR/PDF stack on a worker thread. **OCR must never run
on the API event loop.**

---

## Per-host politeness — ONE throttle memory for every lane

`matrx_scraper.rate_limiter` is the only place a Matrx process decides how hard
it may hit a host. It offers two shapes over one shared memory, and **no caller
may keep a private gate** — that is what let the research auto-scraper hammer a
host the crawler had just backed off from:

- **`acquire(url)`** — token bucket. The crawler's shape: N workers on one host
  stay under `host_rps` in aggregate but may run concurrently up to the burst.
  Raises `TimeoutError` past `max_wait_seconds`; the caller decides.
- **`slot(url, min_interval=…, jitter=…)`** — serialize + end-gap. The research
  auto-scraper's shape: same-host fetches run one at a time, separated by at
  least `min_interval` measured from the END of the previous request, so a slow
  response never stacks the next on top of it. Different hosts stay fully
  concurrent. A slot is politeness, never a gate: an unresolvable host passes
  straight through and a bucket timeout logs and proceeds.

**`throttle_host` / `observe_status` record a per-host backoff FACTOR in a
process-wide store**, so a 429/503 seen by any lane slows every other lane on
that host — the bucket rate is multiplied by it and the slot interval divided by
it. A factor, not an absolute rps, because the lanes have different baselines
(4 rps vs one request per 2s); the lesson is "this host wants half as much".
Throttles expire after `THROTTLE_TTL_SECONDS` (15 min) without a fresh 429, so
one bad minute doesn't pin a host for the life of the process.

`host_key(url)` is THE politeness key — lowercased registrable host, port and a
leading `www.` stripped, so `www.x.com` and `x.com` share one budget (deeper
subdomains are deliberately kept separate: `blog.` and `shop.` are often
different infrastructure, and `*.github.io` is different owners). Keying on the
raw netloc let a burst walk through by alternating the prefix.

Because the store is process-wide it leaks between tests — both suites clear it
in an autouse fixture (`clear_shared_throttles()`).

---

## Browser runtimes — the package owns both

Nothing in `matrx_scraper/` defers to a host-side browser pool, host-installed
binary path, or host-managed session manager.

| Module | Purpose | Used by |
|---|---|---|
| `browser_pool.py` — `PlaywrightBrowserPool` | Fixed-size pool of headless Chromium browsers; the single fail-loud multi-kind screenshot capture (`fetch_with_capture`). | `SiteCrawler`, `web_crawl`, stateless throughput. |
| `ai_browser/session.py` — `BrowserSessionManager` | Long-lived per-`session_id` pages with TTL eviction, for AI loops that navigate once then issue many follow-up calls. | `ai_browser/actions.py`, the MCP server, matrx-ai's `browser_*` tools. |

Both degrade to a clear `ImportError` when Playwright is absent, and share no
global state. Pool size is a code constant (`DEFAULT_BROWSER_POOL_SIZE`), never
configuration. matrx-ai's `browser_*` tools are already thin consumers of
`matrx_scraper.ai_browser.RemoteBrowserClient` — there is ONE session manager;
never reintroduce a second.

> 🚧 **A THIRD runtime is specified and in Phase-0 proof — the Persistent Cloud
> Browser (Browser Manager).** Durable per-user/per-organization Chromium profiles
> that stay logged in, run headed on a private display so a human can take over the
> live browser mid-run, and persist as encrypted checkpoints between runs. **It
> lands in THIS package** as a capability beside the two above — deliberately NOT a
> new package and NOT a second browser API. **Before adding persistent contexts,
> profile directories, session persistence, or human-takeover streaming anywhere,
> read the cross-repo authority:
> `common-docs/projects/persistent-cloud-browser/PLAN.md`** (architecture) and its
> `EXECUTION.md` (workstreams) / `CONTRACTS.md` (the frozen interfaces). Nothing is
> built yet; proof harnesses live in [`proofs/cloud_browser/`](../proofs/cloud_browser/)
> and are NOT shipped code. The existing `BrowserSessionManager` keeps its current
> transient behaviour — persistent-profile runs are the new path, not a rewrite of
> the old one.

### `POST /browser-fetch` — the stateless sibling

Renders ONE public http(s) URL in the pooled Chromium and returns a typed
`BrowserFetchResult` (`success, status_code, final_url, title, content_type,
html, error, truncated`). No session, no DOM automation. It exists for
Chromium-less hosts — aidream ships no browser, so its research auto-scraper
escalates blocked/JS-shell HTTP fetches here instead of parking the source.

Guards (tested in `tests/test_browser_fetch_endpoint.py`) are the contract: the
two SSRF gates above; `timeout_ms` capped at 30s (each in-flight call holds one
of the pooled browsers shared with live crawls); HTML truncated at
`BROWSER_FETCH_MAX_HTML_CHARS` with `truncated=true`; `use_proxy=true` by
default so a retry against a host that already blocked us doesn't burn the
scraper's own IP; **never 500s** — a missing proxy pool or render exception
returns `success=false` with `error` set and logs loudly; `503` when the
deployment has no browser pool. Not covered: DNS-rebinding TOCTOU, since
Chromium resolves independently of the pre-flight check.

### `POST /browser-inspect` — the render-CHECK sibling

Navigates ONE public http(s) URL and returns what the BROWSER saw, not what a
parser can reconstruct: `http_status`, `final_url`, `title`, `console_errors`
(console.* errors AND uncaught `pageerror`s), `failed_requests`, an optional
DOM text/selector probe, and device-profile screenshots as base64.

It is a third endpoint on purpose. `/browser-fetch` returns HTML, and the
`/browser/*` session API drives the DOM — **neither exposes the console or
failed-request streams**, and those only exist while the page is open, so no
caller holding the response can recover them. Answering "did this page render
correctly for a visitor" needs them. The primitive is
`PlaywrightBrowserPool.inspect_url`, beside `fetch` and `capture_url`.

Contract differences from `/browser-fetch`, each deliberate:

- **Wait discipline is stricter** — `wait_until="load"` → bounded best-effort
  `networkidle` → fixed settle, under one hard budget. Reporting "no console
  errors" for a page whose scripts had not run yet is a false pass.
- **`use_proxy` defaults FALSE** — the caller is checking a site it owns;
  routing that through the shared residential pool buys nothing.
- **Kinds are grouped by device profile**, one navigation each (capped at
  `BROWSER_INSPECT_MAX_KINDS`). Console/failed-request evidence merges across
  every navigation — a mobile-only script error is a real defect — while
  status/title/DOM probes come from the first, so the result is deterministic.
- **`user_agent_suffix` is APPENDED to the profile UA**, never a replacement —
  this endpoint's job is "inspect this page as the device", so its identity
  stays the device's. A caller who wants to *replace* the UA outright wants the
  crawl-level override (`CrawlStartRequest.user_agent`, one gate in
  [`user_agents.py`](user_agents.py)), which swaps only the UA string and leaves
  viewport / `is_mobile` / `has_touch` / DPR intact — see
  [`web_crawl/FEATURE.md` § User-Agent override](web_crawl/FEATURE.md).

Same guards as `/browser-fetch`: both SSRF gates (including discarding the
whole result, pixels included, when the browser's redirect chain lands
off-public), a 30s `timeout_ms` ceiling, a `503` with no browser pool, and it
**never 500s** — failures return `success=false` with `error` and a `retryable`
flag. First consumer: `aidream/services/cms_verify` (see its FEATURE.md for why
its previous in-process pool left the feature dead on the server for a month).

---

## AI-callable surface and MCP

`matrx_scraper.ai_tools.ALL_TOOLS` is the ONE descriptor list — 14 browser + 4
scrape `ToolSpec`s, each carrying `name`, `description`, `input_schema`
(full JSON Schema), and an async `handler(args) -> dict`. Every consumer reads
that same list: matrx-ai's registry, provider tool-call payloads, the bundled MCP
server, and plain Python callers.

**Adding an AI-callable operation is a one-place edit:** implement the async
function in its submodule, append a `ToolSpec(...)` to the matching list in
`ai_tools/specs.py`. Every consumer sees it automatically.

🚨 **There are deliberately NO crawl tools here.** Site crawling is the canonical
`web.*` crawler ([`web_crawl/FEATURE.md`](web_crawl/FEATURE.md)), reached over
HTTP through `api/crawl_router.py`. The four `crawl_start` / `crawl_status` /
`crawl_pages` / `crawl_cancel` descriptors addressed the retired
`scraper.crawl_runs` world and were deleted with it on 2026-08-10 (unregistered
in `tool.definition`, no host ever wired their `_ext` handlers, every call
raised). An agent entry point to the canonical crawler is a NEW tool over
`web.crawl_session` — never a revival of these.

MCP server: `python -m matrx_scraper.mcp` (stdio — the ONLY transport), plus
`--list` and `--groups browser,scrape`. It uses the official `mcp` SDK when
installed and falls back to a minimal JSON-RPC-over-stdio loop otherwise.

> **The MCP server has no network transport, deliberately.** An HTTP adapter
> (`POST /tools/call`) existed with zero authentication and zero consumers — it
> handed any caller a JS-executing browser — and was deleted 2026-08-09. Do not
> re-add one: the package's authenticated network surface is the FastAPI
> microservice, and adding a second one duplicates the auth boundary.

---

### Readers declare what they cut — no silent truncation

Every `ai_browser` reader caps its payload, and **a cap that fires is visible to
the model, not just to the schema**: `_cap_text` appends an inline
`[truncated: showing X of Y characters]` marker and fills `truncated` plus
`total_chars` with the pre-truncation size. `query_selectors` also returns
per-selector `match_counts` and a notice row when its element cap fires.

Caps are named once at the top of `ai_browser/actions.py` and imported by the
router, remote client, and AI-tool specs. There is no offset/range reader, so
the marker promises no paging that does not exist. The contract and the
single-source bounds are pinned by `tests/test_browser_reader_truncation.py`.

---

## Host injection seams

The package runs standalone; a host adds durability by implementing protocols.
Defaults are in-memory / no-op / discard.

| Protocol | Default | What a host swaps in |
|---|---|---|
| `CrawlEventSink.emit(event)` | `NoopEventSink` | A durable writer + live broadcast (`web_crawl`'s `DurableCrawlEventSink`). |
| `QueueBackend` | `InMemoryQueueBackend` | `RuntimeWorkQueueBackend` (`[durable]` extra) — the crash-safe frontier. |
| `BodyPersister.persist(request)` | discard | The canonical `matrx-files` writer. |
| `RecipeBackend.get_for_host(host)` | `StaticRecipeBackend(DEFAULT_RECIPES)` | A DB-backed playbook store. |

`configure_ext(**kwargs)` (`_ext.py`) is the object registry. The **real** keys —
grep `get_ext(` before adding one:

| Key | Wired by | Consumed by |
|---|---|---|
| `cache` | host / standalone server | `api/ext_router.py`, `api/scrape_router.py` (`/page-capture`), `service.py` (the quick-scrape lane: `use_cache=True` on the request resolves this ext via `ScrapeService._resolve_cache` and threads it into `scrape_many_stream`; absent ext = uncached scrape, pinned by `tests/test_quick_scrape_cache.py`) |
| `domain_config` | host / standalone server | `ext_router`, research `domain_policy` |
| `browser_pool` | standalone server | `api/scrape_router.py` |
| `file_manager` | host / standalone server | canonical artifact persistence |
| `work_queue_factory` | host / standalone server | `web_crawl/service.py` — **absent means every crawl silently runs the in-memory frontier and dies with its process.** |
| `google_credential_resolver` | embedded host | `web_crawl/gsc_sync.py` — resolves the host-owned Google credential without coupling the package to a host process or vault; the Matrx standalone service uses its explicit aidream bridge only when this seam is absent. |
| `growth_stage_tracker` | optional embedded host | `web_crawl/service.py` — observes crawl/analyze lifecycle without importing a host growth-loop concept; absent in standalone mode is a no-op. |
| `wrapped_brave_search` | optional | `search` |
| `google_credential_resolver` | host | `web_crawl/gsc_sync.py` — async `(credential_ref, site_organization_id, user_id, resource_type?, resource_ref?, site_id?, provider_binding_key?) -> {refresh_token, client_id, client_secret}` (any object with those attrs also works, so a host never imports our type). **This is how a non-Matrx host does GSC at all.** Absent, the module falls back to the platform bridge the Matrx standalone microservice is deployed alongside (`{AIDREAM_URL}/api/google-integrations/internal/credential`); that fallback is for that one deployment, never for a consumer. |

aidream wires all of it in ONE function: `_configure_matrx_scraper()` in
[`aidream/package_integration.py`](../../../aidream/package_integration.py).

### Credentials are NOT env-only — `search.configure_client`

`BraveSearchClient()` still reads `BRAVE_SEARCH_API_KEY` / `BRAVE_API_KEY` from
the environment, and the server hosts keep working untouched. But **the
environment is not a valid credential source on every host.** A downloaded
desktop client (matrx-local) keeps the *user's own* Brave key in an in-app key
store, settable at runtime with no process restart; an env lookup there means
the key is permanently absent. So keys are injectable:

```python
BraveSearchClient(api_key=..., ai_api_key=..., ai_key_is_pro=...)
search.configure_client(api_key)   # install/replace the process-wide client
search.configure_client(None)      # user removed their key — clear it
```

`configure_client(None)` clearing the singleton is part of the contract: a
removed key that keeps answering from a cached client is worse than no key.

Related: `_key_for(use_ai_plan=True)` now **falls back to the base key** when no
AI key is provisioned. `extra_snippets=True` is the default, so raising there
turned every single-key install into a totally broken one — extra snippets are
an enrichment, not a precondition.

### Where the result shape is defined — ONCE

`ScrapeOptions` + `apply_field_flags` live in
[`scrape_options.py`](scrape_options.py), NOT in `service.py`. `service.py` is
the SSE/FastAPI adapter and hard-imports matrx-connect and the matrx-orm
models; a consumer that only wants "scrape these URLs, give me text_data and
links" could not reach the options shape without pulling a DB stack it has no
use for — and would write its own copy, which is how the two ends drift. A
tripwire test asserts importing `scrape_options` loads neither `matrx_orm` nor
`matrx_connect`. `service.ScrapeOptions` / `service._apply_field_flags` stay as
re-exports and keep working.

`search.extract_urls_from_search_results(queries_with_results)` is the
counterpart to `generate_search_text_summary`: the same Brave payload, rendered
for a *scrape pipeline* instead of a reader. Deduplicated, rank-order
preserved, so a caller truncating to an effort budget keeps the best results.
Every search-then-scrape host uses it rather than re-walking `web.results`.

---

## HTTP surface and auth

Routers mount under `/api/scraper` on the standalone service. `/health` and
`/health/ready` are public; everything else authenticates.

| Group | Paths |
|---|---|
| Scrape / search | `POST /quick-scrape` (streaming default) · `/page-capture` (one typed, cache-backed parsed page for server enrichment; failed captures retain both the category in `failure_reason` and the underlying status/transport explanation in `failure_details`) · `/batch` · `/search` · `/search-and-scrape` · `/search-and-scrape-limited` · `/browser-fetch` · `/browser-inspect` |
| Browser sessions | 16 endpoints under `/browser/*` (create, navigate, click, fill, type, select_option, screenshot, wait_for, get_element, query_selectors, eval_js, scroll, get_html, get_text, list, close) |
| Preview | `POST /preview` |
| Retry queue + config | `/content/save` · `/queue/{pending,claim,submit,fail,stats}` · `/config/domains` |
| Canonical crawler | `/crawler/*` — see [`web_crawl/FEATURE.md`](web_crawl/FEATURE.md) |

**Two accepted identities, 401 without one:**

1. **A real user login** — a Supabase user JWT as `Authorization: Bearer <jwt>`.
   `X-API-Key` is not supported.
2. **An approved server** — for server-initiated calls with no person in the loop
   (aidream's browser tools, unattended preview/cron): the shared
   `ADMIN_API_TOKEN` (= aidream's `MATRX_SCRAPER_TOKEN`) as the bearer, plus
   `X-Matrx-User-Id` naming the user to attribute the work to. The work is then
   scoped to THAT user. The token is a *shared secret compared constant-time* —
   **never a user login, never bound to a fixed admin user.**

Implementation is the canonical `matrx_connect.service_auth` primitive (shared
with the sandbox bridge and media-healer), wired onto the routers in
`server/app.py`. It is NOT part of the JWT-only auth middleware.

**Canonical-crawler exception:** `/crawler/*` requires a real Supabase user JWT.
Approved-server authentication is not accepted there.

Stream wire format is NDJSON, one JSON object per line
(`{"event": "...", "data": {...}}`) — the same shape as every aidream streaming
endpoint. Parse each line independently.

---

## Engine persistence — `scraper.*` on the ONE database

The engine's own tables. They live in the SAME Postgres (Matrx Main) as the
crawler's canonical `web.*` — one resolver
(`matrx_scraper.db.register_platform_db`, `SUPABASE_MATRIX_*`) binds both, so
they can never point at different databases. There is no scraper-specific
database and no service-local connection URL: until 2026-08-09 the standalone
service could bind `scraper.*` to a Coolify-local Postgres, and it did — 14 rows
the platform never saw (since migrated into Matrx Main; the fallback is deleted).

| Table | Purpose | Owner in code |
|---|---|---|
| `scrape_parsed_page` | Successful scrape output; L2 of the two-tier cache, TTL via `expires_at`, one `active` row per `page_name`. | `cache.py::TwoTierCache` |
| `scrape_domain` / `_domain_settings` / `scrape_path_pattern` / `_path_override` | Per-domain scrape permission, proxy tier, per-path filter overrides; refreshed every 300s. | `domain_config.py::PostgresDomainConfigStore` |
| `scrape_failure_log` | Append-only failure diagnostics on a retryable category. | `service.py` |
| `scrape_retry_queue` | Lease-based tiered retry queue (`pending`/`claimed`/`completed`/`failed`/`expired`; tiers `desktop` → matrx-local, `extension` → the Chrome extension). | `service.py` + `api/ext_router.py` |

**The retry pipeline** is the reason those tables exist: tier 1 is the scraper
service (datacenter IP + proxies); a retryable failure (`cloudflare_block`,
`blocked`, `bad_status`, `request_error`, `proxy_error`) writes a failure log +
queue row; tier 2 (matrx-local, residential IP) and tier 3 (the extension,
manual navigation) poll `/queue/pending`, claim with a 10-minute lease, scrape
locally, and `POST /queue/submit`. Every success path lands in
`scrape_parsed_page`.

**Access is matrx-orm Models only** — `db/models_scraper.py`. No raw SQL, no
`pool.acquire()`, anywhere. `TwoTierCache` / `PostgresDomainConfigStore` still
accept a `pool` constructor arg for host-wiring back-compat and **ignore it**.

🚨 **They resolve their models in `__init__`, never at module scope.** matrx-orm
is the `[db]` extra, so a module-scope `from matrx_scraper.db.models_scraper
import …` in `cache.py` / `domain_config.py` made the extra mandatory for
everyone: `from matrx_scraper.cache import MemoryCache` — the first line of
matrx-local's no-DB desktop lane — raised `ImportError: cannot import name
'PLATFORM_DB_ENV_PREFIX' from 'matrx_orm'` on any host whose matrx-orm predated
the ONE-database resolver (fixed 2026-08-09; `[db]` now floors matrx-orm at
`>=3.1.49`, the first release that exports it). Constructing the DB-backed class
IS the declaration that the consumer has the extra — that is where it may fail,
and it fails loud. Pinned by `tests/test_host_independence.py`.
`configure_db(<host db name>)` binds the package's model sets to the host's
existing matrx-orm pool; never open a second Supabase pool.

`PostgresDomainConfigStore.start()` surfaces an initial-load failure with a
`CRITICAL` banner naming the exact consequence (scrape permission / proxy / path
rules INERT, `is_scrape_allowed` fails OPEN). It deliberately does not raise —
that would abort the host's whole scraper init — and the periodic refresh keeps
retrying so a boot-time DB blip self-heals.

### 🚨 The `scraper.crawl_*` tables are DEAD

`scraper.sites`, `scraper.site_pages`, and every `scraper.crawl_*` table belonged
to the retired admin crawler. **The canonical crawler is `web_crawl/` on Supabase
`web.*` and never writes or falls back to them.** Their remaining package-side
code — `persistence/`, `scheduling.py`, `db/{site,metrics,preset,extractor}_queries.py`,
and the `Crawl*` models in `db/models_scraper.py` — had exactly one consumer,
`aidream/services/scraper/`, and is deleted with it. Do not build on it. Deletion
is tracked in [`FOUND_DEFECTS.md`](../../../FOUND_DEFECTS.md).

---

## Google provider clients

`performance.py` owns the reusable HTTP clients for PageSpeed Insights v5 and
Google Search Console. There is exactly one of each.

**Provider response JSON is decoded on a worker thread.** PageSpeed and Search
Console bodies can be large enough to freeze every request on the host process;
all async client paths go through `performance._decode_response_json`, never a
direct `response.json()` call on the event loop. A structural test rejects any
future direct JSON decode inside an async function in this module.

`PsiSnapshot` carries request evidence for durable downstream collection: HTTP
status, response headers, request count. Successful calls retain the exact
provider JSON in `raw`. HTTP failures retain parsed provider error JSON, or a
bounded 2,000-character body when the provider returns non-JSON; transport
failures retain their typed error. These let `matrx-seo` persist failed attempts
and quota diagnostics **without a second PageSpeed HTTP client**. A malformed
HTTP-200 body — non-object, or missing `lighthouseResult` — is returned as a
typed failure with its raw evidence, never an empty successful audit.

`scraper.psi_metrics` remains current-state storage keyed by URL and strategy for
dashboard compatibility. Historical append-only samples belong to
`seo.page_performance_snapshot` through the matrx-seo adapter; both paths
intentionally consume the same `PsiClient`.

`GscClient` is likewise the ONE OAuth access-token cache/refresh implementation,
for both the canonical crawler sync and matrx-seo history. The crawler exports a
single-page Search Analytics primitive with raw status/headers for the SEO
bridge; it does not own SEO normalization or `seo.*` persistence.
`python -m matrx_scraper.gsc_bootstrap` mints a refresh token with no host code.
It discovers an optional `.env` from the operator's current project directory,
never by walking above the installed package path.

---

## Canonical SEO metadata rules

`meta_metrics.py` is the ONE Python source for metadata measurement and policy.
It owns the character-width table, font sizes, device pixel allowances, character
windows, status booleans, and issue wording. The scraper, crawl issue detector,
public page audit, registered functions, and matrx-ai's SEO tool **call it rather
than declaring thresholds.**

Current product policy: title 15–60 characters at 600px, description 70–160
characters at 920px. Desktop and mobile intentionally share allowances. The exact
TypeScript mirror is `matrx-frontend/features/marketing/seo/serp/metrics.ts`; its
parity test is generated from this Python implementation.

## Robots.txt — one parser and explainable verdict

`robots_txt.parse_robots_txt` is the package's only robots parser. It implements
longest-match precedence, `*` and `$`, exact-tie `Allow` preference, agent-specific
groups, sitemap discovery, the 500 KiB read limit, and line-numbered syntax errors.
`RobotsDocument.decision_for(path, user_agent)` returns both the verdict and the
winning source rule; `is_allowed` delegates to it. Consumers must show this evidence
instead of re-deriving a rule or returning an unexplained boolean.

---

## Deployment (the standalone service)

Built from `packages/matrx-scraper/Dockerfile` (uv workspace + Playwright/Chromium
+ Tesseract), port 8001, 4 GB cap, deployed on Coolify at
`scraper.app.matrxserver.com` (dev: `dev.scraper.app.matrxserver.com`). **Chromium
runs in this container only** — it is the single source of truth for Playwright
execution; aidream deliberately ships no browser and reaches it over
`MATRX_SCRAPER_URL` (`http://scraper-service:8001` internally, no public
hairpin).

**ONE database, both schemas.** The engine's `scraper.*` tables and the canonical
crawler's `web.*` tables live in the same Supabase project (Matrx Main), reached
through the same `SUPABASE_MATRIX_*` env vars via the one
`register_platform_db()` resolver — `bootstrap_db()` registers `matrx_scraper`,
`bootstrap_web_db()` registers `matrx_web`, and neither accepts a service-local
connection URL. Both bootstraps RAISE if the platform DB cannot be resolved.
Crawl artifact bytes go to private S3 through `matrx-files`.

`GET /health` reports the installed package version, so a stale deployment is
directly observable. `GET /health/version` is the full build-identity probe
(mirrors aidream's contract): `{service, version, git_sha, built_at, branch}` —
`git_sha` resolves `GIT_SHA` (Docker build arg) then `SOURCE_COMMIT` (Coolify
runtime injection) and degrades to `"unknown"`, never raising. Public like the
other health routes; pinned by `tests/test_health_version.py`.

**No startup capability is optional — every one either boots or crashes the
service.** There is nothing left in `_lifespan` that degrades: `bootstrap_web_db()`,
the files/S3 bootstrap, `bootstrap_db()`, the page **cache**, the
**domain-config** store, the **browser pool** (on an image that has Playwright),
and the durable work-item frontier all raise on failure. Being "optional" made
sense only while the `scraper.*` binding itself might legitimately be absent; it
became fatal on 2026-08-09 (`ee01821d6`), so every capability init now runs
against a pool that already bound and probed. An exception past that point is a
missing table, a bad grant, or schema drift — never "the optional thing isn't
here" — and **nothing retries any of them**, so a degraded server stays degraded
forever while answering `/health` with 200. A refused boot is visible in Coolify
in seconds; a silent degradation re-fetches every page at full cost, or ignores
every per-host policy rule, for weeks.

The failure is a **self-identifying red `vcprint` banner** naming the capability,
the underlying exception, what breaks if we continued, and the operator's next
action — then a short `ScraperStartupError` pointing at it (its own type, so a
boot failure greps apart from every other `RuntimeError`). One helper owns the
shape: `server/app.py::_fatal_capability`. Adding a capability to `_lifespan`
means routing its `except` through that helper, never a `print`.

`GET /health/ready` is the **independent second layer** — if a boot gate were
ever softened back into a warning, readiness still refuses to call the server
healthy. It fails 503 on `web_database`, `files_database`, `file_manager`,
`canonical_file_pipeline`, `s3`, `cache`, `domain_config`, and — on any image
where Playwright is importable — `browser_pool`. `database`/`orm` stay visible
but don't block, and nothing probes retired `scraper.crawl_*` tables. Both
layers, and the banner's content, are pinned by
[`tests/test_startup_capability_gates.py`](../tests/test_startup_capability_gates.py);
its regression proofs are the point — reverting a gate to a warning, or keeping
the crash while dropping the scream, each fail it.

Two of these guards were written from live bugs. `domain_config` used to be
required by readiness only `if has_ext("domain_config")` — i.e. only once it had
already started successfully, the one case that cannot fail — so a store that
FAILED to start reported READY while every fetch ran on default policy. And
`bootstrap_db()` failing used to be swallowed as a warning, leaving cache, domain
policy, and the retry queue silently dark.

Required env: `SUPABASE_MATRIX_URL`, `SUPABASE_MATRIX_SECRET_KEY`,
`SUPABASE_MATRIX_JWT_SECRET` (HS256 compatibility; JWKS derives from the Supabase
URL for ES256), `SUPABASE_MATRIX_{HOST,PORT,DATABASE_NAME,USER,PASSWORD}`,
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`,
`AWS_S3_DEFAULT_BUCKET`, `DATACENTER_PROXIES`, `ADMIN_API_TOKEN`,
`BRAVE_API_KEY`, `PORT`.

> **There are NO feature toggles in this service's env, and adding one is a
> defect.** The cache, the domain-config store, and the browser pool are
> unconditional; `ENABLE_CACHE` / `ENABLE_DOMAIN_CONFIG` / `ENABLE_BROWSER_POOL`
> were deleted 2026-08-09 (unset on every real deployment, so they only ever
> defaulted to on). What genuinely varies by image shape — is Playwright
> installed? — is answered by the real capability probe
> `browser_pool.PLAYWRIGHT_AVAILABLE`, which cannot drift from reality the way a
> host-by-host env var does. Same reasoning as `DEFAULT_BROWSER_POOL_SIZE`.

---

## Test warning hygiene

Browser-fetch validation uses Starlette's current 422 constant, and async tests
run on pytest's managed event loop rather than creating unclosed loops. `pyRdfa3`
3.6.5 is the current upstream release but still calls the deprecated
`datetime.utcnow()`; pytest filters only that exact upstream module warning so
project-owned datetime deprecations stay visible.

---

## Change log

- 2026-08-17 — **Persistent Cloud Browser pointer added to Browser runtimes.** A
  third browser runtime (Browser Manager — durable logged-in profiles, headed
  human takeover, encrypted checkpoints) is specified and in Phase-0 proof, and it
  lands in this package. The cross-repo authority is
  `common-docs/projects/persistent-cloud-browser/`; proof harnesses under
  `proofs/cloud_browser/` are not shipped code. Recorded here so nobody builds a
  competing persistent-session path against the transient
  `BrowserSessionManager`.
- 2026-08-15 — **The site soft-delete backfill follows the generated FK
  inventory instead of importing retired tables.** `web.batch_item` and
  `web.batch_job` were removed, but the operator script kept importing their
  generated models and could no longer start. It now resolves every current
  `web.site` descendant from `Site._inverse_foreign_keys`, processes exactly
  those carrying `deleted_at`, and explicitly recognizes
  `endpoint_family_sweep_state` as the one non-soft-delete child. Any new
  non-soft-delete child fails loud until its lifecycle is deliberately ruled.
- 2026-08-15 — Robots decisions now retain the exact winning directive and source
  line, so every crawler, catalogue check, and public tester can explain a verdict
  from the one shared parser.
- 2026-08-14 — `/page-capture` stopped collapsing scrape failures to category
  tokens such as `request_error` / `bad_status`; its typed response now carries
  `ScrapeResult.failure_details` so schedulers and operators see the actual
  transport exception or HTTP status without opening scraper logs.
- 2026-08-11 — `/page-capture` can optionally produce a backlink link-evidence
  screenshot through the same pooled Playwright and canonical files/S3 pipeline
  used by crawls. A deterministic DOM action highlights matching target anchors
  and centers the first match before a desktop viewport capture. The file remains
  org-internal and immutable; screenshot failure is reported separately and never
  discards a successful parsed-page capture. If a third-party font request alone
  exhausts Playwright's screenshot budget, the canonical capture primitive stops
  outstanding page loads, resolves Playwright's internal font-readiness gate for
  that one retry, and tries again; this preserves the rendered DOM while preventing
  an irrelevant font host from suppressing visual evidence. The shared pool also
  starts Playwright with its upstream font-readiness bypass: our own bounded DOM
  settle remains authoritative, and a screenshot may use the browser's rendered
  fallback font rather than fail because decoration never arrived. Rasterization
  and PNG encoding have a 20-second ceiling inside the pool's existing
  30-second-per-image lease, avoiding the former five-second cliff on heavy pages.
  After the bounded DOM settle and any highlight/scroll recipe, the shared capture
  freezes outstanding page loads and disables CSS animations before rasterizing;
  this preserves the chosen evidence frame instead of racing third-party churn.
  The parsed backlink scrape remains proxy-backed; its separate human-review
  screenshot renders directly after the same public-URL gates, avoiding a second
  proxy session whose slow asset tunnel can keep Chromium's renderer busy.

- 2026-08-11 — The existing site probe now performs one verified TLS handshake
  against the canonical HTTPS hostname and persists certificate trust,
  hostname match, issuer, expiry, and failures in
  `web.site.metadata["site_probe"]`. The catalogue's `tls_certificate` bands
  therefore score real evidence; missing or failed evidence remains `n_a` with
  the existing one-click site recrawl remediation.

- 2026-08-10 — Added `POST /page-capture`, the non-streaming server primitive
  for backlink/source enrichment. It applies both public-URL gates, reuses the
  two-tier parsed-page cache, returns bounded analysis text plus only link
  records matching the requested target, and leaves the complete parsed page
  durable in `scraper.scrape_parsed_page`.

- 2026-08-09 — `ai_browser` readers expose truncation inline and structurally;
  every cap has one named source and forcing-function coverage.

- 2026-08-09 — **0.1.106: the SEO tranche `main` had written but never
  published actually ships — and three tags are recorded as off-`main`.**
  `v0.1.99` (278a46924) and `v0.1.100` (715e2ac04) live only on the abandoned
  `port/matrx-scraper-desktop-consumer` branch, and `v0.1.105` (b0b2acae3) is a
  dangling pre-rebase commit whose tree predates the whole SEO tranche. PyPI's
  0.1.105 therefore contains none of `robots_txt.py`,
  `web_crawl/site_analysis.py`, `web_crawl/site_probe.py`, or
  `web_crawl/analysis_write.py` — ~11.6k lines that sat on `main` under a
  version number already taken. `main`'s equivalents of 0.1.99/0.1.100 landed
  as 0.1.101–0.1.105 (the empty-`text_data` bug was fixed at the source in
  93bc74db1, superseding 0.1.100's fallback), so that branch is fully
  superseded — **merging it would revert `main`, not advance it.** Rule this
  leaves behind: never re-tag a version PyPI already has, and never tag a
  commit that is not an ancestor of `origin/main` —
  `scripts/publish-package.sh` bumps the pyproject on whatever branch you are
  standing on, so run it on `main` or the tag captures a tree nobody else has.
- 2026-08-09 — **Archives are classified as archives again; the `archives`
  bucket stops being permanently empty.** `_determine_filetype`'s document
  branch also tested `EXTENSION_PACK["archive"]` and returned `"document"`
  first, so the later `return "archive"` was unreachable and every
  zip/rar/7z/tar/gz/bz2/iso/dmg/tgz/xz link landed in `links["documents"]`.
  Two consumers were reading a category nothing ever filled: `crawler.py`'s
  `("archives", "archive")` resource mapping (so no page ever reported a
  `kind="archive"` resource) and the published `PageLinks.archives` field in
  `api/routers/ts_types/scraper.types.ts`. Because both expect content, the
  archive bucket is the intended behaviour and the document branch lost its
  archive test — **this moves archive URLs out of `documents` and into
  `archives` for newly stored `links` payloads; already-stored payloads keep
  the old placement.** Pinned by
  `test_desktop_consumer_ports.py::test_archives_are_not_classified_as_documents`
  and `::test_extract_fills_the_archives_bucket_separately_from_documents`
  (the pre-existing `test_media_extensions_classify` case asserted the buggy
  `.tar.xz → document` placement and was corrected).
- 2026-08-09 — **Anchor text survives the parse: `ScrapeResult.link_records`.**
  `LinkExtractor.extract()` discarded the visible text of every link, and it is
  unrecoverable without refetching — so a page's own labels for what it points
  at were lost at parse time. `extract_anchors()` now emits one record per
  `<a href>` (url + anchor text + text source + rel/nofollow + link type +
  page region) from the SAME parse. `links` was left exactly as it was —
  bucket → `list[str]` — because it is persisted and read by matrx-local
  (`scrape_store.STORED_FIELDS`), the frontend scraper pages, and `crawler.py`'s
  fallback resource builder; widening it in place would have corrupted stored
  rows. Pinned by `tests/test_anchor_records.py` (capture, empty/whitespace
  text, image-only links, same URL under different anchors, caps, the frozen
  bucket shape, and the fast-mode skip).
- 2026-08-09 — **`text_data` fixed at the source; `parser/scrape_json_to_text.py`
  deleted.** `_build_text_data` handed the `OrganizedData` OBJECT to
  `json_to_text_lines`, a flattener written for a legacy nested-dict page shape
  (`{"H1: Title": {...}, "Lists": [...]}`) that nothing produces any more. It
  matched no branch and returned "" for EVERY html page — and `overview.
  char_count` / `char_count_formatted`, computed from that string, were always
  0. Rendering now lives on `OrganizedData.to_text()`, reusing the `to_content()`
  each content type already implements, so `text_data` is exactly the
  `markdown_renderable` rule's text (the `data` form was the only close live
  shape and it leaks `{"type": ...}` labels — "header", "unassociated" — into
  the prose). The orchestrator's `or extracted.get("markdown_renderable")`
  fallback, which papered over the symptom for `ScrapeResult.text_data` while
  leaving `char_count` at 0, is gone. `_build_text_data` no longer swallows
  exceptions — that `except: return ""` is why this shipped unnoticed. Pinned by
  `tests/test_parse_content_text.py` (pipeline level) and
  `test_desktop_consumer_ports.py::test_html_result_reports_its_text` (result
  level).
- 2026-08-09 — **matrx-local's forked scraper engine deleted; it consumes this
  package now.** Six behaviours the fork had and this package did not were
  ported in first (`tests/test_desktop_consumer_ports.py` pins all six):
  injectable Brave keys + `configure_client`, AI-plan fallback to the base key,
  `extract_urls_from_search_results`, `<meta>` fallbacks for `main_image`
  (og:image / image / thumbnail / msapplication-TileImage — extruct is an
  optional extra, so the opengraph branch is empty without it), bmp/tiff/tif +
  xz link classification, and `stop()` no longer reporting a
  driver-already-closed teardown as an error. `ScrapeOptions` +
  `apply_field_flags` moved to `scrape_options.py` so a no-DB consumer can
  import the shape.
- 2026-08-09 — Doc consolidation: eight overlapping top-level docs collapsed into
  this file, `CLAUDE.md`, `README.md`, and `web_crawl/FEATURE.md`. The
  `scraper.crawl_*` schema reference was deleted with the crawler it described.
- 2026-07 — Egress guard (`install_egress_guard`) closed the
  public-page-fetches-internal SSRF hole on the two JS-executing surfaces.
- 2026-07-20 — GSC credentials moved to the canonical secrets vault; the bespoke
  AES-256-GCM path and hand-authored user models were deleted.
- 2026-07-11 — Every DB access in the package converted to matrx-orm; the
  `pool` constructor args survive only as ignored back-compat.
