# CLAUDE.md — developing matrx-scraper

> **🚨 THE PRIME RULE — SIMPLICITY IS SURVIVAL.** Never add a security layer,
> wrapper, network hop, or abstraction on your own authority. Simple version
> FIRST. Full rule: the workspace root [`../../../CLAUDE.md`](../../../CLAUDE.md).
>
> **Build the platform, not the artifact.** Code that only serves one caller and
> could not be reused is forbidden. Doctrine: [`/PRINCIPLES.md`](../../PRINCIPLES.md).

**Package:** `matrx-scraper` (PyPI), Python 3.12+. **Tier 2** in the package
graph: hard deps on `matrx-utils` + `matrx-files`; `matrx-connect`, `matrx-orm`,
`matrx-runtime`, `matrx-graph` behind extras. Nothing in the Matrx family depends
*on* it except hosts.

It is the canonical web scraping / HTML parsing / site crawling / browser
automation / search surface for the whole family. **Do not add scraping logic
anywhere else.**

---

## The two contracts — read the right one

| Doc | Covers |
|---|---|
| [`matrx_scraper/FEATURE.md`](matrx_scraper/FEATURE.md) | The engine: fetch, the 8-stage parser, SSRF gates, proxy rules, the two browser runtimes, the AI-tool/MCP surface, injection seams, HTTP surface + auth, `scraper.*` engine tables, deployment. |
| [`matrx_scraper/web_crawl/FEATURE.md`](matrx_scraper/web_crawl/FEATURE.md) | The canonical site crawler on Supabase `web.*` — commands, run lease, URL identity, persistence, reconciliation. |
| [`README.md`](README.md) | The standalone/PyPI consumer view. Install, quickstart, running the microservice. |

Touch code → update its `FEATURE.md` in the same change. That is the whole doc
system; do not add a fifth top-level doc.

---

## Boundary rules — these are what make it a package

- ✅ `from matrx_utils import …` / `from matrx_files import …` — hard deps.
- ✅ `from matrx_connect import …` — **only** inside conditionally-imported paths
  guarded by `has_ext("connect")` or a lazy import. Never at module top level of
  a core scraping file.
- ✅ `from matrx_orm import …` — **the one and only DB access path.**
- ❌ **No hand-written SQL and no raw driver access, ever.** Missing ORM
  capability → build the primitive in matrx-orm, then convert the call site.
  Skill: [`eliminate-raw-sql`](../../.claude/skills/eliminate-raw-sql/SKILL.md).
- ❌ No `from matrx_ai import …`, no top-level `from matrx_graph import …` — if a
  scraping step should run a graph, the host composes them.
- ❌ No `from aidream import …`, and nothing from repo-root modules.
- ❌ **No repo-layout and no host-env assumptions.** Never walk up from
  `__file__` to reach a project file (`parents[3]` used to mean "the aidream
  repo root" and read aidream's `.env` — dead in every pip install); resolve
  project paths from the working directory. Never read a host-specific env var
  to reach a host capability — take an injected object via `configure_ext`.
  Both are pinned structurally by `tests/test_host_independence.py`.

Mechanically enforced by [`scripts/check_package_boundaries.py`](../../scripts/check_package_boundaries.py)
(CI job `package-boundaries`). Before adding any import, ask: *would this resolve
in a fresh project with no `aidream/`?*

**Acceptance test for new capability:** pure logic, an HTTP client, an algorithm,
a Pydantic model, a Playwright helper, an env-resolving CLI → in here. A host
service, a dashboard view, a cron worker → in the host, built on the public
surface. If a feature needs both, split it.

---

## Extras — what each one actually gates

| Extra | Without it |
|---|---|
| `browser` (playwright, curl-cffi) | No rendering, no screenshots, no `ai_browser`. |
| `db` (matrx-orm) | Engine still scrapes/crawls; it persists nothing. |
| `durable` (matrx-runtime) | Crawls run the **in-memory** frontier — no resume. `server` includes it deliberately: omitting it once left matrx-runtime absent from the EC2 image and every crawl died with its process. |
| `metadata` (extruct) | Structured-metadata fields stay empty. |
| `dedup` (datasketch, simhash) | Hashing fields stay empty. |
| `pdf` / `ocr` | Those extractors raise. |
| `connect` | Events emit dict-shaped instead of typed. |
| `graph` | `matrx_scraper.graph_nodes` unavailable. |
| `server` | No FastAPI app / CLI. |

🚨 **An extra must be unreachable from a core import.** `cache.py` and
`domain_config.py` imported `matrx_scraper.db.models_scraper` at module scope
purely for their Postgres tier — so `from matrx_scraper.cache import MemoryCache`
(the FIRST line of matrx-local's no-DB desktop lane) loaded matrx-orm and died
with `ImportError: cannot import name 'PLATFORM_DB_ENV_PREFIX' from 'matrx_orm'`.
Resolve an extra's symbols **inside the class that needs them** (`TwoTierCache`
/ `PostgresDomainConfigStore` do it in `__init__`, so a missing or mismatched
dep fails loud at the point of the wrong choice). And **an extra's floor is
load-bearing** — `[db]` is `matrx-orm>=3.1.49` because that is the first
release exporting the ONE-database resolver `db/__init__.py` imports; a lower
floor resolves a pair that raises on import. Both pinned by
`tests/test_host_independence.py`.

---

## Host wiring

`configure_ext(**kwargs)` is an object registry, not a type registry — the live
keys are `cache`, `domain_config`, `browser_pool`, `file_manager`,
`work_queue_factory`, `wrapped_brave_search`, and `google_credential_resolver`.
(The four `crawl_*` handler keys were deleted on 2026-08-10 with the legacy
crawl world — crawling is reached through `api/crawl_router.py`, not an ext.)
**Grep `get_ext(` before inventing a key**; anything else is silently ignored.
`configure_db("<host db name>")` binds the package's model sets to the host's
existing matrx-orm pool. aidream does all of it in one function,
`_configure_matrx_scraper()` in [`aidream/package_integration.py`](../../aidream/package_integration.py).

Crossing the boundary: implement the capability inside this package → if it needs
a new injection hook, add it here with a standalone fallback → wire the host's
concrete implementation in `package_integration.py`.

---

## Testing

```bash
uv run pytest packages/matrx-scraper/tests
```

Must pass with only the sibling packages installed. Browser and Postgres-backed
tests live behind extras and skip when those aren't present.

Two tests are **proofs, not formalities** — extend them rather than working
around them: `tests/test_crawl_chaos_resume.py` (kills a live crawler mid-run and
drains the same batch from a second process) and
`tests/test_browser_session_ssrf.py` (structurally pins both egress-guard install
sites).

---

## What's ahead

Open, verified work — every item is filed in [`FOUND_DEFECTS.md`](../../FOUND_DEFECTS.md);
this is the index, not the record:

- ~~Dead `scraper.crawl_*` code~~ — **DONE 2026-08-09, closed out 2026-08-10**
  (growth-loop `G-CRAWL-DUAL`). The second crawler is
  gone: `db/{site,metrics,extractor}_queries.py`, `persistence/`, `scheduling.py`,
  the 17 `Crawl*`/`Sites`/`SitePages`/`SiteRunDiffs`/`PsiMetrics`/`GscMetrics`/
  `Integrations` models, and their host consumer `aidream/services/scraper/` +
  `/scraper/admin/*` + the dashboard's `features/scraper/`. Tables moved to
  `graveyard`, never dropped. 🚨 **The 7 `scrape_*` engine tables survive** and
  must never be swept up with them — both directions are pinned by
  `tests/test_standalone_db_bootstrap.py`. The retirement is RECORDED in
  `public._schema_migrations` (`0327_one_crawl_world_record_and_guard.sql`) and
  in `platform.deprecated_relations` (all 17 old→new refs), and the four dead
  `crawl_*` AI-tool descriptors were deleted from `ALL_TOOLS` — **there is no
  `crawl_*` tool group and no `crawl_*` `_ext` key.** Crawling is
  `web_crawl/` over `web.*`, reached via `api/crawl_router.py`.
- **No automated coverage** for the MCP server or non-reader `ai_browser`
  actions beyond the SSRF pins. Reader truncation is pinned by
  `tests/test_browser_reader_truncation.py`.
