Metadata-Version: 2.5
Name: crawlutron
Version: 0.4.0rc1
Summary: Policy-driven, self-hosted web crawler and content-extraction service built around FastAPI
Project-URL: Homepage, https://github.com/adambarber/crawlutron
Project-URL: Issues, https://github.com/adambarber/crawlutron/issues
Project-URL: Changelog, https://github.com/adambarber/crawlutron/blob/main/CHANGELOG.md
Project-URL: Security, https://github.com/adambarber/crawlutron/blob/main/SECURITY.md
Author: crawlutron maintainers
License-Expression: MIT
License-File: LICENSE
Keywords: browser,camoufox,crawler,extraction,fastapi,scraper,ssrf-protection
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Text Processing :: Markup :: HTML
Requires-Python: >=3.14
Requires-Dist: crawlutron-core>=0.4.0.dev0
Requires-Dist: crawlutron-mcp>=0.4.0.dev0
Requires-Dist: crawlutron-rest>=0.4.0.dev0
Requires-Dist: crawlutron-safe-url[psl]>=0.4.0.dev0
Requires-Dist: crawlutron-sdk>=0.4.0.dev0
Provides-Extra: dev
Requires-Dist: basedpyright==1.40.2; extra == 'dev'
Requires-Dist: brotli-stubs==1.2.1; extra == 'dev'
Requires-Dist: coverage==7.17.0; extra == 'dev'
Requires-Dist: jscpd==5.3.2; extra == 'dev'
Requires-Dist: mypy==2.4.1; extra == 'dev'
Requires-Dist: pre-commit==4.7.0; extra == 'dev'
Requires-Dist: pytest-asyncio==1.4.1.post1; extra == 'dev'
Requires-Dist: pytest-cov==7.1.0; extra == 'dev'
Requires-Dist: pytest-randomly==5.0.0; extra == 'dev'
Requires-Dist: pytest-timeout==2.4.0; extra == 'dev'
Requires-Dist: pytest-xdist==3.8.0; extra == 'dev'
Requires-Dist: pytest==9.2.0; extra == 'dev'
Requires-Dist: pyyaml==6.0.3; extra == 'dev'
Requires-Dist: ruff==0.17.3; extra == 'dev'
Requires-Dist: types-lxml==2026.2.16; extra == 'dev'
Requires-Dist: types-pyyaml==6.0.12.20261030; extra == 'dev'
Requires-Dist: types-regex==2026.9.29.20261030; extra == 'dev'
Requires-Dist: vulture==2.16; extra == 'dev'
Description-Content-Type: text/markdown

# crawlutron

A policy-driven, self-hosted web crawler and content-extraction service built around
FastAPI. It exposes a single, machine-readable crawl contract that supports:

**Positioning.** crawlutron is a self-hosted, policy-driven crawler for public pages. Its caller security boundary is built-in per-caller authentication (API keys issued and revoked through `/v1/keys`, operator-token gated; keyed jobs and results are readable only under their key, anonymous crawls share a bounded identity). SSRF is architecture, not an add-on: every egress target is classified, fresh-resolved, IP-pinned, and re-validated per hop and per browser subresource — `when the private-target guard is enabled`, warned once per enabled value at first use (`packages/safe-url/src/safe_url/host_policy.py:88-100`). A transport-auth reverse proxy is recommended when exposing beyond localhost (TLS termination, `Server` header stripping, request-id logging), but built-in per-caller auth is the caller security boundary; the proxy is transport hardening, not a substitute for it. The rule framework (`tests/invariants/`) arms and bites every rule, and the pre-commit hook chain gates every commit. See `docs/architecture-ssrf.md`, `docs/architecture-governance.md`, and `SECURITY.md`.

- **HTTP fetch strategies** — transparent declared-crawler HTTP and TLS-fingerprint
  impersonation (curl_cffi).
- **Browser rendering** — headless/headed Firefox via [Camoufox](https://github.com/daijro/camoufox)
  for JavaScript-dependent pages.
- **Document extraction** — PDF/DOCX/PPTX/XLSX/TXT via markitdown, plus main-text
  extraction via trafilatura.
- **Content classification** — 36 crawl classifications (rendered, blocked,
  network_error, robots_disallowed, waf_challenge_detected, etc.) so callers can branch
  on outcome rather than guessing.
- **Discovery endpoints** — capabilities, agent hints, and a `.well-known` service
  descriptor so agents/bots can discover the contract without prior knowledge.
- **Web search** — `POST /v1/search` discovers public URLs from a query through a
  keyed provider (Tavily / Brave / Serper) with a keyless DuckDuckGo fallback.
- **MCP** — `POST /mcp` exposes search/crawl/map/render as tools over the
  Model Context Protocol (2026-07-28, stateless streamable HTTP).
- **SSRF protection** — target URLs are validated to be public HTTP(S) only.
- **Webhooks, batching, recursion, rate limiting, robots.txt respect** — best-effort,
  caller-owned frontier semantics.

## Platforms

Linux and macOS are supported; Windows is not. The service is built on POSIX
primitives the Windows runtime does not provide: `fcntl` advisory file locking
(`packages/core/src/crawlutron/safety/atomic_write.py`, `packages/core/src/crawlutron/orchestration/engine.py`,
`packages/core/src/crawlutron/transport/content_cache.py`), `resource`/sysconf-based memory
sizing (`packages/core/src/crawlutron/runtime/pressure_memory.py`), and cgroup filesystem
readings for the resource gauges (`packages/core/src/crawlutron/runtime/cgroup_sense.py`).
Deploy it as a Linux container or run it directly on Linux/macOS.

## Security residuals and environment escape hatches

Three residual weaknesses are known and accepted, with the assumption that
guards them stated:

- **S1 — HTTPS DNS rebind window.** HTTP targets are IP-pinned end to end.
  HTTPS browser navigation does two fresh DNS resolves but lets the browser
  connect by hostname, so a hostile DNS server can rebind between check and
  connect.
- **S4 — credential origins ride the egress guard.** Credential origins pass
  the same SSRF egress check as crawl targets: a private/loopback credential
  origin requires `CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1`.
- **S5 — caller key secrets are verified with salted PBKDF2-SHA256.** Issued
  secrets are `ck_<key_id>.<body>` where the body is `secrets.token_urlsafe(32)`
  (256-bit entropy). At rest the key row stores a salted PBKDF2-SHA256 digest
  (210,000 iterations, per-key 16-byte salt,
  `packages/core/src/crawlutron/storage/caller_key_store.py`); verification is one primary-key
  lookup plus one PBKDF2 run, with a miss burning the same derivation against
  a fixed dummy digest so timing is uniform. Residual: entropy of the secret,
  not the KDF, is the real guard; do not reuse this pattern for human-chosen
  secrets.

### Dangerous environment escape hatches

Every `CRAWLUTRON_*` variable that weakens a safety default (derived from the
live settings/env code — `packages/safe-url/src/safe_url/host_policy.py`,
`packages/core/src/crawlutron/service/webhooks.py`, `packages/core/src/crawlutron/config/settings.py`,
`packages/core/src/crawlutron/transport/driver_config.py`):

| Variable | Default | Weakened when | Effect |
| -------- | ------- | ------------- | ------ |
| `CRAWLUTRON_ALLOW_PRIVATE_TARGETS` | unset | `1` | Disables the SSRF private-target guard: caller URLs may reach private, loopback, and link-local addresses including cloud metadata endpoints. Logs a `SSRF private-target guard DISABLED` WARNING. |

### Ordinary operational knobs

Two variables are sometimes mistaken for safety hatches; they are not. They
change operational behavior only and are listed in the Configuration table
below:

- `CRAWLUTRON_RESPECT_ROBOTS` (default `true`) — turns robots.txt
  enforcement off. Robots compliance is a crawler courtesy/policy choice, not
  a security boundary of this service.
- `CRAWLUTRON_HEADLESS` (default `true`) — runs the browser headed instead
  of headless. Changes the automation surface, weakens no guard.

See `SECURITY.md` for the threat model and reporting process, and
`docs/reviews/ADVERSARIAL_REVIEW_BROWSER_CRAWLER.md` for the dated adversarial review of
the crawler.

The service is intended to be deployed as a Docker container. A standalone
`docker-compose.yml` lives at the repository root for local development;
production deployments typically front it with a transport-auth reverse proxy (TLS termination, `Server` header stripping, request-id logging); built-in per-caller auth (`/v1/keys`, `X-API-Key`/`Authorization: Bearer`) is the caller security boundary, and the proxy is transport hardening, not a substitute for it.

## Security Posture

crawlutron is a **localhost-only** service. It binds to `127.0.0.1` by
default. Callers authenticate with an API key presented as `X-API-Key` or
`Authorization: Bearer` — jobs and results created under a key are readable
only under it, and key issuance/revocation is operator-gated
(`CRAWLUTRON_KEY_ADMIN_TOKEN` via `/v1/keys`). Keyless requests crawl as a
shared anonymous identity with bounded default quotas. Key verification is
rate-limited (#653): five consecutive failed verifications from the same
presented credential trip a 60-second cooldown that refuses further attempts
with `429 key_verify_throttled` before the PBKDF2 derivation runs again, so
the expensive pre-auth hash cannot be invoked for free in a loop. Buckets
key on a hash of the full presented secret (#675), so garbage under another
caller's key id cannot lock the rightful owner out. Anonymous surface
metering keys on the REMOTE ADDRESS the ASGI transport reports (`client.host`):
behind a reverse proxy every request shares the PROXY's address and its one
bucket — when fronting the service with a proxy, run the metering at the
proxy or deploy as a single-tenant front. Expose it
beyond localhost only behind a transport-auth reverse proxy; built-in per-caller
auth (`/v1/keys`) is the caller security boundary, and the proxy is transport
hardening, not a substitute for it.

What the SSRF defenses cover:

- **DNS pinning on every fetch** — every URL is resolved fresh at fetch
  time and the resulting IP is pinned for the connection, so a DNS
  rebind cannot redirect the request mid-flight.
- **Manual redirect handling** — redirects are followed one hop at a
  time; each hop re-runs the IP-pinning and private-IP check.
- **Browser-side SSRF** — the Camoufox/Playwright navigation is pinned
  to a freshly-validated IP via `Host:` header rewriting; post-redirect
  URLs are re-validated before being used as document fetch targets.
- **Body-size guard** — request bodies are capped at 16 MiB before
  pydantic validation runs.
- **HTML/DOM size guards** — classification and metadata extraction
  truncate inputs at 4 MiB before regex / lxml processing.
- **Per-tenant cookie namespace** — multi-tenant callers should set
  `request_profile.cookie_namespace` so cookies from different callers
  are isolated in the shared cookie jar.


## Adversarial Test Coverage

The test suite was grown alongside the adversarial reviews; the following
suites cover the specific attack classes the security posture is designed
to defeat:

- **Truncation and drain** (`tests/test_body_cap_state_machine.py`) — chunked
  handling, `Connection: close` on a truncated request, per-chunk trailers and
  headers, body-size drain after truncation, the rule that no further bytes are
  admitted once a chunk trips the cap, and the 413 the shipped app returns on a
  body over the cap.
- **File-descriptor ownership** (`tests/invariants/test_ownership.py`) — scratch
  files are owned by the work writing them; the SQLite spill store replaced the
  file spill (`tests/storage/test_crawl_spill_store.py`).
- **Cache locking** (`tests/transport/test_dns_cache.py`,
  `tests/transport/test_robots_policy_consolidation.py`) — concurrent access to
  the DNS and robots caches, including that robots reads go through the one
  chokepoint.
- **Cookie namespacing** (`tests/transport/test_cookie_namespace.py`) — per-tenant
  cookies cannot cross a tenant boundary.
- **Env parsing** (`tests/test_env_parsing.py`) — `parse_bool_env` edge cases,
  including that a typo cannot silently open a policy.
- **Webhook TOCTOU** (`tests/api/test_webhook_delivery.py`) — one resolved
  target per delivery, consolidated at the chokepoint.
- **Decompression bomb tests** — a real gzip bomb (`~1 KB` compressed →
  `1 MB` expanded), plus deflate and brotli bomb cases, verify the decompression
  guard rejects over-cap output before OOM.
- **SSRF tests** (`tests/test_ssrf_route_subresources.py`, `tests/transport/test_dns_cache.py`, `tests/transport/test_redirect_target.py`, `tests/transport/test_browser_target_stability.py`) — DNS
  pinning, route-level interception, WebSocket guard, hostname pinning
  even after a redirect chain, IPv6 private range blocking.
- **Classification corroboration tests** — soft phrases (e.g. "blocked")
  require ≥2 independent signals; the test asserts a single soft signal
  is NOT enough to trigger a defensive classification.
- **Resource admission control** — fake cgroup gates verify the
  two-tier weighted admission policy kills stop-the-world pauses under
  saturation.

This is unusually thorough for an open-source project. New attack
classes should land alongside a dedicated adversarial test that fails
without the fix.

**Coverage is measured, not a badge.** No coverage badge or coverage
ratchet is claimed. The pre-push coverage hook is measured-descriptive
only — no threshold, no fail-under, no ratchet until the owner rules one
(`.pre-commit-config.yaml:229-237`). The safety boundary is the invariant
suite and the fail-closed SSRF/auth guards, not a coverage percentage.
See `docs/architecture-governance.md`.

---

## Build & Run

The service ships with a standalone `docker-compose.yml` and a multi-stage
`Dockerfile` so it can be built and run from a fresh clone without any parent
monorepo or shared-helper tree.

```bash
# Build the image (multi-stage; the Camoufox browser binary is fetched once in an
# isolated stage so editing requirements.txt does not re-download ~1.3 GB).
docker compose build crawlutron

# Run it (recreates only this service).
docker compose up -d crawlutron
```

The container listens on `127.0.0.1:8765`. The served API docs (Swagger UI)
are available at `/docs` and the machine-readable schema at `/openapi.json`.
For local-hostname routing (e.g. `http://crawlutron.local`), front the
container with Caddy or another reverse proxy that owns the `.local` zone.

> **Note on the build:** The Camoufox browser binary is downloaded in a dedicated
> `camoufox-binary` stage that depends *only* on the `camoufox` requirement line,
> then copied into the runtime image. This isolation means an unrelated change to
> `requirements.txt` (for example adding `h2`) no longer invalidates the
> multi-minute binary download.

---

## SQLite schema migrations

Migrations live in `packages/core/src/crawlutron/storage/migrations/`. Every store schema — jobs, results, metrics, coordination — lives in the single `cache/crawler.db`; startup applies that database's pending migrations under yoyo's lock, and an exact pre-ledger table is stamped instead of recreated. The CLI names the schema explicitly because they share one file.

```bash
# Apply pending migrations (normally automatic when the store opens).
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db results

# Add a migration: add a versioned schema to manifest.py, point CURRENT_SCHEMAS at it,
# then add a yoyo step with apply and rollback SQL under the migration directory.
$EDITOR packages/core/src/crawlutron/storage/migrations/manifest.py
$EDITOR packages/core/src/crawlutron/storage/migrations/0001_crawl_jobs_initial.py

# Stamp an existing database after checking it against the declared schema.
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db results

# Roll back one schema's applied migrations in reverse order; do not open the
# store afterward. Rolling one schema back leaves the others applied.
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db results
```

---

## Endpoints

| Method | Path | Purpose |
| ------ | ---- | ------- |
| `GET`  | `/health` | Liveness probe (always `200` if the process is up). |
| `GET`  | `/ready` | Readiness probe; returns `503` when the self-managed resource guard is saturated (memory or CPU at the gate target). |
| `POST` | `/v1/crawl` | Submit one URL (or a batch of URLs) for crawling. |
| `POST` | `/v1/render` | Render a URL with the browser engine and return structured output. |
| `POST` | `/v1/search` | Discover public URLs from a query via the configured search provider. |
| `POST` | `/v1/map` | Crawl a site's link graph and return the discovered URL set (no content extraction). |
| `GET` | `/v1/crawl/{request_id}` | Fetch the result of an async crawl submission. |
| `GET` | `/v1/crawl/{request_id}/trace` | Per-URL strategy attempt trace (statuses, classifications, timings; no page bodies). |
| `POST`/`GET`/`DELETE` | `/v1/watches` | Revalidation watches: register, list, get, delete; sweeps recheck due watches and POST webhooks on change. |
| `GET`/`POST` | `/v1/jobs` | Whole-site resumable jobs: create (with frontier), list, inspect, pause/resume/cancel. |
| `POST` | `/mcp` | MCP 2026-07-28 endpoint exposing search/crawl/map/render as tools. |
| `GET`  | `/v1/capabilities` | Declares supported strategies, classifications, extraction methods, profiles. |
| `GET`  | `/v1/agent-hints` | Machine-readable guidance for agents/bots (request shapes, caveats). |
| `GET`  | `/.well-known/crawler-service` | Service discovery descriptor (links to OpenAPI, agent hints, capabilities). |
| `POST`/`GET`/`DELETE` | `/v1/keys` | Caller API keys: issue, list, revoke (operator token required). Callers present the key as `X-API-Key`. |
| `GET` | `/metrics` | Prometheus-format metrics. Per-caller detail families require the operator token (`X-API-Key: CRAWLUTRON_KEY_ADMIN_TOKEN`); a presented but wrong token records an admin-bucket failure (paced 5/300s), so scrape-guessing griefs the operator doors' cooldown too. |
| `GET` | `/v1/policy-suggestions` | Candidate host policy rules rolled up from blocked-attempt evidence (proposal only; never applied). |

---

## Web search

`POST /v1/search` discovers public URLs from a query and returns SERP-style
results (`{success, query, provider, limit, data: [{url, title, description}]}`).
It does **not** crawl result URLs — pipe them into `POST /v1/crawl`.

The backend is operator-configured, not caller-selected:

- `CRAWLUTRON_SEARCH_PROVIDER` (`auto` | `tavily` | `brave` | `serper` | `duckduckgo`).
- `auto` (default) activates the first configured keyed provider in priority
  order (Tavily to Brave to Serper) and falls back to the keyless DuckDuckGo
  HTML endpoint when no API key is set. The keyless endpoint may be
  captcha-blocked from datacenter IPs, so configure a key for production use.
- A selected provider without its key returns `503 search_provider_unavailable`
  naming the missing `CRAWLUTRON_*_API_KEY` variable.

Result URLs pass the same public-URL validation as crawl targets (loopback and
private targets are dropped). `GET /v1/capabilities` reports the active
provider and the configured provider list.

## MCP

`POST /mcp` exposes `search`, `crawl`, `map`, and `render` as tools over the
**Model Context Protocol (2026-07-28)** — the stateless revision: no
`initialize` handshake, no sessions. Every request carries its protocol
version and client capabilities in `_meta`, and the transport validates the
`MCP-Protocol-Version`, `Mcp-Method`, and `Mcp-Name` headers against the body.

- `server/discover` — supported protocol versions, capabilities, server identity.
- `tools/list` — the fetch, read, and job tools with JSON-Schema inputs.
- `tools/call` — invokes a tool; returns text `content` plus
  `structuredContent`, with `isError` set for execution failures.
- The tools mirror the REST endpoints and delegate to the same handlers, so
  **all four strategies and every extraction method are available through MCP**.
- Webhook/async delivery is not available over MCP.

The endpoint is loopback-only and rejects non-loopback `Origin` headers (DNS
rebinding defense).

## Crawl request shape

`POST /v1/crawl` accepts either a single `url` or a list of `urls`, plus optional top-level
fields: `policy`, `validators` (cached conditional-GET validators), `async_mode`, `webhook_url`,
and `output_format`. The request/browsing profile knobs — `request_profile`, `browser_profile`,
`discovery`, `scope_regex`, `css_selector`, `xpath` — live **inside** `policy`.

```json
{
  "url": "https://example.com/article",
  "policy": {
    "strategies": ["HTTP_DECLARED_CRAWLER", "BROWSER_HEADLESS"],
    "extraction": "full",
    "limits": {
      "timeout_seconds": 30,
      "max_retries": 2,
      "challenge_wait_seconds": 4,
      "max_discovered_links": 10,
      "respect_robots": true
    },
    "request_profile": { "user_agent_family": "generic_crawler" },
    "browser_profile": { "resource_mode": "full" },
    "discovery": { "robots": true, "sitemaps": true, "feeds": true }
  }
}
```

### Strategies (fallback chain)

`strategies` is an **ordered fallback chain**. The engine tries each in turn until
one succeeds, returns a definitive blocked/error classification, or the chain is
exhausted.

| Strategy | Description |
| -------- | ----------- |
| `HTTP_DECLARED_CRAWLER` | Transparent direct HTTP fetch with a declared-crawler identity (lightweight, fast). |
| `HTTP_CFFI_IMPERSONATE` | TLS-fingerprint-impersonated HTTP via `curl_cffi` with browser-shaped headers, for sites that reject non-browser clients. |
| `BROWSER_HEADLESS` | Firefox rendering in headless mode using Camoufox for JavaScript-dependent pages. |
| `BROWSER_HEADED_WARMED` | Headed Firefox with configured warmup URLs for challenge-prone sites. |

### Extraction methods (`policy.extraction`)

- `full` (default) — best-effort main-text extraction, returns both `markdown` and `text`.
- `trafilatura` — main-text only via trafilatura.
- `document` — markitdown-based extraction for PDF/DOCX/PPTX/XLSX/TXT (binary documents).
- `dom` — raw DOM / selected-element extraction (`css_selector` / `xpath`).

---

## Classifications

Every crawl result carries a `classification` from a fixed enum of **36 values**
(defined in `packages/core/src/crawlutron/vocabulary/taxonomy.py`, returned by `GET /v1/capabilities` under
`supported_classifications`). Examples: `rendered`, `not_modified`, `captcha_detected`,
`waf_challenge_detected`, `robots_disallowed`, `navigation_timeout`,
`private_network_target`, `network_error`, `http_error`, `url_budget_exceeded`,
`payload_too_large`. Callers should branch on `classification` (whose
`status: success | blocked | disallowed | retry_exhausted | failed | skipped`
the taxonomy maps onto) rather than on transport status.

---

## Behavior notes

- **SSRF protection** is on by default: only public HTTP(S) targets are allowed.
  Set `CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1` to disable it (local testing only).
  Enabling it logs a WARNING naming the disabled guard, and lets caller-supplied
  URLs reach private, loopback, and link-local addresses including cloud
  metadata endpoints.
- **robots.txt** is respected by default (`respect_robots`). Disallowed targets
  return `robots_disallowed`.
- **Recursion / frontier:** `POST /v1/crawl` recursively expands one in-memory
  frontier for the duration of a request and reports discovered links, sitemaps,
  and feeds; cross-run queue ownership remains with the caller. `POST /v1/jobs`
  instead owns a durable whole-site frontier. Each persisted `frontier` and
  `in_flight` entry is `{"url": "...", "depth": N}`; `url` is the canonical page
  identity and depth 0 is the stored canonical root. Legacy string entries load
  at depth 0 when they are the root and at the effective `max_depth` otherwise,
  preventing untracked additional recursion during upgrade.
- **Whole-site admission:** the engine dispatcher and durable job runner share
  canonical-identity, scope, analytics, document, robots, crawl-depth, and page-cap
  admission. When `discovery.sitemaps` is enabled, root sitemap pages enter the
  durable frontier at depth 0 through that same pipeline. `max_depth=0` is
  single-page. A positive job `max_pages` counts unique canonical page identities
  across the root, pending frontier, in-flight recovery,
  visited pages, and retries. Missing or zero `max_pages` is unbounded for jobs;
  `/v1/crawl` retains its bounded service default. A positive job cap sets
  `cap_truncated=true` in status and `stats.cap_truncated=1` only when it refuses
  or truncates eligible queued or discovered work; normal `BATCH_SIZE` chunking
  does not set it. Already-admitted in-flight work is never dropped: recovery
  preserves at-least-once delivery and records `stats.cap_overcommitted=1`.
- **Crawl duration & async**: a synchronous `POST /v1/crawl` has **no server-side
  time ceiling** — it returns only when the whole bounded recursion finishes, so
  duration scales with `max_depth`, `max_pages`, per-domain politeness delays, and
  the per-URL budget (`limits.total_url_budget_seconds`, default 180s, which fails a
  single tarpit URL, not the crawl). For large or recursive crawls send
  `async_mode=true` (or `webhook_url`) to get an immediate `202` + `status_url`, then
  poll `GET /v1/crawl/{request_id}`; or use `POST /v1/jobs` for durable, resumable,
  pause/cancel-able whole-site jobs. Read timeouts on the sync path must cover the
  chosen policy and batch size.
- **Rate limiting** is enforced via `global_concurrency`, `per_domain_concurrency`,
  and `per_domain_delay_ms`, scoped to a single request: each request runs its
  own scheduler, so concurrent requests each enforce the limits independently
  and a service-wide per-domain bound is not provided (callers pacing many
  concurrent requests own the aggregate). Exceeding `max_batch_size` is rejected.
- **Webhooks / async**: provide `webhook_url` to receive results asynchronously, or
  `async_mode=true` to get a `202 Accepted` with a `status_url` to poll
  `GET /v1/crawl/{request_id}`. Both return immediately instead of blocking on the
  crawl.
- **Conditional GETs**: `HTTP_DECLARED_CRAWLER` sends `If-None-Match` /
  `If-Modified-Since` when prior ETags are known, returning `not_modified` on a `304`.
- **Resource self-tuning** (`#self-tuning`): the engine senses its own cgroup
  memory/CPU ceiling and throttles recursive-batch dispatch to ride at the
  targets instead of OOMing or tripping the kernel CPU throttle. No operator
  tuning is required.
- **HTTP-aware admission (item 7)**: browser crawls are the memory hog (one
  ≈700 MiB (≈600–900 MiB RSS) browser process each), and when browser processes hold memory
  the whole batch used to be throttled — even cheap HTTP work. Today, when the
  upcoming batch is predominantly domains *pinned* to an HTTP strategy (they've
  already proven they fetch without a browser) and CPU has spare headroom, the
  dispatcher uses the **HTTP-aware scale** (`ResourceGate.http_scale_factor`),
  which rides the CPU-implied rate and is *not* dragged down by memory held by
  browser processes of other domains. Browser-bound and cold (unpinned) batches
  keep the conservative combined memory+CPU gate, so the extra concurrency is
  always ramp-bounded (≤ `RECURSIVE_RAMP_MAX × multiplier`) and cannot overshoot
  memory.

---

## Configuration

All settings are read from `CRAWLUTRON_*` environment variables (sensible
defaults shown):

| Variable | Default | Meaning |
| -------- | ------- | ------- |
| `CRAWLUTRON_GLOBAL_CONCURRENCY` | `4` (env-overridable floor) | Concurrency floor for the recursive dispatcher. Real concurrency is self-managed by the resource guard; `/ready` no longer keys off this value. |
| `CRAWLUTRON_PER_DOMAIN_CONCURRENCY` | `2` | Max concurrent tasks per host. |
| `CRAWLUTRON_PER_DOMAIN_DELAY_MS` | `500` | Delay between requests to the same host. |
| `CRAWLUTRON_HTTP_MAX_CONNECTIONS` | _(RAM-derived; floor 100, cap 512)_ | Max concurrent HTTP connections. The pool default is the floor of 100; the weighted admission derives the real HTTP concurrency from the cgroup limit as `(limit − browser-pool reserve − doc-pool reserve) / 48 MiB per HTTP task` — ~34 at 3 GiB with the 2-instance browser pool (~700 MiB each), ~22 with the default 2 doc workers (300 MiB each). Bound to the weighted admission's HTTP concurrency so the pool can't starve or over-provision. |
| `CRAWLUTRON_HTTP_MAX_KEEPALIVE` | _(RAM-derived; floor 20)_ | Max idle keep-alive connections (≈ half of the connection ceiling, bounded to [20, 128]). |
| `CRAWLUTRON_BROWSER_POOL_SIZE` | _(RAM-derived; 2 at 3 GiB)_ | Explicit browser-pool size override. Default is derived from the cgroup limit (reserves ≤50% of RAM for browsers); raising it multiplies RAM use and risks OOM. |
| `CRAWLUTRON_SEARCH_PROVIDER` | `auto` | Search backend: `auto` (first configured keyed provider) or explicit `tavily` / `brave` / `serper` / `duckduckgo`. |
| `CRAWLUTRON_TAVILY_API_KEY` | `""` | Tavily Search API key (activates the `tavily` provider). |
| `CRAWLUTRON_BRAVE_API_KEY` | `""` | Brave Search API key (activates the `brave` provider). |
| `CRAWLUTRON_SERPER_API_KEY` | `""` | Serper.dev API key (activates the `serper` provider). |
| `CRAWLUTRON_SEARCH_LIMIT_DEFAULT` | `5` | Default result count per search. |
| `CRAWLUTRON_SEARCH_LIMIT_MAX` | `20` | Hard cap on results per search. |
| `CRAWLUTRON_SEARCH_TIMEOUT_SECONDS` | `15` | Provider call timeout. |
| `CRAWLUTRON_MAX_BATCH_SIZE` | `50` | Max URLs per batch request. |
| `CRAWLUTRON_WARMUP_URLS` | `""` | Comma-separated URLs fetched once at browser startup to warm engine caches (`browser.py`). Warmup URLs are normalized through the same public-URL policy as crawl targets. Related: `browser_profile.storage_state` paths are honored only under `CRAWLUTRON_CACHE_DIR` (the allowlist root); without a cache dir no storage_state is accepted. |

Two DNS posture values are **not** env-tunable — they are module constants in
`packages/core/src/crawlutron/transport/pinned_dns.py`: `DNS_WORKERS = 32` (the dedicated DNS
resolver thread pool) and `DNS_WAIT_SECONDS = 5.0` (the ceiling on a resolver
wait; a request's own timeout tightens it further). They implement the
pinned-DNS saturation law: when every worker is inside a live resolve, a new
resolve refuses fast (`ResolverBudgetExceeded`, classifiable as `dns_error`)
instead of queueing behind an uncancellable `getaddrinfo`.
| `CRAWLUTRON_DEFAULT_TIMEOUT_SECONDS` | `30` | Default per-request timeout. |
| `CRAWLUTRON_DEFAULT_MAX_RETRIES` | `5` | Default retry count. |
| `CRAWLUTRON_RESPECT_ROBOTS` | `true` | Respect robots.txt by default. |
| `CRAWLUTRON_ALLOWED_HOSTS` | `""` | Optional host allowlist (comma-separated suffix match; case, IDNA, and port normalized). |
| `CRAWLUTRON_DENIED_HOSTS` | `""` | Optional host denylist (comma-separated suffix match; deny wins over allow). |
| `CRAWLUTRON_LOG_LEVEL` | `info` | Log verbosity. |
| `CRAWLUTRON_LOG_FILE` | `./logs/crawlutron.jsonl` (repo-relative; the container image overrides it to `/app/logs/crawlutron.jsonl`) | Structured JSONL log path. |
| `CRAWLUTRON_LOG_MAX_BYTES` | `10485760` | Log rotation size. |
| `CRAWLUTRON_LOG_BACKUP_COUNT` | `3` | Number of rotated log files to keep. |

Host policy is applied at the shared public-URL chokepoint before the SSRF/private-target check. Entries are normalized as case-insensitive IDNA hostnames (`_policy_host_entry` in `packages/safe-url/src/safe_url/host_policy.py` parses each value via `urlparse(...).hostname`); a port in a config value is stripped during normalization and never matched — an entry is a hostname, not a `host:port` pair. A denied host wins over an allowed host, while an empty allowlist permits every otherwise-valid public host.

> The container's production command also reads `CRAWLUTRON_WORKERS` (uvicorn worker count). **Keep this at `1`.** The browser pool is a single global pool sized to the container RAM budget; running more than one uvicorn worker multiplies the browser-pool count across workers (e.g. 4 workers × pool_size 2 = 8 Camoufox processes ≈ 5+ GiB) and blows the 3 GiB memory limit. Do not raise it.

> **Secrets & logs:** Logs and prompt JSONL can contain full page content and
> request metadata. Do not commit generated logs under `logs/`.

---

## API documentation (served)

- Swagger UI: `/docs`
- OpenAPI JSON: `/openapi.json`
- Agent hints: `/v1/agent-hints`
- Capabilities: `/v1/capabilities`
- Service descriptor: `/.well-known/crawler-service`

The `/v1/render` request body is a typed `RenderRequest` (`url`/`urls` plus an
`options` object with `timeout_seconds`, `max_retries`,
`respect_robots`, `include_html`, `include_text`). Malformed bodies are rejected
with `422`.

---

## Project layout

```
.  # repository root (clone of crawlutron)
├── Dockerfile            # Multi-stage build; isolated camoufox-binary stage (context: .)
├── docker-compose.yml    # Standalone local stack (loopback-only)
├── pyproject.toml        # Package metadata + deps (single source; uv.lock pins every pin)
├── uv.lock               # Frozen transitive dependency graph + hashes
├── Makefile              # Gates + convenience targets mirroring scripts/run-tests.sh
├── packages/             # uv workspace member packages, src layout under packages/*/src/
│   ├── core/
│   │   └── src/crawlutron/   # core engine: admission kernel, models, transport, storage, service, …
│   │       ├── api/          # door-plane handler bodies (crawl_endpoint, render_endpoint, map_endpoint)
│   │       ├── cli/          # crawler CLI entry points
│   │       ├── config/       # env.py + settings.py (CRAWLUTRON_* environment surface)
│   │       ├── discovery/    # /v1/capabilities, /v1/agent-hints, /.well-known
│   │       ├── extraction/   # documents.py (trafilatura + selector extraction)
│   │       ├── health/       # /health, /ready (READINESS_STATE singleton)
│   │       ├── mcp/          # MCP protocol support shared with crawlutron_mcp
│   │       ├── models/       # Pydantic/dataclass wire + domain models
│   │       ├── orchestration/# engine.py — orchestration + fallback-chain scheduling
│   │       ├── runtime/      # metrics, memory pressure, worker dispatch, resource lifecycle
│   │       ├── safety/       # validation, bounded I/O, safe regex, SSRF helpers
│   │       ├── search/       # /v1/search provider abstraction (Tavily/Brave/Serper/DuckDuckGo)
│   │       ├── service/      # store_bootstrap, watch_cores, job lifecycle, …
│   │       ├── storage/      # job_store, result_store, watch_store, sqlite_store, migrations/
│   │       ├── transport/    # http.py, browser.py, document.py, content_cache.py, scheduler.py, robots.py
│   │       └── vocabulary/   # taxonomy.py (36-value classification enum), strategy names
│   ├── rest/
│   │   └── src/crawlutron_rest/   # REST facade: FastAPI app + routers + CLI
│   │       ├── main.py        # FastAPI factory (lifespan, app.state, router wiring)
│   │       ├── api/           # one router per concern (crawl, jobs, watches, keys, metrics, policy-suggestions, …)
│   │       └── cli/           # CLI entry (__main__, output, docker, client)
│   ├── mcp/
│   │   └── src/crawlutron_mcp/  # MCP 2026-07-28 endpoint (protocol, tools, server)
│   └── sdk/
│       └── src/crawlutron_sdk/  # client SDK mirroring the REST wire surface
├── tests/                # Behavior tests (no mocks; real HTTP + real Camoufox)
├── docs/                 # Reviews, testing guide, agent contract (docs/references/)
├── scripts/run-tests.sh  # Full battery entrypoint (mirrored by make targets)
└── .github/workflows/ci.yml  # ruff + pytest (fast)
```

---

## Testing

Behavior tests run against the real stack (a local `http.server`, a real Camoufox
browser, and real fixtures — no mocks). Use the dedicated venv:

```bash
pip install -e ".[dev]"
make test          # fast hermetic suite (default)
# or
python3 -m pytest -q
```

The manual test plan lives at `docs/references/crawlutron-manual-test-plan.md`
and the agent integration contract at `docs/references/crawlutron-agent-contract.md`.

## License

MIT — `pyproject.toml` (`license = "MIT"`) and the `LICENSE` file agree; the
`NOTICE` file carries the third-party attributions.
