Components at a glance.

Eleven moving parts make up the system. Most run on the server; only the browser-capture piece runs on the user's machine.

Pipeline components

Listed in the order they fire during a scan. Click any card to jump to its detail page.

Cost breakdown per scan

Architectural map.

Three actors: the customer's computer, the browser-recon server (hosted on Render), and AWS for storage and secrets. The server does most of the work.

Boundaries that matter

• Client ↔ server: exactly two endpoints — POST /scans/<id>/capture and GET /scans/<id>/events. All traffic is HTTPS bearer-authed.
• Server ↔ Postgres: single managed instance on Render. All persistent state lives here: scans, reports, llm_calls, llm_evals, validation_runs, scan_events, users, audit_log.
• Server ↔ S3: raw captures (KMS-encrypted) and prompt/response dumps for every LLM call (for replay + audit).
• Server ↔ Secrets Manager: proxy URLs only. Never logged, never persisted to Postgres.
• Server ↔ external LLMs: Anthropic, OpenAI, xAI. Each has its own API key in env (server-side only).
• Server ↔ proxy vendor: all validation HTTP traffic exits through the configured proxy URLs.

Deployment model

Single Render Web Service running uvicorn browser_recon_server.main:app. Auto-scales based on request rate. Postgres is Render-managed (encrypted disk by default). S3 + Secrets Manager are AWS-side. No worker queue today — long-running pipeline tasks use FastAPI's BackgroundTasks within the same process. If we need to fan out under load, the natural next step is a Celery/RQ worker reading from a Redis queue.

Lifecycle of a scan.

Fourteen ordered steps, from recon scan in the terminal to a report URL in the browser. Steps 1–5 happen on the user's machine; 6–14 happen on the server.

Status transitions

The scans.status column tracks the scan's lifecycle:

Database schema.

Eight tables in Postgres hold every piece of persistent state — users, scans, AI calls, validation results, audit logs.

Core tables

JSONB columns on scans

Most of the scan's processed state lives in JSONB columns on the scans row, rather than in normalised tables. This keeps the schema flexible as the pipeline evolves.

Querying conventions

-- Find recent failed scans:
SELECT id, target_url, status, error_message, created_at
FROM scans
WHERE status = 'errored'
ORDER BY created_at DESC
LIMIT 20;

-- Find the latest validation_runs row per endpoint (T51.7 append-only):
SELECT DISTINCT ON (scan_id, endpoint_id) *
FROM validation_runs
WHERE scan_id = $1 AND status = 'complete'
ORDER BY scan_id, endpoint_id, created_at DESC;

-- Find a scan's event timeline (T53.1):
SELECT step, status, message, created_at
FROM scan_events
WHERE scan_id = $1
ORDER BY created_at ASC;

-- LLM cost rollup per scan:
SELECT scan_id, SUM(cost_usd) AS total_usd
FROM llm_calls
WHERE created_at > now() - interval '7 days'
GROUP BY scan_id
ORDER BY total_usd DESC;

Migrations

Run via Alembic. Every schema change ships a migration in alembic/versions/. To apply on a new environment: rye run alembic upgrade head. To downgrade one: rye run alembic downgrade -1.

Recent migrations:

• f7a3c2d4e5b6_t51_7_add_validation_runs.py — append-only validation_runs table
• c1d2e3f4a5b6_add_llm_evals.py — T48 eval persistence
• (T53.1) add_scan_events.py — pipeline event log

Browser capture.

Launches Chrome on the customer's machine and records every network call via the Chrome DevTools Protocol. The only piece that must run client-side.

What it does

Launches Chrome with --remote-debugging-port=9222 on a fresh, isolated user-data-dir (no extensions, no profile pollution). Connects to the CDP WebSocket, enables Network/Page/Runtime domains, and starts logging every Network.requestWillBeSent + Network.responseReceived + Network.dataReceived event into an in-memory blob. Captures cookies via Network.getAllCookies on capture end.

Real-time anti-bot detection

While capturing, the CLI runs a light detection pass on incoming responses and prints live alerts:

! ANTI-BOT: Imperva / Incapsula detected -- Header 'x-cdn' present
! ANTI-BOT: PerimeterX detected -- URL contains 'px-cloud.net'
! ANTI-BOT: Akamai Bot Manager detected -- URL contains '/akam/'

This is informational only — the authoritative detection step runs server-side after upload. The CLI version gives the user instant feedback so they understand what they're scraping.

Where it lives

Package

browser_recon/capture/

Entry

browser_recon/capture/chrome_launcher.py — finds Chrome, launches it

CDP loop

browser_recon/capture/monitor.py — WebSocket event handler

Anti-bot live

browser_recon/capture/anti_bot_live.py

Output

In-memory dict serialised to JSON on capture end

Inputs / outputs

• Input: target URL, intent text, optional starter template.
• Output: a capture blob — list of requests/responses with full headers, response bodies, cookies, timestamps, plus user-action breadcrumbs (clicks, inputs, form submits).

Database impact

The CLI does not write to the database directly. The capture blob is uploaded to the server in step 6, which stores it in S3 (KMS-encrypted) and creates a row in scans with the S3 key.

How to test it

# Run the CLI against a public test site:
recon scan https://books.toscrape.com

# Capture lasts as long as you browse. Ctrl+C to stop.
# Output dir gets a recon_capture.json with the full blob.

# Unit tests for the capture layer:
rye run pytest tests/unit/test_chrome_launcher.py
rye run pytest tests/unit/test_capture_monitor.py

Find output in the DB after upload

SELECT id, status, target_url, capture_metadata
FROM scans
WHERE id = '<scan_id>';

-- The raw capture is at the S3 key returned in the response:
SELECT s3_capture_path FROM scans WHERE id = '<scan_id>';

Known gotchas

• Chrome must be installed on the customer's machine. The launcher searches the standard install paths per OS; if missing, the CLI fails fast with an install hint.
• Cookies captured during the browse session are bound to the customer's residential IP. When validation later fires through a different proxy IP, those cookies may be invalid (PerimeterX/Akamai bind _pxhd/_abck to IP).
• Capture cookies aren't scrubbed on the CLI side post-T53 — they travel to the server unscrubbed because validation needs them. They're scrubbed server-side after step 10.

Anti-bot detection.

Fingerprints which anti-bot vendors are protecting the target site. Pure pattern-matching, no LLM. Runs in tens of milliseconds.

What it does

Walks the captured requests and responses, scoring them against a rules table:

• Cookies — _abck → Akamai Bot Manager · _pxhd/_pxvid → PerimeterX · __cf_bm/cf_clearance → Cloudflare · datadome → DataDome · incap_ses → Imperva · visid_incap → Imperva.
• Response headers — server: cloudflare, x-cdn: incapsula, x-akamai-transformed, cf-ray, x-datadome.
• URL patterns — paths under /akam/, hosts ending in .px-cloud.net, .datadome.co, recaptcha/api2/.
• JavaScript fingerprints — script tags loaded from known anti-bot CDNs.

Each match contributes evidence with a confidence weight. Findings are aggregated per vendor and emitted with a severity tier (high/medium/low) reflecting how restrictive that vendor typically is in production scraping.

Where it lives

Package

browser_recon_server/detection_server/ (post-T53) or browser_recon/detection/ (pre-T53)

Entry

detection.runner.run_detection(capture) -> DetectionResult

Rules

detection/rules/anti_bot.py, detection/rules/auth.py, detection/rules/cors.py

Aggregation

detection/aggregator.py — combines per-rule matches into per-vendor findings

Output shape

{
  "findings": [
    {
      "kind": "anti_bot",
      "vendor": "Akamai Bot Manager",
      "severity_tier": "high",
      "severity_label": "high",
      "confidence": 0.85,
      "evidence": [
        {"signal": "cookie", "value": "_abck", "confidence_weight": 0.3, ...},
        {"signal": "url_pattern", "value": "path: /akam/", ...}
      ],
      "per_endpoint": {"https://...": "presence detected", ...}
    }
  ]
}

Database impact

• Writes to scans.findings (JSONB column)
• Emits a scan_events row at start + complete (T53.1)

How to test it

rye run pytest tests/unit/test_detection_anti_bot.py
rye run pytest tests/unit/test_detection_runner.py

Find output in the DB

-- All findings for a scan, sorted by severity:
SELECT jsonb_path_query(findings, '$[*]') AS finding
FROM scans
WHERE id = '<scan_id>';

-- Count scans by primary anti-bot vendor:
SELECT
  jsonb_path_query_first(findings, '$[*] ? (@.kind == "anti_bot") .vendor') AS vendor,
  count(*)
FROM scans
WHERE status = 'complete'
GROUP BY vendor;

Adding a new detection rule

1. Open detection/rules/anti_bot.py (or the appropriate kind file).
2. Add a Rule(...) entry with vendor name, signal type, pattern, confidence weight.
3. Add a unit test in tests/unit/test_detection_anti_bot.py using a captured sample of the signal.
4. Run the test suite to confirm no regression in scoring on existing scans.

Endpoint analysis.

Structures the raw capture into a typed endpoint inventory with templated URLs, response shapes, dependency edges, pagination patterns, and auth signals.

What it does

Five sub-passes, all deterministic:

1. URL templating — collapses /api/users/12345 and /api/users/67890 into /api/users/<id> with two example values. Recognises numeric ids, UUIDs, long hex slugs.
2. Response shape extraction — for each endpoint, summarises the JSON shape (key list, array depth, response_size in bytes).
3. Dependency chain inference — scans request bodies for values that appeared in earlier response bodies. If endpoint B's request contains a token A's response surfaced, record edge A → B.
4. Pagination detection — recognises ?page=N, ?cursor=…, ?offset=N&limit=M patterns across URL families.
5. Auth signal extraction — flags endpoints that carry Authorization, require cookies, or send/receive CSRF tokens.

Where it lives

Package

browser_recon_server/analysis_server/ (post-T53) or browser_recon/analysis/

Entry

analysis.runner.run_analysis(capture, detection) -> AnalysisResult

Templating

analysis/url_template.py

Bucket signals

analysis/bucket_signals.py — auth + dependency signals

Pagination

analysis/pagination.py

Output shape

{
  "endpoints": [
    {
      "id": "ep_017",
      "url_template": "/api/products/<id>",
      "method": "GET",
      "hostname": "example.com",
      "observed_count": 5,
      "response_shape": {"shape": "object{name,price,sku}", "size_bytes": 54000},
      "required_cookies": ["_abck", "session_id"],
      "required_header_values": {"Referer": "https://example.com/products"},
      ...
    }
  ],
  "framework_hints": [{"name": "Next.js", "evidence": "_next/api path"}],
  "flow_timeline": [{"step_type": "click", "target": "button#search", ...}],
  "cors_summary": [...]
}

Database impact

• Writes to scans.analysis_output (JSONB)
• The endpoints list is the universe Bucket A/B/C classification operates on in step 9
• Emits start + complete events in scan_events

How to test it

rye run pytest tests/unit/test_analysis_runner.py
rye run pytest tests/unit/test_url_template.py
rye run pytest tests/unit/test_bucket_signals.py

Find output in the DB

-- Endpoint inventory for a scan:
SELECT jsonb_array_length(analysis_output->'endpoints') AS n_endpoints
FROM scans WHERE id = '<scan_id>';

-- Endpoints with required cookies:
SELECT ep
FROM scans, jsonb_array_elements(analysis_output->'endpoints') AS ep
WHERE id = '<scan_id>'
  AND jsonb_array_length(ep->'required_cookies') > 0;

Intent filter.

First LLM call. Classifies captured endpoints into Bucket A (the data), Bucket B (prerequisites), Bucket C (noise) based on the user's typed intent.

What it does

Most captures contain 30–80 endpoints. Most are noise — analytics pixels, telemetry beacons, third-party tag managers. The intent filter sends the user's intent text + the endpoint inventory to Claude Sonnet with a structured-output prompt, and gets back three bucketed lists:

• Bucket A — the data the user actually wants (product detail, search results, reviews).
• Bucket B — prerequisites the scraper must hit first (session bootstrap, config endpoints, anti-bot init).
• Bucket C — pure noise (Adobe Analytics pixels, Bloomreach, Criteo retargeting).

Why Sonnet

Judgement-heavy task. A getReviews endpoint might look like noise to a cheap model but it's load-bearing when the user said "scrape reviews." Cheaper models also over-prune Bucket B (the T48 eval showed Grok-3-mini and gpt-5.4-nano dropping checkout/payment session endpoints scrapers actually need). The cost overhead for Sonnet over Grok-3-mini is ~$0.025/scan — minor compared to the cost of shipping a broken scraper.

Where it lives

Prompt

browser_recon_server/prompts/intent_filter.py

Output schema

FilterOutput Pydantic model

Caller

browser_recon_server/scan_pipeline.py:_run_intent_filter

Output shape

{
  "bucket_a": ["ep_012", "ep_019", "ep_020", ...],
  "bucket_b": ["ep_003", "ep_004", "ep_005", ...],
  "bucket_c": ["ep_001", "ep_002", ...],
  "rationale": "Bucket A contains the core data endpoints...",
  "confidence_entries": [{"endpoint_id": "ep_012", "confidence": 0.92}, ...]
}

Database impact

• Writes to scans.bucket_assignment (JSONB)
• One row in llm_calls with cost, tokens, S3 prompt+response keys
• Events emitted in scan_events

How to test it

rye run pytest tests/unit/test_intent_filter_prompt.py

# Re-run on an existing scan (admin only):
recon llm-eval --scan-id <scan_id> --prompts intent_filter --provider-b claude-sonnet-4-6 --force

Find output in the DB

-- Bucket assignment for a scan:
SELECT bucket_assignment FROM scans WHERE id = '<scan_id>';

-- Find the LLM call row:
SELECT model, cost_usd, input_tokens, output_tokens, s3_response_path
FROM llm_calls
WHERE scan_id = '<scan_id>' AND prompt_name = 'intent_filter';

Active validation.

The T51 work. Fires real HTTP requests through operator-owned proxies to confirm which library + proxy combo actually works for each Bucket A endpoint.

What it does

For each Bucket A endpoint, walks a four-step cascade:

1. Library comparison — fans out 3 libraries (requests, httpx, curl_cffi) × 2 proxy tiers (datacenter, residential) in one parallel ThreadPoolExecutor wave. If all 6 attempts block, falls back to cloudscraper as Phase B. Picks the winner by preference order (simplest library that works).
2. Header reduction — tier list skips Accept-Encoding, sec-fetch-*, sec-ch-ua-* without testing; remaining headers tested in parallel waves with a 2-axis (library × proxy) cascade on failures, capped at 6 attempts per header.
3. Cookie dependency — three scenarios (cold / warmup / full) fired in parallel with the same cascade for failures.
4. Rate-limit probe — 3 rounds × 5 requests at delays [2.0s, 0.5s, 0.2s] with the winning library/proxy. Tags result with proxy_rotation_mode so synthesis treats rotating-residential measurements as a lower bound.

For each Bucket B endpoint: just one verification request with the scan-level winner. Mini fan-out only on failure (no header/cookie/rate-limit probing on B).

Where it lives

Package

browser_recon_server/validation_server/ (post-T53) or browser_recon/validation/

Orchestrator

validation/__init__.py::validate(...)

Library cascade

validation/library_compare.py

Header reduction

validation/header_reduce.py

Cookie probe

validation/cookie_dependency.py

Rate-limit

validation/rate_limit_probe.py

Cascade helpers

validation/_cascade.py (T51.5)

Proxy credentials

Read from AWS Secrets Manager (post-T53.5) or server env vars (pre-T53). Never from customer's environment. Customer's shell stays clean of operator secrets.

Bandwidth tracking

Every HTTP attempt records request_bytes + response_bytes. Rolled up to a bandwidth_summary per endpoint and used to compute per_1k_requests_usd from the configured per-tier $/GB rate:

cost_per_1k = (avg_req_bytes + avg_resp_bytes) * 1000 / 1e9 * rate_usd_per_gb

# Datacenter: $2.00/GB · Residential: $3.00/GB

Output shape

Three projections per endpoint, all persisted:

• validation_data — full attempt log, source of truth. Every library × proxy attempt, every header probe, every rate-limit round.
• llm_view — ~250 token compact projection for the synthesis prompt (was ~4,600 tokens pre-T51.8).
• report_view — medium-detail projection for the Jinja Validation section.

Database impact

• Writes one row per endpoint to validation_runs (append-only — re-runs insert new rows)
• Writes to scans.validation for the orchestrator's typed reconstruction
• Emits scan_events rows during long validation passes (e.g. "Validating 2/4 Bucket A endpoints")

How to test it

rye run pytest tests/unit/test_library_compare.py
rye run pytest tests/unit/test_header_reduce.py
rye run pytest tests/unit/test_cookie_dependency.py
rye run pytest tests/unit/test_rate_limit_probe.py
rye run pytest tests/unit/test_validation_orchestrator.py
rye run pytest tests/integration/test_two_round_validation.py

Find output in the DB

-- All validation rows for a scan:
SELECT endpoint_id, status, duration_ms,
       validation_data->>'best_library' AS best_lib,
       validation_data->'bandwidth_summary'->>'per_1k_requests_usd' AS cost
FROM validation_runs
WHERE scan_id = '<scan_id>'
ORDER BY created_at;

-- LLM-view projection (the slim shape synthesis sees):
SELECT llm_view FROM validation_runs
WHERE scan_id = '<scan_id>' AND endpoint_id = 'ep_017';

Known limits

• Cookies captured locally are bound to the customer's residential IP; validation through a different proxy may trigger 412 challenges (we see this on PerimeterX-protected sites). The recommendation surfaces "cookie warmup required" when this is the case.
• Rate-limit probe through rotating-residential proxies fires from a different IP per request, so the target physically cannot rate-limit. The result carries proxy_rotation_mode: "rotating" and a caveat string; the synthesis prompt is instructed to recommend ≥1.5 s delay regardless of measured value.
• Spot-check semantics: ~100 requests over ~30 s ≠ 50,000 requests/hour from a sticky IP. The report copy says "spot-check, not production-scale."

Secret scrubber.

Removes confidential values from the in-memory capture after validation, before persistence and downstream LLM calls. Secrets only ever live in server RAM transiently.

What it does

What stays

• Cookie / header names (the synthesis prompt needs to tell the user "your scraper will need a _abck cookie")
• Non-sensitive headers (Accept-Language, Referer, Content-Type)
• Bucket A response shapes (so synthesis can write data = r.json()["products"])
• URL paths + query parameters (rare for secrets to live there; if regex finds tokens we scrub)

Where it lives

Module

browser_recon_server/scrubber.py (post-T53) or browser_recon/scrubber.py

Entry

scrub_capture(blob) -> scrubbed_blob

Rules

scrubber/patterns.py — regex library for token detection

When it runs

Step 11 of the pipeline — after validation, before synthesis. This ordering is critical: validation needs the real cookies to test cookie-warmup scenarios; synthesis only needs names + shapes. The S3 capture blob is rewritten in-place after scrubbing so secrets never persist long-term.

Database impact

• Rewrites the capture blob at the same S3 key (KMS-encrypted; old version is the unscrubbed pre-validation copy)
• Sets scans.scrubbed_at timestamp
• All subsequent JSONB writes (synthesis input, report_view, llm_view) contain only scrubbed data

How to test it

rye run pytest tests/unit/test_scrubber.py
rye run pytest tests/unit/test_scrubber_patterns.py

Confirm scrubbing worked

# Pull the persisted capture from S3 and grep for secret patterns:
aws s3 cp s3://browser-recon-prod/captures/<scan_id>.json.gz - \
  | gunzip \
  | grep -E '"value": "(eyJ|Bearer |[a-f0-9]{32})'

# Should return nothing for a scrubbed capture.

Scan synthesis.

The main LLM call. Produces the recommendation, the verdict, and the runnable starter script in one combined Claude Sonnet 4.6 invocation.

What it does

One scan_synthesis call (combined prompt from T16) that gets fed:

• The user's intent + Bucket A/B endpoint inventory (filtered, scrubbed)
• Detection findings (anti-bot vendors with severity tier)
• The compact validation slot from step 10 (library_matrix, best library/proxy, min_required_headers, cookie scenarios, rate-limit, bandwidth cost)
• Target identity (URL, domain)

Returns three structured sub-outputs in a single response:

1. recommendation — {primary_library, impersonation, proxy_type, confidence, cost_band, rationale}. The actionable verdict.
2. verdict — 2–3 sentence plain-English headline at the top of the report.
3. starter_code — runnable Python script that uses the recommended library + headers + cookies + rate-limit delay from validation. T52.1's "minimum comments" rule keeps it terse.

Why combined into one call

Before T16 this was 3 separate prompts (recommendation, verdict, starter_code) that each re-sent the same ~80k of input. Combining saved roughly two-thirds of the input tokens. T51.8 then cut another ~80% by replacing the bloated validation_results_* payload with the slim llm_view.

Where it lives

Prompt

browser_recon_server/prompts/scan_synthesis.py

System prompt

SYSTEM_PROMPT constant — superset of the three legacy prompts

Output schema

CombinedSynthesisOutput Pydantic model

Orchestrator

browser_recon_server/synthesis_orchestrator.py

Legacy rollback

BROWSER_RECON_USE_LEGACY_SYNTHESIS=1 env flag re-enables the 3-call fan-out

Database impact

• Writes to scans.synthesis (JSONB, holds all three sub-outputs)
• Writes scans.primary_recommendation, cost_band_low_usd, cost_band_high_usd, confidence at the top level for dashboard queries
• One llm_calls row with cost + S3 prompt+response keys

How to test it

rye run pytest tests/unit/test_scan_synthesis_prompt.py
rye run pytest tests/unit/test_synthesis_orchestrator.py

# Re-run on an existing scan via the eval system:
recon llm-eval --scan-id <scan_id> --prompts scan_synthesis --provider-b claude-sonnet-4-6 --force

Find output in the DB

-- Top-level recommendation fields:
SELECT primary_recommendation, cost_band_low_usd, cost_band_high_usd, confidence
FROM scans WHERE id = '<scan_id>';

-- Full synthesis output:
SELECT synthesis FROM scans WHERE id = '<scan_id>';

-- Starter code only:
SELECT synthesis->'starter_code'->>'code'
FROM scans WHERE id = '<scan_id>';

Notes & difficulty drivers.

Two cheap parallel Grok-3-mini calls that produce report sections orthogonal to the recommendation.

What they do

• notes — 5 short pointers the scraper engineer should know but the recommendation doesn't surface (e.g. "GraphQL introspection appears enabled", "review pagination uses ?page=N", "this endpoint requires X-APOLLO-OPERATION-NAME header"). Renders as bullets in section 02 of the report.
• difficulty_drivers — ranked list of what makes this site hard to scrape. Each driver has a severity tier (easy / medium / high) and a one-sentence explanation. Drives section 04 of the report.

Parallelism

Run via synthesis_orchestrator.py's thread pool, fanning out simultaneously with scan_synthesis. Combined wall-clock is ~max of the slowest, not sum.

Where they live

Notes

browser_recon_server/prompts/notes.py

Drivers

browser_recon_server/prompts/difficulty_drivers.py

Orchestrator

synthesis_orchestrator.py

Database impact

• Writes to scans.notes and scans.difficulty_drivers (JSONB)
• Two llm_calls rows (one per prompt)

Find output in the DB

SELECT
  notes->'notes' AS notes,
  difficulty_drivers->'drivers' AS drivers
FROM scans WHERE id = '<scan_id>';

Report renderer.

Final pipeline stage. Reconstructs typed objects from persisted JSONB and renders the HTML report through Jinja templates.

What it does

Reads every JSONB column from the scans row, reconstructs typed dataclasses via browser_recon_server/report_renderer.py, and renders templates/report.html through Jinja. The template orchestrates 8 section partials in order:

Download mode (T52)

The same template renders into a self-contained HTML file when download_mode=True is passed in the context. Static CSS/JS get inlined; dynamic features (Rerun, Feedback, CSRF meta) are stripped. Served via GET /r/<slug>/download with Content-Disposition: attachment.

Where it lives

Renderer

browser_recon_server/report_renderer.py

Top template

templates/report.html

Partials

templates/partials/*.html

Filters

templates/__init__.py — curl_snippet, httpx_snippet Jinja filters

Database impact

• Updates scans.status = 'complete' and scans.completed_at
• Updates reports row with the rendered fields the dashboard query needs
• Final scan_events row: (step='render', status='complete') — this is what unblocks the CLI's polling loop

How to test it

rye run pytest tests/unit/test_report_renderer.py
rye run pytest tests/unit/test_report_templates.py
rye run pytest tests/unit/test_report_partials_t12.py
rye run pytest tests/unit/test_reports_endpoint.py

View a report

# Live (authenticated):
https://browser-recon.com/r/<slug>

# Self-contained download:
https://browser-recon.com/r/<slug>/download

Scan events & polling.

Append-only event log of pipeline step transitions. The CLI polls every 1.5 s and renders a live spinner. Replaces the silent 90-second wait.

How it works in one paragraph

Every time the server starts or finishes a step (detection, validation, synthesis, etc.), it writes a row to a scan_events table. The CLI asks the server every 1.5 seconds "what's new since I last asked?" and prints any new rows as live progress. When the final "render complete" event arrives, the CLI stops polling and prints the report URL.

Three pieces

1. The table

CREATE TABLE scan_events (
  id           uuid         PRIMARY KEY DEFAULT gen_random_uuid(),
  scan_id      uuid         NOT NULL REFERENCES scans(id) ON DELETE CASCADE,
  step         text         NOT NULL,    -- 'detection' | 'analysis' | 'intent_filter' | ...
  status       text         NOT NULL,    -- 'started' | 'complete' | 'errored'
  message      text,                       -- human-readable progress text
  metadata     jsonb,                      -- optional: endpoint counts
  created_at   timestamptz  NOT NULL DEFAULT now()
);
CREATE INDEX idx_scan_events_scan_id_time ON scan_events (scan_id, created_at);

2. The emit helper

def emit_event(session, scan_id, step, status="started",
               message="", metadata=None):
    session.add(ScanEvent(
        scan_id=scan_id, step=step, status=status,
        message=message, metadata=metadata or {},
    ))
    session.flush()

3. The polling endpoint

@router.get("/scans/{scan_id}/events")
def get_events(scan_id: UUID, since: datetime | None = None):
    # Returns events strictly newer than `since`, ascending order.
    # Includes a `cursor` (latest created_at) so the CLI can advance.

CLI spinner

$ recon scan https://walmart.com
✓ detection      complete   2 anti-bot vendors detected
✓ analysis       complete   38 endpoints categorized
✓ intent_filter  complete   Bucket A: 4 · B: 8 · C: 26
⠹ validation     running    Validating 4 Bucket A endpoints (2/4) · 18s
  scrub          pending
  synthesis      pending
  render         pending

Where it lives

Migration

alembic/versions/<hash>_add_scan_events.py

Model

browser_recon_server/models.py::ScanEvent

Helper

browser_recon_server/events.py::emit_event

Endpoint

browser_recon_server/routes/scans.py::get_events

CLI polling

browser_recon/cli/poll.py

Find output in the DB

-- Full event timeline for a scan:
SELECT step, status, message, created_at
FROM scan_events
WHERE scan_id = '<scan_id>'
ORDER BY created_at ASC;

-- Per-step duration:
SELECT step, status, created_at,
       lead(created_at) OVER (PARTITION BY scan_id ORDER BY created_at)
         - created_at AS duration
FROM scan_events
WHERE scan_id = '<scan_id>';

LLM eval matrix.

Re-runs scan synthesis (or any individual prompt) through alternate LLM providers for quality comparison. Persisted to llm_evals with an admin UI matrix.

What it does

Given an existing scan, re-runs any prompt (typically scan_synthesis or intent_filter) through a different model — Claude, OpenAI GPT, xAI Grok — using the same input the production call saw. Results land in llm_evals as an append-only history. The admin UI at /admin/evals/<scan_id> renders a matrix where rows are prompts and columns are models; clicking any cell opens a side-by-side diff modal against the production response.

Why it exists

Quality validation. Lets the operator answer questions like:

• Could we switch scan_synthesis from Claude Sonnet to Grok-3-mini and save 80% of the cost? (Answer per the Staples eval: no — cheap models give overconfident wrong recommendations on protected sites.)
• Does intent_filter hold up on GPT-5.4? (Answer: gpt-5.4-mini is competitive on small sites; over-prunes Bucket B on complex ones.)
• Are LLM outputs stable across re-runs? (Same model, two runs — append-only history makes this trivially queryable.)

Where it lives

Routes

browser_recon_server/routes/evals.py

Runner

browser_recon_server/eval_runner.py

Lock

browser_recon_server/eval_lock.py — scan-level lock so only one eval runs per scan at a time

Templates

templates/admin_evals_index.html, admin_evals_scan.html, partials/evals_matrix.html

CLI

recon llm-eval --scan-id <id> --provider-b <model>

Database impact

• Appends to llm_evals on every run (no UNIQUE; re-runs preserve history)
• S3 keys per response saved at s3_response_path

How to test it

# From the admin UI: navigate to /admin/evals, pick a scan,
# click "Start new eval", pick prompts + model, confirm spend.

# From the CLI (operator only):
recon llm-eval --scan-id 82f42438-... \
               --prompts scan_synthesis,intent_filter \
               --provider-b grok-3-mini \
               --confirm-spend 1.00

Find output in the DB

-- Latest complete eval per (scan, prompt, model):
SELECT DISTINCT ON (scan_id, prompt_name, model)
       prompt_name, model, provider, cost_usd, response_body
FROM llm_evals
WHERE scan_id = '<scan_id>' AND status = 'complete'
ORDER BY scan_id, prompt_name, model, created_at DESC;

The recon CLI.

The customer-facing command-line tool. Installed once via pipx, used by typing recon in a terminal.

Install

pipx install browser-recon

# If pipx isn't installed:
python3 -m pip install --user pipx
python3 -m pipx ensurepath
pipx install browser-recon

Commands

What ships in the wheel (v0.3.0)

Only non-proprietary glue. The wheel contains:

• browser_recon/capture/ — Chrome launcher + CDP listener
• browser_recon/cli/ — entry point, prompts, login, config
• browser_recon/client.py — thin httpx wrapper around the server API
• browser_recon/poll.py — scan-events polling loop + rich spinner

Roughly 2 MB installed. No detection rules, no validation logic, no LLM prompts, no scoring heuristics. Everything proprietary lives server-side.

What ran where (pre-T53 vs post-T53)

CLI ↔ server protocol

Exactly two endpoints:

POST /scans/<id>/capture       // upload raw blob, returns 202 + status_url
GET  /scans/<id>/events?since   // poll for pipeline progress events

Both authenticated via Authorization: Bearer rec_live_…. TLS 1.2/1.3 in transit (Render terminates).

Config file

~/.recon/config.toml:

[auth]
api_key = "rec_live_..."

[server]
base_url = "https://browser-recon.com"

Source

Repo

github.com/<org>/browser-recon

Entry

browser_recon/cli/main.py::main

Login

browser_recon/cli/login.py

Config

browser_recon/cli/config.py — reads ~/.recon/config.toml

Build

rye build → wheel in dist/

Publish

twine upload dist/* or tag-driven GitHub Actions release

From signup to first scan.

What a brand-new customer does, in order. Five touchpoints: sign up, get an API key, install the CLI, log in, run a scan.

1 · Signup

Customer visits browser-recon.com. Sees the landing page (templates/landing.html). Clicks "Sign up", enters email + password (no email verification required in MVP — Render manages the form). Server creates a row in users with role='user' and a free-tier credit allocation.

2 · Dashboard login

After signup, redirected to /dashboard/login. Magic-link email flow (templates/dashboard_login.html → dashboard_login_sent.html). User clicks link, lands at /dashboard with a session cookie.

3 · API key creation

From the dashboard, clicks "Create API key" → server generates a rec_live_… token, hashes it into api_keys, shows the plaintext once (templates/dashboard_api_key_created.html). User copies it.

4 · CLI install + login

pipx install browser-recon
recon login
# prompts for API key, validates against server, saves to ~/.recon/config.toml

5 · First scan

recon scan https://example.com

# 1. CLI picks a starter template (products / stocks / etc.)
# 2. CLI asks: "Describe what data you want to scrape:"
# 3. Chrome launches. User browses the site, clicks around, navigates.
# 4. Ctrl+C → flow confirm summary
# 5. CLI uploads capture, polls events, prints live progress.
# 6. When render completes, CLI prints report URL:
#    https://browser-recon.com/r/<slug>

Web UI navigation

Permissions / roles

Admin operations.

A separate web UI at /admin/* for the team to monitor health, costs, AI quality, and individual scans.

Layout

Full-width Tailwind layout (templates/admin_layout.html) separate from the user-facing dashboard. Sidebar nav, sticky header with role chip, content area on the right.

Pages

Where it lives

Routes

browser_recon_server/routes/admin.py, routes/evals.py

Templates

templates/admin_*.html + partials

Auth guard

require_role(["admin", "super_admin"]) dependency

Promoting a user to super_admin

Set BROWSER_RECON_SUPER_ADMIN_EMAIL=… in Render env. The next time a user with that email logs in, their users.role auto-promotes. Used for initial setup; subsequent promotions happen via the admin UI's user management page.

Audit trail

Every admin action (role change, scan soft-delete, eval start) appends to audit_log. Find recent admin activity:

SELECT actor_id, action, target_type, target_id, created_at
FROM audit_log
ORDER BY created_at DESC
LIMIT 50;

Concerns, gaps & pending work.

Honest accounting of what this guide doesn't cover, what's incomplete in the system itself, and what's worth doing next. Read this last.

Documentation gaps

Things this guide could cover but doesn't. Not blockers — just acknowledged gaps if the guide ever becomes the primary onboarding doc.

Code-level dead weight (post-T53 cutover)

The T53 thin-CLI migration left a few orphan modules. None are broken; they're just unreachable now and worth deleting in a follow-up sweep.

Architectural caveats (won't change soon)

• Cookies captured locally are bound to the customer's IP. When validation later fires through a different proxy IP, those cookies are often invalid against the target's anti-bot challenge. Synthesis surfaces this as "cookie warmup required" in the recommendation. There's no clean fix that doesn't involve running Chrome through the proxy too, which would defeat the residential-IP advantage of the customer's network.
• Rate-limit probing through rotating-residential proxies is structurally noisy. Each probe request hits the target from a different IP, so the target physically cannot rate-limit a single source it never sees twice. The result carries a proxy_rotation_mode: "rotating" flag + caveat string; synthesis is instructed to recommend ≥1.5 s delay regardless of measured value.
• The capture upload is unscrubbed. Validation needs the real cookies + auth headers. Scrubbing happens server-side, after validation, before persistence. Secrets live in server RAM transiently. Trade-off accepted in T53 planning.
• Pre-T54.5: AWS Secrets Manager not yet in use. Proxy credentials still live in Render env vars. Migration to AWS Secrets is a planned follow-up.
• Pre-T54.5: PyPI publishing setup not yet done. The wheel builds cleanly (dist/browser_recon-0.3.0-py3-none-any.whl, 128 KB) but hasn't been uploaded to PyPI yet. Customers can't pipx install browser-recon until that step.

Pending tasks (next-up)

Notable wins shipped recently

• T51 (validation redesign): validation wall-clock per Bucket A endpoint went from ~150 s to ~25 s. The new pipeline is two-axis (library × proxy) cascade with parallelism, bandwidth tracking, and a persisted JSON shape rich enough to drive real cost projections.
• T51.8 (LLM payload compaction): scan_synthesis input tokens dropped 65–81% (Walmart: 81k → 29k, Staples: 84k → 16k). Synthesis cost dropped 52–69%.
• T52 (report polish): theme-consistent starter code + Download Report button (self-contained HTML).
• T52.1 (snippet comment trim): curl + httpx + starter code now ship minimal comments.
• T53 (thin-CLI cutover): CLI wheel went from ~20 MB to 128 KB. No proprietary code on customer machines. Operator secrets never travel to the customer's shell. Customers see a live progress spinner instead of silent 90-second waits.

Doc revision history

This guide is a living document. Each major feature ships an update.