From bundled monolith to thin client.

Schema

CREATE TABLE scan_events (
  id           uuid         PRIMARY KEY DEFAULT gen_random_uuid(),
  scan_id      uuid         NOT NULL REFERENCES scans(id) ON DELETE CASCADE,
  step         text         NOT NULL,
  status       text         NOT NULL,    -- 'started' | 'complete' | 'errored'
  message      text,                       -- human-readable progress text
  metadata     jsonb,                      -- optional: endpoint counts, partials
  created_at   timestamptz  NOT NULL DEFAULT now()
);

CREATE INDEX idx_scan_events_scan_id_time ON scan_events (scan_id, created_at);

Helper

def emit_event(session, scan_id, step, status="started",
               message="", metadata=None):
    session.add(ScanEvent(
        scan_id=scan_id, step=step, status=status,
        message=message, metadata=metadata or {},
    ))
    session.flush()

Endpoint

@router.get("/scans/{scan_id}/events")
def get_events(scan_id: UUID, since: datetime | None = None):
    # since = ISO timestamp of last event the CLI received
    # Returns events strictly newer than `since`, ordered by created_at ASC.
    # Includes a `cursor` (latest created_at) so the CLI can advance.

Acceptance

• Alembic migration creates the table + index
• Existing pipeline (today's synchronous synthesis path) calls emit_event at every step boundary; at minimum: detection, analysis, intent_filter, validation, synthesis, render
• Endpoint returns the new events; since is exclusive

What moves

Same code, new import paths. Keep browser_recon/validation/ etc. on disk during the transition so nothing breaks; delete them in T53.6 once the new endpoint (T53.3) is verified end-to-end. Run the existing test suite (3,636 tests) against the copies under their new module names — most tests pass as-is with an import update.

Acceptance

• Server can import + call the moved modules at their new paths
• Existing CLI still works (legacy code path intact)
• Tests duplicated under tests/unit/server_* all green

Request shape

POST /scans/{scan_id}/capture
Authorization: Bearer rec_live_...
Content-Type: application/json

{
  "target_url": "https://walmart.com",
  "intent_text": "Need to search product, get data and reviews.",
  "starter_template": "products",
  "capture": { /* raw browser capture blob, unscrubbed */ }
}

Response (202 Accepted)

{
  "scan_id": "82f42438-...",
  "status": "queued",
  "events_url": "/scans/82f42438-.../events",
  "report_url": "/r/<slug>"            // known eagerly; only valid after render
}

Server-side flow

• Validate auth + payload shape
• Insert scans row with status='processing'
• Put raw blob in S3 (KMS-encrypted) under captures/<scan_id>.json.gz
• Queue background task that calls the orchestrator (T53.4)
• Return 202 with the status URL — total latency under 500 ms

One function that walks the 8 stages in order, emitting events at each boundary. Pure composition over the modules T53.2 ported.

def run_pipeline(scan_id: UUID, session: Session) -> None:
    capture = load_capture_from_s3(scan_id)

    with step(session, scan_id, "detection"):
        detection = run_detection(capture)
        save_detection(session, scan_id, detection)

    with step(session, scan_id, "analysis"):
        analysis = run_analysis(capture, detection)
        save_analysis(session, scan_id, analysis)

    with step(session, scan_id, "intent_filter"):
        buckets = run_intent_filter(analysis, capture.intent_text)
        save_buckets(session, scan_id, buckets)

    with step(session, scan_id, "validation"):
        proxies = load_proxy_secrets()           # T53.5: AWS Secrets
        validation = run_validation(buckets, proxies)
        save_validation(session, scan_id, validation)

    with step(session, scan_id, "scrub"):
        scrubbed = scrub_capture(capture)
        save_scrubbed_blob(session, scan_id, scrubbed)

    with step(session, scan_id, "synthesis"):
        synthesis, notes, drivers = run_synthesis_orchestrator(
            scan_id, scrubbed, detection, buckets, validation,
        )
        save_synthesis(session, scan_id, synthesis, notes, drivers)

    with step(session, scan_id, "render"):
        render_report(session, scan_id)

    mark_complete(session, scan_id)

The step context manager emits started on enter, complete on clean exit, errored with the exception message on any failure (and re-raises so the scan marks as failed).

What changes

Currently library_compare.py reads DATACENTER_PROXY and RESIDENTIAL_PROXY from environment variables. After the port (T53.2), the server-side copy reads them from AWS Secrets Manager via the existing boto3 setup.

Implementation

import boto3, json
from functools import lru_cache

@lru_cache(maxsize=1)
def _proxy_secrets() -> dict[str, str]:
    client = boto3.client("secretsmanager", region_name=AWS_REGION)
    resp = client.get_secret_value(SecretId=PROXY_SECRET_ARN)
    return json.loads(resp["SecretString"])

def load_proxy_secrets() -> dict[str, str | None]:
    secrets = _proxy_secrets()
    return {
        "datacenter":  secrets.get("DATACENTER_PROXY"),
        "residential": secrets.get("RESIDENTIAL_PROXY"),
        "dc_rotation": secrets.get("DATACENTER_PROXY_ROTATION_MODE", "sticky"),
        "res_rotation": secrets.get("RESIDENTIAL_PROXY_ROTATION_MODE", "rotating"),
    }

IAM

Server's task role gets secretsmanager:GetSecretValue for the proxy secret ARN. Render env now needs only AWS credentials (or OIDC integration if available).

Acceptance

• Proxy URLs no longer present in Render env
• load_proxy_secrets() returns the same shape _proxy_env() returns today
• Secret rotation works without code redeploy (lru_cache TTL or manual flush on SIGHUP)

pyproject.toml

[tool.hatch.build.targets.wheel]
packages = ["browser_recon"]   # was: ["browser_recon", "browser_recon_server"]

Filesystem

rm -rf browser_recon/validation
rm -rf browser_recon/analysis
rm -rf browser_recon/detection
rm browser_recon/scrubber.py
rm browser_recon/reporting/markdown_local.py

Update browser_recon/cli/main.py to remove imports of the deleted modules. The CLI now physically cannot do validation locally.

Version bump

[project]
name = "browser-recon"
version = "0.3.0"   # was 0.2.5

Acceptance

• rye build produces a wheel with browser_recon_server absent from RECORD
• Test suite still green after the deletions (tests for the moved modules now live under tests/unit/server_*)

Three changes

1. Replace local validation with capture upload. After capture finishes, instead of running validate() locally, the CLI calls POST /scans/<id>/capture with the raw blob.
2. Add a polling loop that calls GET /scans/<id>/events?since=<cursor> every 1.5 seconds, advancing the cursor each round.
3. Render a rich.live.Live spinner per step. Each step has a status (pending · running · complete · errored) and a message line.

Mock — what the customer sees

Acceptance

• CLI emits no requests besides POST /capture and GET /events after capture
• Spinner reflects every step transition within 1.5–3 s
• Errored steps print the error message inline, then the CLI exits with code 1
• --no-progress flag still ships scan to completion without the spinner (CI use)

One-time setup

1. Create a PyPI account at pypi.org/account/register/
2. Enable 2FA (required for publishing)
3. Generate an API token scoped to the browser-recon project (after first manual upload, you can scope it; for the very first upload you need an account-wide token)
4. Save the token to a password manager and to GitHub Secrets as PYPI_API_TOKEN
5. (Optional) Create a TestPyPI account at test.pypi.org for dry-run uploads

Build + upload (local)

rye build                              # produces dist/browser-recon-0.3.0-py3-none-any.whl + .tar.gz
twine check dist/*                      # validates metadata before upload
twine upload dist/*                     # prompts for token; uploads to PyPI

# Dry-run via TestPyPI first if you want:
twine upload --repository testpypi dist/*
pipx install --index-url https://test.pypi.org/simple/ browser-recon==0.3.0

Automated release (GitHub Actions)

One workflow file at .github/workflows/release.yml triggers on a version tag, runs the test suite, builds the wheel, and pushes to PyPI:

name: release
on:
  push:
    tags: ["v*"]
jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write          # for PyPI trusted publishing
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install rye twine
      - run: rye sync && rye run pytest -q
      - run: rye build
      - uses: pypa/gh-action-pypi-publish@release/v1
        with:
          packages-dir: dist/

Trusted publishing (preferred over tokens)

PyPI supports OIDC-based trusted publishing from GitHub Actions — no API token needed. Register your repo + workflow in the PyPI project settings under "Publishing". This is the modern recommended approach.

What customers run

pipx install browser-recon              # preferred — isolated venv
recon login                             # enter API key
recon scan https://walmart.com          # go

Acceptance

• pipx install browser-recon works on a fresh Linux/Mac/Windows machine
• recon --version reports 0.3.0
• Wheel size < 5 MB (target ~2 MB)
• Tag-driven release pipeline works end-to-end

Versioning headers

Every request from the CLI sends User-Agent: browser-recon/0.3.0 so the server can log version distribution and gracefully reject ancient clients with a clear "please upgrade" message.

Update notice on the CLI

CLI hits GET /version on startup. If its version is older than the latest released, prints a non-blocking one-liner:

browser-recon 0.3.0 (a newer version 0.3.1 is available; run `pipx upgrade browser-recon`)

Legacy CLI back-compat

Customers on v0.2.x still call the old POST /scans/<id>/complete endpoint with their locally-computed validation result. Keep that endpoint alive for one release cycle (until v0.4.0). Customers who haven't upgraded by then get a 410 Gone with the upgrade message.

Migration messaging

One email to all active customers two weeks before deprecating the legacy endpoint. Dashboard banner for super-admin showing the version-distribution count.