Stop guessing what works. Measure it through the right proxy.

What changes

Read DATACENTER_PROXY and RESIDENTIAL_PROXY from env. Every HTTP call inside library_compare.py threads a proxy_url argument through. Per-library ladder: try datacenter once; on block/fail, fire 3 sequential requests through residential. Residential outcome:

• 3 / 3 pass → record working_tier = "residential", spot-check passed
• 2 / 3 pass → record working_tier = "residential", label as flaky
• 0–1 / 3 pass → library marked blocked under residential too

When env vars are unset

Validation runs on the developer's direct IP — it's the user's CLI machine, not a server. The report's Validation section surfaces a No proxies configured banner and the synthesis system prompt is told proxy_status="direct_ip" so its recommendation includes a caveat.

Env vars to wire

DATACENTER_PROXY=http://user:pass@dc.example.com:8080
RESIDENTIAL_PROXY=http://user:pass@residential.example.com:8080
PROXY_DATACENTER_PRICE_USD_PER_GB=2.00
PROXY_RESIDENTIAL_PRICE_USD_PER_GB=3.00

Acceptance

• Walmart re-validation with both env vars set produces working_tier on every passing library
• Same scan without env vars set produces direct-IP results + "No proxies configured" banner in the report

What changes

Replace the serial for lib_name, func in _LIBRARY_FUNCS.items() loop in library_compare.py:351 with a ThreadPoolExecutor(max_workers=6) that fans out 3 libraries × 2 proxy attempts in one wave. Once Phase A returns, pick the winner by preference order, not lowest latency:

# Preference cascade — first passing library wins.
LIBRARY_PREFERENCE = ["requests", "httpx", "curl_cffi"]
LIBRARY_FALLBACK   = ["cloudscraper"]  # Phase B only

passing = [
    lib for lib in LIBRARY_PREFERENCE
    if phase_a[lib].working_tier is not None
]

if passing:
    best = passing[0]                   # requests > httpx > curl_cffi
else:
    best = run_phase_b(LIBRARY_FALLBACK)  # cloudscraper, both proxies

Why preference order, not latency

A scraper that works under requests is the simplest possible production artifact — fewer deps, more readable starter code. Latency differences inside validation are noise (single sample, 200–400ms variance is normal). Preference order is a deliberate simplicity bias.

Cloudscraper as Phase B only

cloudscraper pulls in js2py, can take 4–15s when it has to solve a challenge, and fails noisily. It fires only when all 6 Phase A attempts fail — meaning the site is genuinely hard and we need a JS-aware tool. On easy sites it never runs.

Bucket B endpoints — inherit scan-level winner, verify cheaply

Bucket A endpoints run the full library × proxy fan-out. Bucket B endpoints do not — they're prerequisites (bootstrap calls, session config), and production code will hit them with the same (library, proxy_tier) the scraper already uses for Bucket A. Running an independent ladder for each B endpoint would waste requests on data we know we won't act on differently.

Expected cost: 1 request per B endpoint on most scans (winner inherits cleanly). The mini-fan-out only fires when a B endpoint has stricter checks than the A endpoints — rare, but the cascade catches it.

Dependency

rye add cloudscraper

Acceptance

• 3 Phase A libraries × 2 proxies fire in < 2s wall-clock (was ~6s serial)
• Cloudscraper never appears in library_compare.attempts when any Phase A library passed
• Cloudscraper appears in library_compare.attempts when Phase A is all-blocked
• Bucket B reachability completes in < (1.5s × N_b_endpoints) when scan_winner inherits cleanly

Tier list (skip-without-testing)

Parallel + cascade algorithm

Result shape (one header)

{
  "Referer": {
    "required_under": [
      {"library": "requests", "proxy": "datacenter"},
      {"library": "requests", "proxy": "residential"}
    ],
    "optional_under": [
      {"library": "curl_cffi", "proxy": "datacenter"}
    ],
    "attempts": [ /* full attempt log, persisted */ ]
  }
}

What the report shows

A two-row strip: REQUIRED (collapsed to a single combo recommendation) and DROPPED. The richer per-library detail lives in the evidence accordion. Rationale: production scrapers want one config, not a decision tree.

Acceptance

• 15-header endpoint completes header reduction in < 6s wall-clock (was ~26s)
• Tier-skipped headers never appear as attempts in the persisted blob

Algorithm

Three scenarios (cold, warmup, full) fired in parallel with (passing[0], working_tier[0]). Any scenario that fails cascades through the same 6-attempt ladder as T51.3.

Result shape

{
  "per_scenario": {
    "cold":   { "passes_under": [{"lib":"curl_cffi","proxy":"datacenter"}], ... },
    "warmup": { "passes_under": [{"lib":"requests","proxy":"residential"}], ... },
    "full":   { "passes_under": [...] }
  },
  "minimum_scenario": {
    "requests":     "warmup_required",
    "curl_cffi":    "cold",
    "cloudscraper": "warmup_required"
  },
  "caveat": "point-in-time; anti-bot cookies have TTL"
}

Acceptance

• 3 scenarios × winning combo fire in < 3s parallel
• Cascade only fires on failed scenarios

Numerics

Rotation labelling — the critical addition

If the winning proxy is rotating-residential, the probe fires 5 requests from 5 different IPs and the target physically cannot rate-limit. We still run the probe (per your call), but the result carries an explicit warning so the synthesis prompt does not take the measured delay at face value:

{
  "library_used": "requests",
  "proxy_used": "residential",
  "proxy_rotation_mode": "rotating",        // NEW
  "measurement_caveat": "Cadence measured across rotating IPs.
                          Production single-IP behavior may trigger
                          limits sooner. Treat estimated_safe_delay_s
                          as a lower bound; run extensive tests.",
  "estimated_safe_delay_s": 0.75,
  "rounds": [ ... ]
}

Synthesis prompt addendum

If `proxy_rotation_mode == "rotating"` on a rate_limit result,
prefer a conservative production delay (≥ 1.5 s) in your recommendation
regardless of the measured `estimated_safe_delay_s`. Surface the caveat
in the verdict.

Acceptance

• Probe completes in < 15s worst-case
• proxy_rotation_mode set correctly based on a per-proxy config flag

Per-request additions

Every library function in library_compare.py already returns status, elapsed_ms, body_full, headers. Add two fields, computed before the verbose ones are stripped:

"request_bytes"  : 1840,   # sum(name+value+4) for headers + len(body)
"response_bytes" : 54120,  # sum of response headers + body

Per-endpoint rollup

{
  "bandwidth_summary": {
    "request_count": 38,
    "bytes_sent_total": 70450,
    "bytes_received_total": 1872400,
    "avg_request_bytes": 1854,
    "avg_response_bytes": 49273,
    "per_1k_requests_mb": 49.3
  }
}

Cost formula (per-tier rate, current vendor)

rate_usd_per_gb = {
    "datacenter":  PROXY_DATACENTER_PRICE_USD_PER_GB,   # 2.00
    "residential": PROXY_RESIDENTIAL_PRICE_USD_PER_GB,  # 3.00
}[best_proxy_tier]

cost_per_1k_requests_usd = (
    avg_request_bytes + avg_response_bytes
) * 1000 / 1e9 * rate_usd_per_gb

# Walmart product page on datacenter:
#   (1854 + 49273) * 1000 / 1e9 * 2.00 = $0.102 per 1k requests
# Same endpoint forced to residential:
#   (1854 + 49273) * 1000 / 1e9 * 3.00 = $0.153 per 1k requests

Synthesis prompt impact

The cost_band_low_usd and cost_band_high_usd fields stop being hand-picked. The bandwidth_summary is passed in; the prompt instructs the model to compute the band from observed bytes × the vendor rate, plus a 20% retry headroom.

Acceptance

• Every persisted attempt carries request_bytes and response_bytes
• Walmart re-validation report shows a computed cost per 1k requests, not a guessed range

Why append-only

A site's defenses change. If today's validation says datacenter works and a re-run tomorrow says datacenter blocked, both outcomes are evidence. Same pattern as llm_evals (T48): every run inserts a row, the latest non-errored row per (scan_id, endpoint_id) is "current."

Schema

CREATE TABLE validation_runs (
  id                  uuid          PRIMARY KEY DEFAULT gen_random_uuid(),
  scan_id             uuid          NOT NULL REFERENCES scans(id),
  endpoint_id         text          NOT NULL,
  status              text          NOT NULL,   -- 'complete' | 'errored' | 'partial'
  created_at          timestamptz   NOT NULL DEFAULT now(),
  duration_ms         int           NOT NULL,
  validation_data     jsonb         NOT NULL,   -- the rich blob from T51.8
  error_message       text
);

CREATE INDEX idx_validation_runs_scan_endpoint
  ON validation_runs (scan_id, endpoint_id, created_at DESC);

Read path

SELECT DISTINCT ON (scan_id, endpoint_id) *
FROM validation_runs
WHERE scan_id = $1 AND status = 'complete'
ORDER BY scan_id, endpoint_id, created_at DESC;

Acceptance

• Alembic migration creates the table + index
• Re-running validation on a scan inserts a new row, doesn't overwrite

Two derived projections over validation_data

The persisted blob is the source of truth; the LLM and the report each read a slim derived view. Nothing recomputes from raw attempts at read time — the projections are materialized during validation and stored as separate JSONB keys on the same row (llm_view, report_view).

LLM payload (~250 tok / endpoint, was ~4,600)

{
  "endpoint_id": "ep_017",
  "method": "GET",
  "best": {
    "library": "requests",
    "proxy_tier": "datacenter",
    "elapsed_ms": 320
  },
  "library_matrix": {
    "requests":     {"datacenter": "ok:200",
                       "residential": "skipped"},
    "httpx":        {"datacenter": "ok:200",
                       "residential": "skipped"},
    "curl_cffi":    {"datacenter": "ok:200",
                       "residential": "skipped"},
    "cloudscraper": {"phase": "B_skipped"}
  },
  "min_required_headers": ["User-Agent", "Referer"],
  "headers_tested": 5, "headers_skipped": 12,
  "cookies": "warmup_required",
  "rate_limit": {
    "safe_delay_s": 0.75,
    "proxy_rotation_mode": "static",
    "trigger": "429@0.2s/round3"
  },
  "bandwidth": {"per_1k_requests_usd": 0.128}
}

Report section — Validation

Mock below. Same identity as the rest of the report; rendered server-side from report_view.