Metadata-Version: 2.4
Name: switchback
Version: 0.6.1
Summary: One cost-ordered scrape cascade (HTTP → stealth browser → paid), shared by every tool.
Author-email: Akash Kodavuru <akash@theaklabs.com>
License: MIT
Project-URL: Homepage, https://github.com/akash-kr/switchback
Project-URL: Repository, https://github.com/akash-kr/switchback
Project-URL: Issues, https://github.com/akash-kr/switchback/issues
Project-URL: Changelog, https://github.com/akash-kr/switchback/blob/main/CHANGELOG.md
Keywords: web-scraping,scraper,crawler,html-to-markdown,markdown,cloudflare-bypass,anti-bot,stealth,playwright,patchright,camoufox,curl-cffi,cloudscraper,firecrawl,headless-browser,residential-proxy,llm,rag,ai-agents,opentelemetry
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: markdownify
Requires-Dist: beautifulsoup4
Requires-Dist: pypdf>=6.19
Requires-Dist: requests>=2.33
Requires-Dist: curl_cffi>=0.15
Provides-Extra: cloudflare
Requires-Dist: cloudscraper; extra == "cloudflare"
Provides-Extra: browser
Requires-Dist: patchright; extra == "browser"
Provides-Extra: camoufox
Requires-Dist: camoufox[geoip]; extra == "camoufox"
Provides-Extra: firecrawl
Requires-Dist: firecrawl-py; extra == "firecrawl"
Provides-Extra: server
Requires-Dist: fastapi; extra == "server"
Requires-Dist: uvicorn; extra == "server"
Provides-Extra: mcp
Requires-Dist: mcp; extra == "mcp"
Provides-Extra: youtube
Requires-Dist: youtube-transcript-api>=1.0; extra == "youtube"
Provides-Extra: documents
Requires-Dist: markitdown[docx,pptx,xls,xlsx]; extra == "documents"
Provides-Extra: tracing
Requires-Dist: opentelemetry-sdk; extra == "tracing"
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc; extra == "tracing"
Provides-Extra: all
Requires-Dist: cloudscraper; extra == "all"
Requires-Dist: patchright; extra == "all"
Requires-Dist: camoufox[geoip]; extra == "all"
Requires-Dist: firecrawl-py; extra == "all"
Requires-Dist: fastapi; extra == "all"
Requires-Dist: uvicorn; extra == "all"
Requires-Dist: opentelemetry-sdk; extra == "all"
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

<!-- switchback -->

```
███████╗██╗    ██╗██╗████████╗ ██████╗██╗  ██╗██████╗  █████╗  ██████╗██╗  ██╗
██╔════╝██║    ██║██║╚══██╔══╝██╔════╝██║  ██║██╔══██╗██╔══██╗██╔════╝██║ ██╔╝
███████╗██║ █╗ ██║██║   ██║   ██║     ███████║██████╔╝███████║██║     █████╔╝
╚════██║██║███╗██║██║   ██║   ██║     ██╔══██║██╔══██╗██╔══██║██║     ██╔═██╗
███████║╚███╔███╔╝██║   ██║   ╚██████╗██║  ██║██████╔╝██║  ██║╚██████╗██║  ██╗
╚══════╝ ╚══╝╚══╝ ╚═╝   ╚═╝    ╚═════╝╚═╝  ╚═╝╚═════╝ ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝
```

<div align="center">

# switchback — Python web scraper: any URL → clean Markdown

**One cost-ordered scrape cascade — HTTP → stealth browser → paid — shared by every tool.**

Give it a URL; it tries the cheapest way to get clean Markdown first and only escalates
to a heavier (slower, costlier) tier when the cheap one is walled. Stops at the first success.

[![PyPI](https://img.shields.io/pypi/v/switchback)](https://pypi.org/project/switchback/)
[![Python](https://img.shields.io/pypi/pyversions/switchback)](https://pypi.org/project/switchback/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![CI](https://github.com/akash-kr/switchback/actions/workflows/ci.yml/badge.svg)](https://github.com/akash-kr/switchback/actions/workflows/ci.yml)

⭐ If switchback saved you a paid-API bill, a star helps others find it.

</div>

---

## Why

Most scrapers either give up on hard pages or send *everything* through an expensive
headless browser / paid API. **switchback** orders the methods by cost and walks them
cheapest-first, per host, learning which tier wins where so the next run starts there.
The easy majority stays free; only genuinely-walled hosts pay for the heavy tiers.

- **Cost-ordered cascade** — free APIs → cheap HTTP → anti-bot solver → stealth browser → paid API.
- **Per-host memory (botwall)** — remembers the winning tier per host, skip-lists hard blockers, auto-skips hosts stuck on the paid tier.
- **Cost-scoped residential egress** — routes *only* walled hosts through a residential proxy, never the easy majority.
- **One shape, three entry points** — Python library, CLI (JSON on stdout), or an HTTP service.
- **Observable** — every attempt is an OpenTelemetry span; logs ship trace-correlated to any OTLP backend (Jaeger, Tempo, SigNoz).
- **Runs with any subset installed** — each tier imports its deps lazily; a missing one is just a tier miss.

## Quickstart

```bash
pip install switchback                 # core: cheap tiers (0/1) + search
```

```python
from switchback import scrape

for r in scrape(["https://arxiv.org/abs/1706.03762"]):
    print(r.source_method, len(r.markdown))
```

```bash
python -m switchback https://example.com/article    # JSON on stdout — bridge for any language
```

That's the whole loop. Add tiers as you need them (see [Install](#install)).

<p align="center">
  <img src="https://raw.githubusercontent.com/akash-kr/switchback/main/docs/assets/demo.gif" width="880"
       alt="switchback scraping a Cloudflare-walled Quora page: tier_1 to tier_4 come up empty or walled, tier_5 (Camoufox) returns clean Markdown in 20.2s; on the second run botwall remembers the host and goes straight to tier_5">
</p>

<sub>A real run, time-compressed. Run 1 climbs the cascade until a tier gets through. Run 2 hits the
same host, and botwall starts at the tier that won last time. The paid tier is never touched.</sub>

## The cascade (stop at first success)

| Tier | Strategy | Cost |
|---|---|---|
| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC, YouTube transcripts; extend: job boards) | free, cleanest |
| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs and office/ebook files | cheap |
| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) — on by default (opt out: `SCRAPER_DISABLE_CLOUDSCRAPER`) | cheap-ish (~5s/host) |
| tier_4 | Stealth headless browser (`patchright`, Chromium) | heavy |
| tier_5 | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
| tier_6 | Residential-IP browser over CDP (`BU_CDP_URL`) — off unless configured | heavy (remote egress) |
| tier_7 | Firecrawl (paid, env-gated, audited) | paid, last resort |

Every URL has a wall-clock budget (`SCRAPER_DEADLINE_S`, default 45s) checked between
tiers so one URL can't run the whole cascade of timeouts. Each tier attempt records
latency + outcome (`ok` / `short_content` / `rate_limited` / `miss` / `not_applicable`)
to its span and the botwall event log; the root span carries total latency and the final
outcome (incl. `deadline_exceeded`).

Search (query → URLs) is separate from the scrape cascade: `switchback.search()` /
`python -m switchback.api --search <query>`, backed by a local SearXNG.


## Install

```bash
pip install switchback                 # core: normalization + cheap tiers (tier_1/tier_2) + search
pip install "switchback[cloudflare]"   # + tier_3 Cloudflare/anti-bot solver (cloudscraper)
pip install "switchback[server]"       # + HTTP service (fastapi, uvicorn) incl. /metrics + /traces
pip install "switchback[browser]" && patchright install chromium   # + tier_4 stealth Chromium
pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
pip install "switchback[all]"          # all of the above
pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)
```

`mcp`, `youtube` and `documents` are not part of `[all]`, so upgrading an existing
`[all]` install doesn't pull new dependencies (`documents` brings markitdown and
onnxruntime). Add them by name. YouTube transcripts and office files also need
their switch (`SCRAPER_YOUTUBE=1`, `SCRAPER_DOCUMENTS=1`), because they change what
those URLs return; without it they behave exactly as before, installed or not.

For tier_3's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
3.x fork (PyPI's `cloudscraper` is the older v1/v2 — PyPI forbids pinning a
git-URL dep inside a published package, so install it alongside):

```bash
pip install "cloudscraper @ git+https://github.com/VeNoMouS/cloudscraper@3.0.0"
```

Or run the whole thing as a container:
`docker build -t switchback . && docker run -p 8799:8799 switchback`.

### Production / cold-start deployment

The two heavy tiers pull dependencies that often can't be baked into a base image
and land *after* boot (e.g. an async install thread on Azure). Until they're
ready, those tiers report **`unavailable`** (a distinct outcome carrying the exact
fix) and the cascade falls through — they are never silently skipped. Checklist:

- **tier_4 is the real workhorse for Cloudflare/JS sites** — make sure its browser
  is installed: `patchright install chromium` (note: **patchright**, not vanilla
  `playwright`). On a cold start, run this in your post-boot install step/thread;
  tier_4 flips to ready once it finishes.
- **tier_3 needs the cloudscraper 3.x fork** (above) to attempt stealth. With the
  frozen PyPI `cloudscraper` it reports `unavailable` and fails fast (no wasted
  solve budget) instead of erroring mid-cascade. tier_3 is a *weak* solver for
  modern Cloudflare — treat it as a cheap try before the browser, not the primary.
- **Install Node.js** for tier_3's v3 JS-VM challenges — faster and thread-safe
  vs. the pure-Python js2py fallback (relevant under concurrent load).
- **Bound tier_3's solve budget** with `SCRAPER_TIER_3_TIMEOUT_S` (default `25`;
  the old `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` is still honored) so an unsolvable
  challenge can't eat the per-URL deadline before the browser tier runs. Lower it
  (e.g. `12`) if tier_3 rarely wins on your hosts.

### Scraping URLs other people submit

If the URLs come from users you don't fully trust (a "save this link" box, a
feed reader, a public API), two opt-in switches harden the engine. Both default
to off, so nothing changes unless you set them:

```bash
export SCRAPER_DISABLE_CLOUDSCRAPER=1   # tier_3 runs site JavaScript outside a browser sandbox
export SCRAPER_BROWSER_BLOCK_PRIVATE=1  # browser tiers can't reach your machine's or network's private services
```

- **Why tier_3:** solving a Cloudflare JS challenge means running JavaScript the
  site sent, in js2py or Node's `vm` module, and neither is a security boundary.
  The browser tiers run page JavaScript inside the browser's own sandbox instead.
- **Install tier_4's browser first** (`patchright install chromium`): with tier_3
  off, it is what clears Cloudflare. `--doctor` shows tier_3 as `off` and still
  exits 0.
- **Why the browser guard:** a scraped page can redirect, frame or `fetch()` an
  address like `http://127.0.0.1:…` or a cloud metadata endpoint, and whatever
  answers comes back as scraped content. With the guard, those requests (and
  WebSockets) are refused and service workers are disabled so they can't route
  around it. Check the URL you pass in yourself too: the guard covers what the
  *page* asks for in the browser tiers, not the HTTP tiers' own requests.

**Verify readiness on the box** with the preflight check (doubles as a healthcheck
— exit 0 when the capable tiers are ready):

```bash
switchback --doctor          # or: python -m switchback --doctor
```

## Use it from your app

Three interchangeable entry points — all return the same shape
(`[{url, source_method, markdown}]`, successes only):

**Python library**
```python
from switchback import scrape
for r in scrape(["https://arxiv.org/abs/1706.03762"]):
    print(r.source_method, len(r.markdown))

# Need failures + reasons too? scrape_detailed returns a ScrapeOutcome per URL
# (ok, final_outcome, error_class, status_code, and the per-tier attempts):
from switchback import scrape_detailed
for o in scrape_detailed(["https://www.pcmag.com/news"]):
    if not o.ok:
        print(o.url, o.final_outcome, o.error_class, o.status_code)
```

**CLI** (JSON on stdout — bridge for any language)
```bash
python -m switchback https://example.com/article        # or: switchback <url>
```

**HTTP service** (language-agnostic; one warm process keeps the browser pool hot)
```bash
switchback-server                                    # listens on :8799
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"]}'
curl 'localhost:8799/search?q=web+scraping'
```

Non-Python callers: see [clients/node_bridge.md](clients/node_bridge.md). Python
callers that want HTTP-with-CLI-fallback can drop in
[clients/python_client.py](clients/python_client.py).

**MCP server** (for agents: Claude Code, Cursor, any MCP client)
```bash
pip install "switchback[mcp]"
claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search
```

## Agent-ready extras

Off by default: without them every result is exactly what it was before, the same
`ScrapeResult` / `ScrapeOutcome` with the same fields, and the same CLI and HTTP
JSON. Ask for any combination per call; results then come back as
`ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras` (subclasses, so existing
`isinstance` checks still hold) with the extras you asked for filled in and the
rest `None`. The CLI and HTTP JSON add only the keys you asked for.

```python
from switchback import scrape
[r] = scrape("https://example.com/product", state=True, appendix=True,
             query="what does it cost?")
r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]
```

```bash
switchback --state --appendix --query "what does it cost?" https://example.com/product
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'
```

| extra | what you get |
| --- | --- |
| `state` | The JSON the page embeds for its own frontend: Next.js `__NEXT_DATA__`, Nuxt, `window.__X__` globals (incl. `JSON.parse("…")`), schema.org JSON-LD. Often cleaner than the text (prices, ids, dates). Only valid JSON is returned. `{}` when the winning tier had no HTML (API mirrors, PDFs, docs, Firecrawl markdown). |
| `appendix` | The menus, header, footer and sidebars that are stripped from the main content, as separate markdown, one entry per region. |
| `query` → `highlights` | The passages that answer the question, best first, each with a citation `link`: a [text fragment](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments) URL that opens the page scrolled to, and highlighting, that passage (for a YouTube transcript, the video at that moment). If the text isn't visible on the page (e.g. it came from the `<title>` or image alt text) the browser simply opens the page at the top. |

`state` and `appendix` need the fetched HTML, so they skip the content cache
(`SCRAPER_CONTENT_TTL_S`) and fetch; `highlights` work on cached content too.

**Grading highlights.** The default grader is local BM25 keyword scoring: free,
offline, no data leaves the box. For better relevance, opt into
[TypeSafe](https://typesafe.ai)'s Jev model, which answers "does this passage
answer the question?" for every passage and returns a calibrated probability
(passages under `SCRAPER_HIGHLIGHTS_MIN_SCORE`, default 0.5, are dropped, so an
off-topic page honestly returns `[]`):

```bash
export SCRAPER_HIGHLIGHTS_GRADER=jev
export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai
```

Jev is billed per input token (about $0.0004 for a 10k-token page). If the call
fails, that request falls back to BM25 (`"grader": "bm25"`); the scrape never fails
because of highlights.

## Cost-scoped residential egress

The dominant reason hard hosts wall you is the **datacenter IP**, not the
fingerprint. When a host repeatedly walls the local tiers (a 403/429 or a
bot-wall page, `SCRAPER_BOTWALL_EGRESS_AFTER` times) it's flagged `needs_egress`
and the cascade reruns through a **residential proxy** — but only for that host:

```bash
export SCRAPER_EGRESS_PROXY="http://user:pass@p.webshare.io:80"
```

The easy majority that already succeeds free at the datacenter IP stays direct,
so you never spend (often metered) residential bandwidth on it. Escalation tries
the cheap HTTP tiers through the proxy first (~0.2MB/page) before the heavier
browser tiers. [Webshare](https://www.webshare.io/)'s free plan includes ~1GB/mo
of residential bandwidth — enough for low-volume hard-host recovery at $0. Use
`SCRAPER_PROXY` instead to force *every* request through a proxy.

## Metrics & reporting

The engine derives all metrics from its own state files (no external store): the
botwall event log (one row per tier attempt, incl. the detected challenge vendor)
and the per-host DB (winning tier, per-vendor `challenge_counts`).

```bash
curl localhost:8799/metrics            # cost savings vs Firecrawl, coverage,
                                       # overall + per-tier latency, outcomes
curl localhost:8799/metrics/domains    # per-domain: error codes, challenges, latency
python -m switchback.flags             # periodic digest: domains stuck on Firecrawl,
                                       # escalated to egress, top challenged (cron-friendly)
```

Both endpoints accept `?minutes=N` to window the event-derived sections. The
**savings** figure compares engine spend (Firecrawl invocations only) against a
Firecrawl-everything baseline, charging the hard-page credit multiplier
(`BENCH_FIRECRAWL_HARD_MULT`) for URLs that needed a browser/residential tier or
hit a challenge — i.e. exactly the ones Firecrawl bills more for.

## Configuration

All configuration is via environment variables. The engine runs with missing
pieces: each tier imports its deps lazily and a missing one just counts as a tier
miss. Tracing no-ops if OTel isn't installed/configured.

<details>
<summary><b>Tracing (optional)</b></summary>

```bash
export OTEL_SERVICE_NAME=switchback
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
```
</details>

<details>
<summary><b>Env gates</b> — enable/disable tiers and integrations</summary>

- `SCRAPER_DISABLE_FIRECRAWL` — skip tier_7
- `FIRECRAWL_API_KEY` — enable tier_7
- `SCRAPER_DISABLE_CAMOUFOX` — turn off tier_5 (on by default; needs `pip install camoufox` + `camoufox fetch`)
- `SCRAPER_DISABLE_CLOUDSCRAPER` — turn off tier_3 (on by default). Recommended when you scrape URLs other people submit; see [Scraping URLs other people submit](#scraping-urls-other-people-submit)
- `SCRAPER_BROWSER_BLOCK_PRIVATE` — opt-in, off by default: tiers 4 and 5 refuse any request a page makes to a loopback, private, link-local (incl. cloud metadata) or CGNAT address
- `BU_CDP_URL` — enable tier_6 residential browser by pointing at a CDP endpoint
- `SCRAPER_PROXY` — route *all* tiers/URLs through a proxy
- `SCRAPER_EGRESS_PROXY` — route only walled hosts through a proxy (see [Cost-scoped residential egress](#cost-scoped-residential-egress))
- `SEARXNG_URL` — defaults to `http://localhost:8888`
- `SCRAPER_ARXIV_CONTACT` — optional email added to the User-Agent of arXiv API calls (arXiv asks clients to identify themselves; the default names only the project URL)
- `SCRAPER_STATE_DIR` — where the botwall DB/event log + session cache live
- `SCRAPER_COOKIES_FILE` — Netscape `cookies.txt` to scrape login-gated hosts (injected into the HTTP and browser tiers)
- `SCRAPER_CAPTCHA_PROVIDER` + `SCRAPER_CAPTCHA_API_KEY` — opt-in, off by default: wire a third-party solver (2captcha/capsolver/capmonster/anticaptcha/deathbycaptcha/9kw) into tier_3 for Turnstile/reCAPTCHA/hCaptcha on CF hosts. **Paid**, billed per solve by the provider.
</details>

<details>
<summary><b>Tunables</b> — budgets, timeouts, caches, backoff</summary>

- `SCRAPER_OUTPUT_FORMAT` — output shape: `markdown` (default) · `markdown_trimmed` · `html` · `html_selectors` (see [Output formats](#output-formats))
- `SCRAPER_DEADLINE_S` — per-URL budget (45s)
- `SCRAPER_FIRECRAWL_FALLBACK_AFTER_S` — after this many seconds on a URL, stop trying the local tiers and fall back to Firecrawl, so a hard host doesn't burn the whole deadline before the paid last resort gets a turn (25s; 0 = off)
- `SCRAPER_TIER_<N>_TIMEOUT_S` — per-tier timeout in seconds, `N` = 1–7 (tier_1 apis · tier_2 http · tier_3 cloudscraper · tier_4 browser · tier_5 camoufox · tier_6 residential · tier_7 firecrawl). Defaults: 15/15/**25**/15/**45**/**30**/15 (the three bold ones keep their prior budgets; everything else is 15s). The pre-0.5.0 `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` / `SCRAPER_CAMOUFOX_TIMEOUT_MS` / `SCRAPER_RESIDENTIAL_TIMEOUT_MS` are still honored when the new var is unset. Note: `tier_7` (Firecrawl) was previously unbounded — its 15s default now bounds the paid tier, so raise `SCRAPER_TIER_7_TIMEOUT_S` if slow hosts get cut off
- `SCRAPER_BROWSER_CONCURRENCY` — max simultaneous headless browsers (default 1)
- `SCRAPER_BOTWALL_URL_SKIP_COOLDOWN_H` — auto-skip re-test window (24h; 0 = never)
- `SCRAPER_BOTWALL_EGRESS_AFTER` — local-tier failures before a host escalates to the residential tier (default 2)
- `SCRAPER_SESSION_TTL_S` — cf_clearance reuse window (1800s)
- `SCRAPER_DISABLE_SESSION_CACHE` — turn off cf_clearance reuse
- `SCRAPER_CONTENT_TTL_S` — URL→result cache TTL (**0 = off**; set e.g. 86400 to skip re-scraping a page within a day)
- `SCRAPER_BACKOFF_BASE_MS` / `SCRAPER_BACKOFF_MAX_MS` — exponential backoff between tiers after a rate-limit/timeout (base 0 = off)
- `SCRAPER_TIER_RETRIES` — same-tier retries before falling through (default 0 = off; `N` → up to `1+N` tries per tier), with per-tier overrides `SCRAPER_TIER_RETRIES_<TIER>` (e.g. `SCRAPER_TIER_RETRIES_TIER_4=2`)
- `SCRAPER_TIER_RETRY_ON` — failure classes eligible for a same-tier retry (default `timeout,rate_limited,connection`; widen to include `botwall,http_block` behind a rotating residential proxy so each retry gets a fresh IP). Retries are bounded by `SCRAPER_DEADLINE_S`; enabling them on the paid Firecrawl tier bills per attempt
- `SCRAPER_LOGIN_HOOK` — `pkg.module:func` returning `{cookie: value}` for a host (see [Logged-in sessions](#logged-in-sessions))
- `SCRAPER_EXTRACTION_FILE` — per-domain extraction prefs JSON (default `config/extraction.json`)
- `SCRAPER_HIGHLIGHTS_GRADER` — `bm25` (default) or `jev` (needs `TYPESAFE_API_KEY`; see [Agent-ready extras](#agent-ready-extras))
- `SCRAPER_HIGHLIGHTS_TOP_K` — highlights returned per URL (5)
- `SCRAPER_HIGHLIGHTS_MIN_SCORE` — Jev probability below which a passage is dropped (0.5)
- `SCRAPER_JEV_MODEL` / `SCRAPER_JEV_TIMEOUT_S` — Jev model alias (`jev-latest`) and per-request timeout (20s)
- `SCRAPER_YOUTUBE` — opt-in: `1` returns YouTube transcripts from tier_1 (needs `switchback[youtube]`)
- `SCRAPER_DOCUMENTS` — opt-in: `1` converts .docx/.pptx/.xlsx/.xls/.epub to markdown in tier_2 (needs `switchback[documents]`)
- `SCRAPER_YOUTUBE_LANGS` — transcript languages to try, in order (`en`; e.g. `en,hi`). YouTube blocks most cloud IPs; when it does, the transcript is retried once through `SCRAPER_EGRESS_PROXY` if set
- `SCRAPER_TRACE_SESSION` — opt-in: capture a Playwright trace (screenshots + DOM + network) per browser-tier attempt, written to `state/traces/`
- `BENCH_FIRECRAWL_USD` / `BENCH_FIRECRAWL_HARD_MULT` — cost model for the savings report
</details>

### Logged-in sessions
Beyond a static `SCRAPER_COOKIES_FILE`, wire `SCRAPER_LOGIN_HOOK` to a callable
`func(host) -> {cookie: value}`. When an authenticated host trips a login/bot
wall, the engine calls the hook once, persists the returned cookies per host, and
overlays them on every tier (and future runs), then re-runs that URL on a fresh
budget. The hook owns the site-specific login mechanics; the engine stays generic.

### Session traces
With `SCRAPER_TRACE_SESSION=1`, each browser-tier attempt writes a Playwright
trace zip to `state/traces/`. Manage them over HTTP — `GET /traces` (list),
`GET /traces/{id}` (download), `DELETE /traces/{id}` — and open one with
`playwright show-trace <zip>`. Off by default (traces are MBs each).

### Output formats
Markdown is the default and is unchanged. Pick a different shape globally with
`SCRAPER_OUTPUT_FORMAT`, or per call:

```python
from switchback import scrape
scrape(["https://example.com/article"])                    # markdown (default)
scrape(["https://example.com/article"], fmt="html")        # raw HTML
scrape(["https://example.com/article"], fmt="markdown_trimmed")
```

```bash
switchback --format html_selectors https://example.com/article
curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"format":"html"}'
```

| format | what you get |
| --- | --- |
| `markdown` | whole-page markdown (boilerplate stripped + per-domain prefs) — **default** |
| `markdown_trimmed` | markdown with extra ad/nav/boilerplate lines removed |
| `html` | the raw HTML exactly as fetched, untouched |
| `html_selectors` | cleaned HTML (boilerplate strip + per-domain `drop`/`selector`), not converted |

The chosen content rides in the result's `markdown` field; in the CLI/server JSON
the key is `markdown` for markdown formats and `html` for html formats. The
API/PDF tiers (arXiv synth, PDF→text) have no HTML, so html formats fall back to
their text for those sources.

### Per-domain extraction
Markdown of the whole page is the default. To scope a site to its content node or
strip site-specific noise, declare prefs per host in `config/extraction.json`
(see [config/extraction.example.json](config/extraction.example.json)); every
tier's normalize step picks them up automatically.

## Roadmap

Not built yet:

- **Layout-aware markdown** — keep pricing grids and card layouts built from
  `<div>`s (not `<table>`s) as tables, using the browser tiers' rendered layout.
- **Scanned pages and images** — OCR / vision-model text for PDFs with no text
  layer and for images.
- **Audio and caption-less video** — speech-to-text transcription.
- **Hosted service** — managed browsers at high concurrency, stored results and
  shareable result links. switchback stays a library you run yourself.
- Smaller gaps: Next.js App Router flight data (`self.__next_f`) in `state`, and
  YouTube chapters in transcripts.

## Contributing

Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Start with the
cascade runner in `switchback/orchestrator.py`.

## Responsible use

This engine is for lawful data collection. You are responsible for respecting
each target site's Terms of Service, `robots.txt`, and rate limits, and for
having the right to access the content you fetch. The stealth / anti-bot tiers
(`cloudscraper`, `patchright`, `camoufox`) exist to handle legitimate access
friction (e.g. generic bot interstitials on public pages) — not to evade access
controls, paywalls, or authentication you aren't authorized to bypass. The
software is provided "as is", without warranty (see [LICENSE](LICENSE)).

## License

MIT — see [LICENSE](LICENSE). Third-party dependencies and their licenses are
listed in [NOTICE](NOTICE); all are permissive (MIT / BSD-3-Clause / Apache-2.0)
and compatible with this project's MIT license.
