Metadata-Version: 2.5
Name: escalator
Version: 0.1.0
Summary: Self-hosted scraping gateway. Climb the cheapest rung that works.
Project-URL: Homepage, https://github.com/ruslanstarikov/escalator
Project-URL: Repository, https://github.com/ruslanstarikov/escalator
Project-URL: Issues, https://github.com/ruslanstarikov/escalator/issues
Project-URL: Changelog, https://github.com/ruslanstarikov/escalator/releases
License: Unlicense
Keywords: cdp,chrome,markdown,mcp,proxy,scraper,scraping,web-scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: The Unlicense (Unlicense)
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Utilities
Requires-Python: <3.14,>=3.11
Requires-Dist: curl-cffi>=0.7
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.27
Requires-Dist: lxml-html-clean>=0.4
Requires-Dist: nodriver>=0.38
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic-settings>=2.6
Requires-Dist: pydantic>=2.9
Requires-Dist: readability-lxml>=0.8.1
Requires-Dist: rich>=13.0
Requires-Dist: trafilatura>=1.12
Requires-Dist: uvicorn[standard]>=0.32
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: fastmcp>=2.0; extra == 'mcp'
Description-Content-Type: text/markdown

# escalator

Give it a URL, get clean Markdown. It climbs the cheapest rung that works — a
plain HTTP fetch, the same fetch through a residential proxy, then a stealth
browser — and stops at the first one that comes back with real content.

<!-- TODO: terminal cast. `asciinema rec`, then embed the SVG here:
     a fresh machine running `uvx escalator init` and then `escalator scrape`,
     start to finish, in under two minutes. -->

```
$ escalator scrape https://en.wikipedia.org/wiki/Web_scraping | head -3
# Web scraping

**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](...)
```

## Quickstart

```bash
# 1. install uv (https://docs.astral.sh/uv/getting-started/installation/)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. configure this machine -- finds your browser, or fetches one
uvx escalator init

# 3. use it
uvx escalator scrape https://en.wikipedia.org/wiki/Web_scraping
```

That is the whole on-ramp. `init` writes one config file, and nothing has to be
hand-edited before the first run. If anything looks wrong: `escalator doctor`.

> **On a minimal Linux** (a bare container, a fresh VPS) Chrome needs system
> libraries that a desktop already has. escalator does not install them for you —
> it prints the exact `apt-get` line and `escalator doctor` repeats it. One
> command, once:
>
> ```bash
> sudo apt-get update && sudo apt-get install -y \
>   libnss3 libnspr4 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libdrm2 \
>   libxkbcommon0 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxrandr2 \
>   libgbm1 libglib2.0-0t64 libpango-1.0-0 libcairo2 libasound2t64 \
>   libatspi2.0-0t64 libxcb1 libdbus-1-3 libexpat1
> ```

**No batteries, by design.** escalator ships unconfigured — no bundled browser, no
bundled proxies, no telemetry, nothing written outside its own directories. `init`
exists so that configuring it is a two-minute conversation instead of a README
scavenger hunt.

## Installation

| how | command | when |
|---|---|---|
| uv (no install) | `uvx escalator init` | trying it out |
| uv (persistent) | `uv tool install escalator` | you want it on PATH |
| pipx | `pipx install escalator` | you already use pipx |
| pip | `pip install escalator` | inside an existing venv |
| Docker | `docker run --rm ghcr.io/ruslanstarikov/escalator doctor` | servers |

Everything needed for the core flow is in the default install, including the
browser rung — it brings no browser with it, which is what `escalator browser
install` is for. One extra exists: `escalator[mcp]` adds the MCP face, and the
Docker image includes it.

### Docker

```bash
docker run --rm \
  -e ESCALATOR_SERVER_API_KEYS=your-key \
  -p 8000:8000 -v ./data:/data \
  ghcr.io/ruslanstarikov/escalator serve
```

The image carries a pinned browser and is configured entirely by environment
variable — see [`docker-compose.example.yml`](docker-compose.example.yml) for the
proxy wiring. It runs as uid 1000, so a bind-mounted `./data` must be writable by
it.

## Commands

```
escalator init [--yes]     configure this machine; --yes for scripts
escalator doctor [--json]  check everything, one fix per failure
escalator browser list     every browser found, and which one wins
escalator browser install  download Chrome for Testing into the data dir
escalator scrape URL       one page to stdout, so it pipes
escalator serve            the HTTP API and the MCP face
escalator --version        tool, python, platform
```

## The ladder

```
policy      robots.txt (cached) + rate limit  → may short-circuit (skip/deny/wait)
http        curl_cffi, impersonate=chrome     → ~100ms; clears undefended sites
http_proxy  same, routed via residential IP   → beats datacenter-IP bans
browser     nodriver, headless Chrome         → JS/SPA + Cloudflare-class defenses
                    │
                    └─ walled on the last rung? → status="challenged". Surrender.
```

Two things make this more than a `for` loop:

**200 OK is not success.** A rung that returns HTTP 200 carrying a Cloudflare
interstitial has not succeeded. `core/detect.py` classifies every response *after*
extraction — `content`, `thin`, or `blocked` — and only `content` counts. Without
that the ladder would never escalate, and the cache would learn "http works" for a
domain that serves junk forever.

**The cache forgets.** A learned start-rung that only ratcheted upward would drift
every domain toward browser+proxy and quietly inflate your proxy bill. Entries
carry `learned_at`; past `ladder.tier_cache_ttl_hours` a domain retries one rung
cheaper.

See [DESIGN.md](DESIGN.md) for why it is shaped this way — and for what it
deliberately refuses to do.

## Configuration

One file, written by `init`, at the platform config dir
(`~/.config/escalator/config.toml` on Linux, `~/Library/Application
Support/escalator/config.toml` on macOS). Override the location with `--config`.

Precedence, everywhere:

```
CLI flag  >  environment  >  config.toml  >  default
```

Every key has an environment variable, which is how the Docker image is
configured without a file existing at all:

| config key | env var | default | what it does |
|---|---|---|---|
| `browser.path` | `ESCALATOR_BROWSER_PATH` | — | Absolute path to a Chrome/Chromium binary. Empty = find one. |
| `browser.headless` | `ESCALATOR_BROWSER_HEADLESS` | `true` | false needs a display (or Xvfb), and is harder to detect. |
| `browser.via_proxy` | `ESCALATOR_BROWSER_VIA_PROXY` | `true` | Route renders through the proxy too. Costs bandwidth. |
| `browser.max_concurrent` | `ESCALATOR_BROWSER_MAX_CONCURRENT` | `4` | Chrome is the RAM ceiling on a small box. |
| `browser.timeout_ms` | `ESCALATOR_BROWSER_TIMEOUT_MS` | `30000` | Per-fetch deadline for the browser rung. |
| `proxy.enabled` | `ESCALATOR_PROXY_ENABLED` | `false` | The switch. Everything below is ignored while this is false. |
| `proxy.url` | `ESCALATOR_PROXY_URL` | — | http://user:pass@host:port, or socks5://... |
| `proxy.list` | `ESCALATOR_PROXY_LIST` | — | Several exits, used round-robin. Combined with url. |
| `http.timeout_ms` | `ESCALATOR_HTTP_TIMEOUT_MS` | `10000` | Per-fetch deadline for the two http rungs. |
| `ladder.min_content_chars` | `ESCALATOR_LADDER_MIN_CONTENT_CHARS` | `200` | Below this many extracted chars a page is 'thin' and the ladder climbs. |
| `ladder.tier_cache_ttl_hours` | `ESCALATOR_LADDER_TIER_CACHE_TTL_HOURS` | `72` | How long a learned rung survives before decaying one step cheaper. |
| `politeness.respect_robots` | `ESCALATOR_POLITENESS_RESPECT_ROBOTS` | `true` | Your box, your call. |
| `politeness.rate_limit_rps` | `ESCALATOR_POLITENESS_RATE_LIMIT_RPS` | `1.0` | Per-domain. 0 disables the gap entirely. |
| `politeness.user_agent` | `ESCALATOR_POLITENESS_USER_AGENT` | a Chrome UA | Used for robots.txt matching. |
| `server.api_keys` | `ESCALATOR_SERVER_API_KEYS` | — | Bearer keys for `escalator serve`. This list IS the truth: removing one revokes it. |
| `server.host` | `ESCALATOR_SERVER_HOST` | `127.0.0.1` | 127.0.0.1 keeps it off the local network. Containers want 0.0.0.0. |
| `server.port` | `ESCALATOR_SERVER_PORT` | `8000` | Port for `escalator serve`. |
| `storage.data_dir` | `ESCALATOR_STORAGE_DATA_DIR` | — | Database and managed browsers. Empty = the platform default below. |
| `storage.request_log_limit` | `ESCALATOR_STORAGE_REQUEST_LOG_LIMIT` | `5000` | Rows kept in request_log; trimmed on insert. |

Data — the SQLite database and any downloaded browser — lives in the platform data
dir, overridable with `ESCALATOR_STORAGE_DATA_DIR`. Nothing is ever written
outside it.

### Where the browser comes from

`escalator browser list` shows the search, in order:

1. an explicit path — `--browser-path`, then `ESCALATOR_BROWSER_PATH`, then
   `browser.path`. If it is set and wrong, that is an error naming the path, never
   a silent fall-through.
2. browsers installed on this machine: real Google Chrome first, then Chromium,
   then Edge and Brave.
3. a browser `escalator browser install` downloaded earlier.

If none of those find anything, you get an error naming the two commands that fix
it. Resolution never downloads on its own — a server request or a cron job should
not install software as a side effect.

## Using the server

```bash
escalator serve   # 127.0.0.1:8000 by default
```

```
POST /scrape   {url, markdown?, min_tier?, max_tier?, timeout_ms?}  -> FetchResult
GET  /healthz                                                       -> {status, version}
```

Authenticate with `Authorization: Bearer <key>`, where the key is one of
`server.api_keys`. That list *is* the truth: remove a key and it is revoked on the
next start. There is no endpoint to mint one.

A wall comes back as `200 OK` with `{"status": "challenged"}`, not as an HTTP
error. That is deliberate: an agent on the other end can react to it. Retrying in a
loop will not help — escalator does not solve CAPTCHAs, by design.

With the `[mcp]` extra the same ladder is exposed at `/mcp` as one tool,
`scrape_url(url, force_browser=False)`.

## Troubleshooting

Start here:

```bash
escalator doctor
```

It checks Python, the config file, the data directory, browser resolution, an
actual headless launch, and — if a proxy is configured — one real request through
it, reporting the egress IP and country with the password masked. Every ❌ comes
with the one line that fixes it, and the exit code is non-zero if anything failed,
so scripts can use it too. `escalator doctor --json` for machines.

| symptom | what it usually is |
|---|---|
| `no Chrome-family browser found` | `escalator browser install` |
| `error while loading shared libraries` on Linux | doctor names the package to install |
| everything returns `challenged` | you need a residential proxy: `escalator init` |
| `/scrape` returns 401 | no key configured, or it was removed from `server.api_keys` |
| slow first browser fetch | Chrome cold start; escalator retries the launch once |

If it still will not work, paste the whole `escalator doctor` output into an
issue — that is what the last line of it asks for, and it is the fastest route to
an answer.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md). In short: `uv sync`, `uv run pytest`.

## License

Released into the public domain — see [UNLICENSE](UNLICENSE). No warranty, no
attribution required, do what you like with it.
