Metadata-Version: 2.4
Name: flarecrawl
Version: 0.31.1
Summary: Cloudflare Browser Run CLI - Firecrawl-compatible, cost-efficient at scale
Project-URL: Homepage, https://github.com/0xDarkMatter/flarecrawl
Project-URL: Repository, https://github.com/0xDarkMatter/flarecrawl
Project-URL: Changelog, https://github.com/0xDarkMatter/flarecrawl/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/0xDarkMatter/flarecrawl/issues
Author: 0xDarkMatter
License-Expression: MIT
License-File: LICENSE
Keywords: anti-bot,browser-rendering,cdp,cloudflare,crawler,curl-cffi,data-extraction,firecrawl,headless-browser,markdown,playwright,screenshot,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: httpx[http2]>=0.25.0
Requires-Dist: keyring>=24.0
Requires-Dist: lxml>=6.1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: selectolax>=0.3.0
Requires-Dist: typer>=0.9.0
Requires-Dist: websockets>=16.0
Provides-Extra: cdp
Requires-Dist: websockets>=13.0; extra == 'cdp'
Provides-Extra: cookies
Requires-Dist: browser-cookie3>=0.20; extra == 'cookies'
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.3.0; extra == 'dev'
Provides-Extra: local-browser
Requires-Dist: playwright>=1.40.0; extra == 'local-browser'
Provides-Extra: mcp
Requires-Dist: mcp>=1.0.0; extra == 'mcp'
Provides-Extra: perf
Requires-Dist: aiolimiter>=1.1.0; extra == 'perf'
Requires-Dist: aiosqlite>=0.19.0; extra == 'perf'
Requires-Dist: opentelemetry-api>=1.20.0; extra == 'perf'
Requires-Dist: opentelemetry-instrumentation-httpx>=0.41b0; extra == 'perf'
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == 'perf'
Requires-Dist: orjson>=3.9.0; extra == 'perf'
Requires-Dist: protego>=0.3.0; extra == 'perf'
Requires-Dist: rbloom>=1.0.0; extra == 'perf'
Requires-Dist: uvloop>=0.19.0; (sys_platform != 'win32') and extra == 'perf'
Provides-Extra: recipes
Requires-Dist: pyyaml>=6.0; extra == 'recipes'
Provides-Extra: secure
Provides-Extra: stealth
Requires-Dist: curl-cffi>=0.7.0; extra == 'stealth'
Provides-Extra: videos
Requires-Dist: yt-dlp>=2026.6.9; extra == 'videos'
Description-Content-Type: text/markdown

# 🔥 Flarecrawl CLI

[![Forma](https://img.shields.io/badge/forma-experimental-orange.svg)](https://github.com/forma-tools/forma)
[![GitHub](https://img.shields.io/badge/github-0xDarkMatter%2Fflarecrawl-blue?logo=github)](https://github.com/0xDarkMatter/flarecrawl)
[![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Cloudflare](https://img.shields.io/badge/cloudflare-browser--run-orange?logo=cloudflare)](https://developers.cloudflare.com/browser-rendering/)

> Cloudflare Browser Run CLI — Firecrawl-compatible, cost-efficient at scale.

CLI that wraps Cloudflare's [Browser Run API](https://developers.cloudflare.com/browser-rendering/rest-api/) with the same command structure as Firecrawl. Supports scraping, crawling, URL discovery, screenshots, PDFs, and AI-powered data extraction — all running on Cloudflare's headless Chromium infrastructure. Now with direct Chrome DevTools Protocol (CDP) access for persistent browser sessions, real-time debugging, and authenticated scraping. Cost-efficient alternative for high-volume use cases (free 10 min/day, then $0.09/hr).

## Recent Updates

| Version | Date | Changes |
|---------|------|---------|
| **v0.31.1** | 2026-06-17 | **Bug-fix + security patch.** `crawl --wait -o` now flushes completed records on timeout instead of discarding them — across default, `--json`, and `--ndjson` output modes (#2). CDP commands (including the `p6` and `recipe` direct-construction paths) report a clear missing-`websockets` dependency *before* the auth check — no more misleading "Not authenticated" or raw traceback, and the `flarecrawl[cdp]` install hint survives Rich markup (#3). Security: `lxml>=6.1.0` (XXE via `iterparse`/`ETCompatXMLParser` on untrusted sitemap/RSS XML) and `yt-dlp>=2026.6.9`; lock refresh clears `cryptography`/`idna`/`starlette` advisories; added a grouped `.github/dependabot.yml`. Docs + skills brought current to v0.31.x. |
| **v0.31.0** | 2026-06-13 | **MCP server.** New `flarecrawl mcp` — a Model Context Protocol stdio server exposing the toolkit to agents (Claude Code, Cursor, …). 36 tools in a three-tier surface (5 T1 composite — `read_page`, `research_web`, `site_overview`, `extract_data`, `check_page_changes`; 17 T2 curated; 9 T3 `*_raw` full-fidelity) plus 5 orientation tools. `capabilities()` keystone returns the whole catalogue, permissions, and worked recipes in one call; content tools default to `--agent-safe`; `meta.blocked` bot-wall verdicts auto-suggest recovery (Akamai → stealth/p6, CF-1020 → terminal). `--read-only` mode excludes write/destructive tools; optional `flarecrawl[mcp]` extra; 11 declared coverage gaps. Internally, the 6,882-line `cli.py` split into a `cli/` command-family package — pure refactor, `--help` byte-identical, full suite green. |
| **v0.30.1** | 2026-06-02 | **Stack fingerprinting.** New `flarecrawl tech-detect URL` subcommand — identify what a site runs (CMS, framework, JS libs, CDN, analytics, payment, hospitality SaaS) over a single HTTP GET, zero CF browser time. Also surfaces as `--tech-detect` on `scrape`/`crawl`/`fetch` and `Client.detect_tech()`. ~7,500 upstream Wappalyzer fingerprints + 71-entry custom overlay (Craft CMS, Tailwind, Roam via `X-ROAM: HIT` header across 20 sites, plus AU hospitality/tourism SaaS — Mews, Cloudbeds, Bokun, Resy, Tock, TheFork, Triptease, Eventbrite, ATDW). Filters: `--min-confidence`, `--only-categories`, `--exclude-categories`. New `--cdp` flag routes through CF Browser Run + injects a `Runtime.evaluate` JS-globals probe (~5,500 property paths) — same machinery as `scrape --cdp --tech-detect`, unlocks the ~880 fingerprints that only fire via window globals. Batch via `-i FILE -w N`, streaming via `--ndjson`, local HTML via `--stdin`. 60-site bench, P/R/F1 = 1.000. Discoverable via `flarecrawl guide tech-detect`. |
| **v0.29.0** | 2026-05-17 | **Agent discoverability.** New `flarecrawl guide [topic]` — emits the packaged AGENTS.md (force-included in the wheel, works after a bare install with no repo) as on-tool orientation: `guide` overview + Quick Reference + topic index, `guide <topic>` one section with fuzzy + alias resolution (`hard-targets`, `json`, `errors`, `rules`, `auth`), `guide --list`. |
| **v0.28.0** | 2026-05-16 | **Hard-target field-report closure.** New `flarecrawl p6` mint→replay primitive (local-Chromium cookie-shell mint → curl_cffi TLS replay, proactive re-mint, cumulative egress cool-down, terminal CF-1020 fast-fail). Machine-readable `meta.blocked` (Akamai/Cloudflare/Imperva/DataDome/PerimeterX/CloudFront). `flarecrawl session inspect` offline jar freshness. Windows cp1252 output crash fixed. `--json` no longer overrides `--output`/backend. HAR flushed on selector timeout. Recipe capture pre-armed before nav; `schema_version: 1`. 801 tests |
| **v0.27.0** | 2026-05-16 | **Fetch routing + content-type coverage.** `fetch` no longer tracebacks on non-HTML types — new raw-text branch returns body verbatim without CF auth (fixes Google Maps KML, `text/xml`, `text/csv`, etc.). `--only-main-content` link-density gate (≥60%) prevents nav-soup from leaking into output. RFC 6839 `+json`/`+xml` suffix types (RSS, Atom, JSON-LD, GeoJSON, KML/vnd, SOAP) now route to text/JSON branch instead of binary download. E2E test corpus with local HTTP server. 1404 tests |
| **v0.26.0** | 2026-05-09 | **Headless evasion.** `humanize` module — synthesised mouse moves + scrolls + idle gaps, auto-on for headless `--browser local`. 8 additional `stealth_init.js` evasions (chrome.runtime.id, AudioContext noise, voices, Battery, WebGL2, MediaDevices, outerW/H). `SyncCDPPage.send()` for raw CDP from sync code. Idempotent CDP close. 730 tests |
| **v0.25.1** | 2026-05-09 | **Recipe + organize-by polish.** `for_each` and `capture_download` recipe steps. `--then-fetch-organize-by extension/content-type/thumbnail` for sub-categorised downloads. Windows cp1252 console fallback in `_output_text` |

For older releases, see [CHANGELOG.md](CHANGELOG.md).

## Why Flarecrawl?

| | Firecrawl | Flarecrawl |
|---|---|---|
| **Pricing** | Per-page credits ($99/100K) | Time-based (free 10 min/day, then $0.09/hr) |
| **Crossover** | Cheaper below ~8K pages/mo | **11x cheaper** at 100K pages/mo |
| **JS rendering** | Yes | Yes (headless Chromium on edge) |
| **Persistent browser sessions** | No | **Yes** (`--cdp --keep-alive`) |
| **Live browser debugging** | No | **Yes** (`--live-view` via Chrome DevTools) |
| **Interactive auth (OAuth/2FA/CAPTCHA)** | No | **Yes** (`--interactive` — login in DevTools) |
| **Form filling** | No | **Yes** (`interact` command with human-like timing) |
| **Session recordings** | No | **Yes** (`--record` — rrweb format, 30-day retention) |
| **WebMCP tool discovery** | No | **Yes** (`webmcp discover/call`) |
| **Stack fingerprinting** | No | **Yes** (`tech-detect` — single GET, ~7,500 upstream + 104 custom Wappalyzer fingerprints, `--only-categories` filtering) |
| **AI extraction** | Spark models | Workers AI (included) |
| **MCP server** | Yes (remote) | **Yes** — local stdio (`flarecrawl mcp`), 36 tools in a 3-tier surface (composite/curated/raw), `capabilities()` keystone, agent-safe by default |
| **Agent-safe sanitisation** | No | **Yes** (`--agent-safe` — 13 sanitisers, DeepMind taxonomy) |
| **Paywall bypass** | Limited | **Multi-strategy cascade** (`--paywall --stealth`) |
| **Content negotiation** | No | **Yes** (auto `Accept: text/markdown`, zero browser time) |
| **PDF generation** | No | **Yes** |
| **Web search** | Yes | Yes (Jina Search API) |
| **Cookie persistence** | No | **Yes** (`--save-cookies`/`--load-cookies`) |
| **Real HAR capture** | No | **Yes** (CDP `Network.enable`) |
| **Custom CDP backends** | No | **Yes** (`FLARECRAWL_CDP_ENDPOINT` — Oxylabs, Bright Data, local Chrome) |
| **Batch mode** | Limited | **Yes** (file input, NDJSON output, up to 50 workers) |
| **Direct HTTP spider** | Single-engine | **`flarecrawl spider`** — direct HTTP, no browser cost, 50-100 concurrent, built for millions of URLs |
| **Resume interrupted crawls** | No | **Yes** (`--resume JOB_ID`) — kill the process, come back hours later, pick up exactly where you left off |
| **Weekly-refresh mode** | No | **Yes** (`--refresh-days 7`) — only re-fetch pages that changed since last run; unchanged = zero browser cost |
| **Fair multi-domain scheduling** | No | **Yes** — round-robin so one fast site can't starve 53,999 others |
| **Adaptive politeness** | No | **Yes** (`--adaptive-delay`) — automatically slows on slow hosts, honours `Retry-After` |
| **Auto-retry with backoff** | Limited | **Yes** — per-URL retry budget, exp backoff, dead-letter inspection |
| **Circuit breaker per domain** | No | **Yes** — pauses domains after 10 consecutive failures |
| **robots.txt compliance** | Partial | **Yes** (protego, with per-host crawl-delay) |
| **URL canonicalisation** | No | **Yes** — strips `utm_*`, `gclid`, `fbclid`, etc. so tracking-param permutations don't duplicate crawls |
| **OpenTelemetry tracing** | No | **Yes** (`--tracing console\|json\|otlp`) |
| **Self-hosted** | Yes | Cloudflare infrastructure (or any CDP backend) |
| **Favicon extraction** | Via branding | Dedicated command |
| **Branding extraction** | Yes | Not yet |

Different pricing models suit different use cases. Flarecrawl's time-based pricing is particularly cost-efficient for high-volume crawls.

## Use Cases

### AI agent perception layer

Flarecrawl is often the first thing an AI agent sees when it reads the web.
`--agent-safe` sanitises scraped content against adversarial attacks
([AI Agent Traps](https://ssrn.com/abstract=6372438), Google DeepMind 2026)
before it enters an LLM context window or RAG pipeline. 13 sanitisers
defend against hidden text injection, prompt injection, and semantic
manipulation — with findings reported in structured JSON metadata.

For agents that speak [MCP](https://modelcontextprotocol.io/), `flarecrawl mcp`
exposes the whole toolkit directly — no shelling out — with `--agent-safe`
sanitisation on by default and bot-wall verdicts that suggest their own
recovery steps. See [MCP Server](#mcp-server).

### Scraping behind authentication

The hardest scraping problem isn't parsing HTML — it's getting past the login
page. `--interactive` opens a real browser in Chrome DevTools where you
complete OAuth flows, solve CAPTCHAs, or handle 2FA manually. Cookies are
auto-saved and reused on subsequent scrapes. No more extracting tokens from
browser DevTools and pasting them into headers.

### High-volume content extraction

Batch mode with up to 50 parallel workers (`--workers 50`) scrapes thousands
of pages concurrently. Combine with `--paywall` (multi-strategy extraction
cascade), `--stealth` (browser TLS fingerprinting), and `--agent-safe` for
production pipelines that handle the real web — paywalls, bot detection, and
adversarial content included.

### Site monitoring and change detection

`--diff` compares current content against the cached version. Combine with
`--har` for network-level visibility into what changed. `--record` saves
full session recordings for audit trails. Use `--keep-alive` for cost-efficient
repeated checks on the same site.

### Documentation and knowledge base building

`flarecrawl download` saves entire sites as markdown files. `flarecrawl discover`
finds every URL via sitemaps, RSS feeds, and link crawling. Content negotiation
(`Accept: text/markdown`) fetches server-rendered markdown directly from
compatible sites — zero browser time, higher quality output.

### API and structured data extraction

`flarecrawl extract` uses Cloudflare Workers AI for natural language data
extraction ("Get all product names and prices"). `flarecrawl schema` extracts
LD+JSON, OpenGraph, and Twitter Cards. `flarecrawl openapi` discovers and
downloads API specifications.

### Stack fingerprinting and competitive intel

`flarecrawl tech-detect` identifies what tech a site runs — CMS,
frameworks, JS libraries, CDN, analytics, payment processors, niche
SaaS — over a single HTTP GET with zero CF browser time. Useful for
market research (how many AU restaurant sites run SevenRooms vs
ResDiary?), competitive audits, and lead qualification (find every
site on Mews or Cloudbeds). Bundled corpus ships ~7,500 upstream
Wappalyzer fingerprints plus a 104-entry AU hospitality/tourism overlay
(Roam, ATDW, Localis, Simpleview, Mews, Cloudbeds, Bokun, Resy, Tock,
TheFork, Triptease, Eventbrite, …). `--cdp` mode routes through CF
Browser Run + injects a `Runtime.evaluate` JS-globals probe (~5,500
property paths) for SPAs that only declare their stack at runtime.

## CDP: Persistent Browser Control

Flarecrawl v0.14.1 adds direct Chrome DevTools Protocol (CDP) access via
Cloudflare's Browser Run WebSocket endpoint. This is an opt-in mode (`--cdp`)
that gives you a persistent, controllable browser session instead of
fire-and-forget REST calls.

**REST (default)** — each command spins up a fresh browser, navigates, extracts,
and destroys. Stateless and cheap. Good for 90% of scraping.

**CDP (`--cdp`)** — you get a live browser session that stays open. Navigate,
interact, inspect, then navigate again — all within the same browser context.
The browser remembers cookies, localStorage, and DOM state between operations.

### When to use CDP

| Scenario | Why CDP |
|----------|---------|
| **Scraping behind login** | `--interactive` opens DevTools, you log in manually (OAuth, 2FA, CAPTCHA), cookies auto-saved for future scrapes |
| **Debugging failed scrapes** | `--live-view` opens Chrome DevTools pointed at the remote browser — see console errors, inspect DOM, watch network |
| **Complex JS execution** | `--js-eval` via CDP uses real `Runtime.evaluate` — async/await works, returns typed objects, not the REST addScriptTag hack |
| **Multi-step workflows** | `--keep-alive 60` holds the browser open — scrape, screenshot, extract without re-navigating (1/3 the browser time cost) |
| **Network debugging** | `--har` captures real browser network traffic (redirects, blocked resources, timing waterfall), not just flarecrawl API metadata |
| **Audit trails** | `--record` saves rrweb session recordings — replay exactly what the browser did |

### CDP examples

```bash
# Interactive auth: login in DevTools, cookies saved, then scrape
flarecrawl scrape https://private-app.example.com --interactive --json

# Debug a scrape in real-time
flarecrawl scrape https://broken-site.com --live-view

# Proper JS execution (async, typed returns)
flarecrawl scrape https://spa.example.com --cdp --js-eval "await fetch('/api/data').then(r => r.json())"

# Persistent session: navigate once, do multiple things
flarecrawl scrape https://example.com --cdp --keep-alive 120 \
  --save-cookies session.json --har traffic.har --json

# Record a session for debugging later
flarecrawl scrape https://example.com --record --record-output debug.json

# Reuse saved cookies for authenticated scraping
flarecrawl scrape https://private-app.example.com --load-cookies session.json --cdp --json

# Manage CDP sessions
flarecrawl cdp sessions --json
flarecrawl cdp close
```

### Install CDP support

CDP requires the `websockets` package (optional dependency):

```bash
uv pip install websockets
# or with extras
uv pip install flarecrawl[cdp]
```

Without `websockets`, all non-CDP features work normally. The `--cdp` flag
will print a clear install instruction if the dependency is missing.

## Quick Start

```bash
git clone https://github.com/0xDarkMatter/flarecrawl.git
cd flarecrawl
uv tool install --editable .
flarecrawl auth login
flarecrawl scrape https://example.com
flarecrawl scrape https://example.com --json | jq '.data.content'
```

## Key Commands

| Command | Description |
|---------|-------------|
| `flarecrawl guide [topic]` | Agent orientation — when/why each command, JSON shapes, exit codes, footguns (start here) |
| `flarecrawl mcp [--read-only]` | Start the MCP stdio server — 36 tools for AI agents (Claude Code, Cursor, …). See [MCP Server](#mcp-server) |
| `flarecrawl scrape URL` | Scrape page to markdown (or html/links/images/summary) |
| `flarecrawl tech-detect URL` | Stack fingerprinting (Wappalyzer-style) — CMS, framework, JS libs, CDN, analytics, payment, niche SaaS. Single HTTP GET, zero CF browser time. `--cdp` for SPAs (CF Browser Run + JS-globals probe). |
| `flarecrawl crawl URL --wait --limit N` | Crawl site with async job system |
| `flarecrawl download URL --limit N` | Save site pages to disk as markdown/html |
| `flarecrawl extract PROMPT --urls URL` | AI-powered structured data extraction |
| `flarecrawl fetch URL -o file` | Content-type aware download — 4-branch routing: binary, JSON, raw text (XML/CSV/RSS/KML/YAML/…), HTML via CF |
| `flarecrawl openapi URL --probe` | Discover OpenAPI/Swagger specs on a site |
| `flarecrawl screenshot URL -o page.png` | Capture full or partial page screenshots |
| `flarecrawl pdf URL -o page.pdf` | Render page as PDF |
| `flarecrawl map URL` | List all links on a page |
| `flarecrawl discover URL` | Discover URLs via sitemaps, feeds, and links |
| `flarecrawl search QUERY` | Web search via Jina API (requires `JINA_API_KEY`) |
| `flarecrawl schema URL` | Extract LD+JSON, OpenGraph, Twitter Cards |
| `flarecrawl favicon URL` | Extract favicon/icon URLs |
| `flarecrawl p6 MINT_URL --jar jar.json --target URL` | Mint→replay anti-bot primitive (Akamai/Cloudflare/Imperva) |
| `flarecrawl session inspect @name` | Offline cookie-jar freshness verdict (exit ≠0 unless fresh) |
| `flarecrawl session list` | Manage saved cookie sessions |

## MCP Server

Flarecrawl exposes a [Model Context Protocol](https://modelcontextprotocol.io/) server
so AI agents (Claude Code, Cursor, etc.) can use flarecrawl tools directly.

```bash
uv pip install 'flarecrawl[mcp]'
```

**Wire up in Claude Code** (`.mcp.json`):

```json
{
  "mcpServers": {
    "flarecrawl": {
      "command": "flarecrawl",
      "args": ["mcp"]
    }
  }
}
```

**36 tools** across 4 groups:

| Group | Count | Description |
|-------|------:|-------------|
| Orientation | 5 | `capabilities`, `guide`, `diagnostics`, `permissions_check`, `schema_generate` |
| T1 Composite | 5 | `read_page`, `research_web`, `site_overview`, `extract_data`, `check_page_changes` |
| T2 Curated | 17 | `web_search`, `fetch_url`, `page_screenshot`, `tech_detect`, `crawl_start`, ... |
| T3 Raw | 9 | `scrape_raw`, `p6_raw`, `spider_raw`, ... (full CLI fidelity) |

Agents should call `capabilities()` first — one fetch returns the full tool
catalogue, permissions, coverage map, and worked recipes. Content tools default
to `--agent-safe` sanitisation (the MCP consumer is an LLM context window).
Read-only mode (`flarecrawl mcp --read-only`) excludes write/destructive tools.

**Coverage gaps** (CLI-only, declared in `capabilities()` with workarounds):
`videos`, `authcrawl`, `frontier`, `batch`, `auth login/logout`, `cache clear`,
`rules *`, `cdp *`, `webmcp *`, `session save/delete/show/validate`, and the
`--interactive`/`--live-view`/`--headed` flags.

## Install

```bash
git clone https://github.com/0xDarkMatter/flarecrawl.git
cd flarecrawl
uv tool install --editable .
```

### Secure credential storage (recommended)

By default, flarecrawl stores credentials in `~/.config/flarecrawl/config.json` (plaintext). For OS-native keyring storage:

```bash
uv tool install --editable . --with keyring
# or: uv pip install flarecrawl[secure]
```

With `keyring` installed, tokens move to:
- Windows: Credential Manager
- macOS: Keychain
- Linux: Secret Service / GNOME Keyring / KWallet

Existing plaintext credentials are migrated automatically on first use.

## Authentication

### 1. Create an API token

1. Go to https://dash.cloudflare.com/profile/api-tokens
2. Click **Create Token**
3. Select **Create Custom Token**
4. Configure:
   - **Token name:** `Flarecrawl` (or anything)
   - **Permissions:** Account → Browser Rendering → Edit
   - **Account Resources:** Include → your account
5. Click **Continue to summary** → **Create Token**
6. Copy the token (shown only once)

### 2. Find your Account ID

1. Go to https://dash.cloudflare.com
2. Click any domain (or the account overview)
3. Look in the right sidebar under **Account ID**
4. Copy the 32-character hex string

### 3. Authenticate

```bash
# Interactive — opens browser to Cloudflare dashboard for guided setup
flarecrawl auth login

# Non-interactive (CI/CD)
flarecrawl auth login --account-id YOUR_ACCOUNT_ID --token YOUR_TOKEN
```

### 4. Verify

```bash
flarecrawl auth status
flarecrawl --status   # Shows auth + pricing info
```

### Environment Variables (CI/CD)

```bash
export FLARECRAWL_ACCOUNT_ID="your-account-id"
export FLARECRAWL_API_TOKEN="your-api-token"
```

## Commands

### scrape — Fetch page content

```bash
# Default: markdown output to stdout
flarecrawl scrape https://example.com

# HTML output
flarecrawl scrape https://example.com --format html

# JSON envelope (for piping)
flarecrawl scrape https://example.com --json

# Multiple URLs (scraped concurrently)
flarecrawl scrape https://a.com https://b.com https://c.com --json

# Batch mode: file input with NDJSON output and configurable workers
flarecrawl scrape --batch urls.txt --workers 5

# From a file of URLs (backward-compatible alias for --batch)
flarecrawl scrape --input urls.txt --json

# With timing info
flarecrawl scrape https://example.com --timing

# Filter JSON fields
flarecrawl scrape https://example.com --json --fields url,content

# Extract links only
flarecrawl scrape https://example.com --format links --json

# Take screenshot via scrape
flarecrawl scrape https://example.com --screenshot -o page.png

# Wait for JS rendering (SPAs, Swagger UIs)
flarecrawl scrape https://example.com --js

# Bypass response cache
flarecrawl scrape https://example.com --no-cache

# Custom page load strategy
flarecrawl scrape https://example.com --wait-until networkidle2
```

**Formats:** `markdown` (default), `html`, `links`, `screenshot`, `json` (AI extraction)

### HTTP Basic Auth

All commands support `--auth user:password` for sites protected by HTTP Basic Auth:

```bash
flarecrawl scrape https://intranet.example.com --auth admin:secret
flarecrawl crawl https://intranet.example.com --wait --limit 50 --auth admin:secret
flarecrawl download https://intranet.example.com --limit 20 --auth user:pass
flarecrawl screenshot https://intranet.example.com --auth user:pass -o page.png
```

### Content filtering

```bash
# Strip nav/header/footer, keep main article content
flarecrawl scrape https://example.com --only-main-content

# Keep only specific CSS selectors
flarecrawl scrape https://example.com --include-tags "article,.post"

# Remove specific elements
flarecrawl scrape https://example.com --exclude-tags "nav,footer,.sidebar"
```

### Custom headers & mobile

```bash
# Custom HTTP headers
flarecrawl scrape https://example.com --headers "Accept-Language: fr"
flarecrawl scrape https://example.com --headers '{"X-Api-Key": "abc123"}'

# Custom User-Agent (identify your crawler, or try bypassing paywalls)
flarecrawl scrape https://example.com --user-agent "MyBot/1.0 (contact@example.com)"
flarecrawl scrape https://paywalled.example.com --user-agent "Googlebot/2.1"

# Mobile device emulation (iPhone 14 Pro viewport)
flarecrawl scrape https://example.com --mobile
flarecrawl screenshot https://example.com --mobile -o mobile.png
```

### Images, summaries & structured data

```bash
# Extract all image URLs from a page
flarecrawl scrape https://example.com --format images --json

# AI-powered content summary
flarecrawl scrape https://example.com --format summary --json

# Extract LD+JSON, OpenGraph, Twitter Cards
flarecrawl scrape https://example.com --format schema --json

# Dedicated schema command with type filtering
flarecrawl schema https://example.com --json
flarecrawl schema https://example.com --type ld-json --json
flarecrawl schema https://example.com --type opengraph --json
```

### Webhooks

```bash
# POST crawl results to a URL when complete
flarecrawl crawl https://example.com --wait --limit 10 --webhook https://hooks.example.com/crawl

# With custom headers (e.g. auth token)
flarecrawl crawl https://example.com --wait --limit 10 \
  --webhook https://hooks.example.com/crawl \
  --webhook-headers "Authorization: Bearer token123"
```

### CSS selector extraction & JS execution

```bash
# Extract content from specific CSS selector
flarecrawl scrape https://example.com --selector "main" --json

# Wait for a CSS element before capturing (SPAs, lazy-load)
flarecrawl scrape https://example.com --wait-for-selector ".loaded" --json

# Run JavaScript and return the result
flarecrawl scrape https://example.com --js-eval "document.title" --json
flarecrawl scrape https://example.com --js-eval "document.querySelectorAll('a').length" --json
```

### Stdin piping & HAR capture

```bash
# Process local HTML without API call
cat page.html | flarecrawl scrape --stdin --only-main-content
curl https://example.com | flarecrawl scrape --stdin --format schema --json

# Save request metadata to HAR file
flarecrawl scrape https://example.com --har requests.har --json

# Save raw HTML alongside output (for archival/reprocessing)
flarecrawl scrape https://example.com --backup-dir ./html-backup
flarecrawl download https://example.com --limit 20 --backup-dir ./html-backup
```

### URL discovery

```bash
# Discover all URLs via sitemaps, RSS feeds, and page links
flarecrawl discover https://example.com --json

# Sitemaps only
flarecrawl discover https://example.com --no-feed --no-links --json

# With limit
flarecrawl discover https://example.com --limit 100 --json
```

### Cookie banner removal, language, archive fallback

```bash
# Remove cookie banners, GDPR modals, newsletter popups
flarecrawl scrape https://eu-site.example.com --magic

# Request content in a specific language
flarecrawl scrape https://example.com --language de

# Fallback to Internet Archive if page returns 404
flarecrawl scrape https://dead-link.example.com --archived
```

### Enhanced content extraction

Multi-strategy content extraction that applies per-site optimisations before
falling back to standard browser rendering. Useful for sites that serve content
in SSR HTML but hide it with client-side JavaScript, or sites behind bot
detection that block default request patterns.

```bash
# Enhanced extraction with multi-strategy cascade
flarecrawl scrape https://example.com/article --paywall

# JSON output shows which extraction strategy worked
flarecrawl scrape https://example.com/article --paywall --json
# metadata.source indicates the strategy used

# Batch mode
flarecrawl scrape --batch urls.txt --paywall --workers 5

# No Cloudflare auth required (direct HTTP strategies)
# With auth: falls through to browser rendering if direct strategies fail
```

Strategies are tried in order of speed and cost. Direct HTTP strategies consume
zero browser time. When Cloudflare auth is configured, per-site header
optimisations are also applied to browser rendering requests as a fallback.

Optional dependency for improved compatibility: `pip install curl_cffi`

### Stealth mode

Opt-in browser TLS fingerprint impersonation for direct HTTP requests. Uses
`curl_cffi` to send requests with a real Safari/Chrome TLS handshake, avoiding
bot detection systems that fingerprint JA3/JA4 hashes.

```bash
# Stealth mode for content negotiation and direct fetches
flarecrawl scrape https://example.com --stealth

# Combine with paywall extraction
flarecrawl scrape https://example.com --paywall --stealth --json

# Batch mode
flarecrawl scrape --batch urls.txt --stealth --workers 5
```

Requires: `pip install curl_cffi`

Without `--stealth`, requests use Python's default TLS stack (httpx/OpenSSL)
which is identifiable by bot detection systems. With `--stealth`, the TLS
Client Hello is indistinguishable from a real Safari browser.

### P6 — mint → replay anti-bot primitive

For hard targets (Akamai / Cloudflare / Imperva / CloudFront) where one
stealth fetch isn't enough: a local Chromium navigates a mint URL so the
wall deposits its cookie shells, then `curl_cffi` replays the real targets
carrying that jar plus a genuine Chrome TLS handshake.

```bash
flarecrawl p6 https://site.com/ --jar ./jar.json \
  --target https://site.com/api/data --output-dir ./out

flarecrawl p6 https://site.com/ --jar ./jar.json \
  --targets-from urls.txt --json
```

Built-in: proactive re-mint when the jar goes stale, **cumulative**
exponential cool-down keyed on total re-mints (the Akamai egress-escalation
trap — a tight per-target re-mint loop keeps the whole egress IP flagged),
terminal fast-fail on Cloudflare 1020 (keyed on the egress, not the session
— minting can't help), and a resume journal next to the jar.

Check jar freshness offline anytime (no network, no burned request):

```bash
flarecrawl session inspect @site            # or a path: ./jar.json
flarecrawl session inspect ./jar.json --json
```

Verdict is `fresh` | `stale` | `expired` | `empty`; exit code is non-zero
unless `fresh`, so a connector can re-mint proactively instead of after a
block. Every `scrape` / `fetch --json` / `recipe` result also carries a
machine-readable `meta.blocked` `{blocked, vendor, kind, terminal, signal}`
so connectors stop string-matching their own bot-wall heuristics.

See [docs/HARD-TARGETS.md](docs/HARD-TARGETS.md) for the full escalation
ladder.

### Content cleanup

Markdown output is automatically cleaned of common ad placeholders, share
buttons, newsletter prompts, copyright lines, and navigation chrome.
No flag needed - applied to all markdown output by default.

For HTML output, use `--clean` to strip ad/promo DOM elements:

```bash
# Strip ad containers, social share widgets, cookie banners from HTML
flarecrawl scrape https://example.com --format html --clean --json
```

### Agent-safe mode

Sanitise scraped content against adversarial AI agent traps. See
[Agent Safety](#agent-safety) for full details.

```bash
flarecrawl scrape https://example.com --agent-safe --json
flarecrawl scrape https://example.com --agent-safe --paywall --stealth --json
flarecrawl crawl https://example.com --wait --limit 50 --agent-safe
```

### Web search

Search the web and optionally scrape each result. Uses Jina Search API.

```bash
# Search and get results as JSON
flarecrawl search "python web scraping" --json

# Search and scrape each result through the normal pipeline
flarecrawl search "topic" --scrape --limit 5 --paywall --json

# Pipeline: search -> extract URLs -> batch scrape
flarecrawl search "query" --json | jq -r '.data[].url' > urls.txt
flarecrawl scrape --batch urls.txt --paywall --stealth --workers 5
```

Requires `JINA_API_KEY` env var (free at https://jina.ai/api-key).
With `--scrape`, all scrape flags apply (`--paywall`, `--stealth`, `--only-main-content`, `--clean`).

### Proxy support

Route all HTTP requests through a proxy. Affects CLI-to-Cloudflare API
connections and direct HTTP (stealth, negotiate, paywall). Does NOT affect the
Cloudflare browser-to-target connection (CF's browser uses its own IP).

```bash
# HTTP proxy
flarecrawl scrape https://example.com --proxy http://proxy:8080

# SOCKS5 proxy
flarecrawl scrape https://example.com --proxy socks5://localhost:9050

# Set default via env var
export FLARECRAWL_PROXY=socks5://localhost:9050
flarecrawl scrape https://example.com   # uses env var proxy
```

Supported on all commands: `scrape`, `crawl`, `download`, `extract`, `search`.

### Per-site rules

Customisable per-site header rules for enhanced extraction. Default rules ship
with the package; user overrides are loaded from `~/.config/flarecrawl/rules.yaml`.

```bash
# List all loaded rules
flarecrawl rules list --json

# Show rules for a domain
flarecrawl rules show www.nytimes.com

# Add a custom rule
flarecrawl rules add example.com --referer https://www.google.com/ --cookie "auth=1"

# Show file paths
flarecrawl rules path
```

Rules use Ladder-compatible YAML format:

```yaml
- domain: www.example.com
  headers:
    Referer: "https://www.google.com/"
    Cookie: "session=abc"
- domains:
    - a.com
    - b.com
  headers:
    Referer: "https://t.co/x?amp=1"
```

### Markdown content negotiation

Sites on Cloudflare (Pro+) can serve markdown directly via `Accept: text/markdown`
content negotiation. Flarecrawl auto-detects this on every scrape — when a site
supports it, content is fetched via a simple HTTP GET instead of headless Chromium.

```bash
# Auto-detect (default) — tries content negotiation first
flarecrawl scrape https://blog.cloudflare.com/some-post

# Force browser rendering (skip negotiation)
flarecrawl scrape https://blog.cloudflare.com/some-post --no-negotiate

# JSON output shows the source
flarecrawl scrape https://blog.cloudflare.com/some-post --json
# metadata.source: "content-negotiation" (no browser) or "browser-rendering"
# metadata.markdownTokens: 1234 (from x-markdown-tokens header)
# metadata.contentSignal: {"ai-train": "yes", ...}
```

Benefits when negotiation succeeds:
- **Faster** — ~100-200ms vs 2-3s for browser rendering
- **Cheaper** — zero browser time consumed
- **Higher quality** — server-side conversion by the site owner
- **Domain cached** — one probe per domain, batch-friendly

### Change tracking

```bash
# Compare current content against cached version
flarecrawl scrape https://example.com --diff --json
```

### crawl — Crawl a website

```bash
# Start crawl and wait for results
flarecrawl crawl https://example.com --wait --limit 50

# With progress indicator
flarecrawl crawl https://example.com --wait --progress --limit 100

# Fire and forget (returns job ID)
flarecrawl crawl https://example.com --limit 50

# Check status of running crawl
flarecrawl crawl JOB_ID --status

# Get results from completed crawl
flarecrawl crawl JOB_ID

# Filter paths
flarecrawl crawl https://docs.example.com --wait --limit 200 \
  --include-paths "/docs,/api" --exclude-paths "/zh,/ja"

# Stream results as NDJSON (one record per line)
flarecrawl crawl JOB_ID --ndjson --fields url,markdown

# Skip JS rendering for faster crawl
flarecrawl crawl https://example.com --wait --limit 100 --no-render

# Follow subdomains
flarecrawl crawl https://example.com --wait --allow-subdomains

# Save to file
flarecrawl crawl https://example.com --wait --limit 50 -o results.json
```

### map — Discover URLs

```bash
# List all links on a page
flarecrawl map https://example.com

# JSON output
flarecrawl map https://example.com --json

# Include subdomains
flarecrawl map https://example.com --include-subdomains

# Limit results
flarecrawl map https://example.com --limit 20 --json
```

### download — Save site to disk

```bash
# Download as markdown files to .flarecrawl/
flarecrawl download https://docs.example.com --limit 50

# Download as HTML
flarecrawl download https://example.com --limit 20 --format html

# Filter paths
flarecrawl download https://docs.example.com --limit 100 \
  --include-paths "/docs" --exclude-paths "/changelog"

# Skip confirmation prompt
flarecrawl download https://example.com --limit 10 -y
```

Files are saved to `.flarecrawl/<domain>/` with sanitized filenames.

### extract — AI-powered data extraction

```bash
# Extract structured data with a natural language prompt
flarecrawl extract "Get all product names and prices" \
  --urls https://shop.example.com --json

# With JSON schema for structured output
flarecrawl extract "Extract article metadata" \
  --urls https://blog.example.com \
  --schema '{"type":"json_schema","schema":{"type":"object","properties":{"title":{"type":"string"},"date":{"type":"string"}}}}'

# Schema from file
flarecrawl extract "Extract data" --urls https://example.com --schema-file schema.json

# Multiple URLs
flarecrawl extract "Get page title" --urls https://a.com,https://b.com --json

# Batch mode: parallel extraction with NDJSON output
flarecrawl extract "Get page title" --batch urls.txt --workers 5
```

Uses Cloudflare Workers AI for extraction (no additional cost).

### screenshot — Capture web pages

```bash
# Default: saves to screenshot.png
flarecrawl screenshot https://example.com

# Custom output path
flarecrawl screenshot https://example.com -o hero.png

# Full page
flarecrawl screenshot https://example.com -o full.png --full-page

# Specific element
flarecrawl screenshot https://example.com --selector "main" -o main.png

# Custom viewport
flarecrawl screenshot https://example.com --width 1440 --height 900 -o wide.png

# JPEG format
flarecrawl screenshot https://example.com --format jpeg -o page.jpg

# JSON output (base64 encoded)
flarecrawl screenshot https://example.com --json
```

### pdf — Render pages as PDF

```bash
# Default: saves to page.pdf
flarecrawl pdf https://example.com

# Custom output
flarecrawl pdf https://example.com -o report.pdf

# Landscape A4
flarecrawl pdf https://example.com -o report.pdf --landscape --format a4

# JSON output (base64 encoded)
flarecrawl pdf https://example.com --json
```

### favicon — Extract favicon URL

```bash
# Get the best (largest) favicon
flarecrawl favicon https://example.com

# Show all found icons
flarecrawl favicon https://example.com --all

# JSON output
flarecrawl favicon https://example.com --json
flarecrawl favicon https://example.com --all --json
```

Renders the page, parses `<link rel="icon">`, `<link rel="apple-touch-icon">`, and related tags. Returns the largest icon found. Falls back to `/favicon.ico` if no `<link>` tags found.

### usage — Track browser time

```bash
# Show today's usage and history
flarecrawl usage

# JSON output
flarecrawl usage --json
```

Tracks the `X-Browser-Ms-Used` header from each API response locally. Free tier is 600,000ms (10 minutes) per day.

### auth — Authentication

```bash
flarecrawl auth login                    # Interactive
flarecrawl auth login --account-id ID --token TOKEN  # Non-interactive
flarecrawl auth status                   # Human-readable
flarecrawl auth status --json            # Machine-readable
flarecrawl auth logout                   # Clear credentials
```

### cache — Response cache management

```bash
flarecrawl cache status                  # Show entries, size, path
flarecrawl cache status --json           # Machine-readable
flarecrawl cache clear                   # Remove all cached responses
```

Responses are cached for 1 hour by default. Use `--no-cache` on any command to bypass.

### negotiate — Domain capability cache

```bash
flarecrawl negotiate status              # Show domains that support text/markdown
flarecrawl negotiate status --json       # Machine-readable
flarecrawl negotiate clear               # Reset domain cache
```

Tracks which domains respond to `Accept: text/markdown`. Positive results cached 7 days, negative 24 hours.

### Performance features

- **Response caching** — 1-hour TTL, saves redundant browser renders
- **Connection pooling** — persistent httpx session with HTTP/2 support
- **Resource rejection** — skips images/fonts/media/stylesheets for text extraction
- **JS rendering** — opt-in via `--js` flag (waits for networkidle0)

### Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `FLARECRAWL_ACCOUNT_ID` | — | Cloudflare account ID |
| `FLARECRAWL_API_TOKEN` | — | Cloudflare API token |
| `FLARECRAWL_CACHE_TTL` | 3600 | Cache TTL in seconds |
| `FLARECRAWL_MAX_RETRIES` | 3 | Max retry attempts |
| `FLARECRAWL_MAX_WORKERS` | 10 | Max parallel workers |
| `FLARECRAWL_TIMEOUT` | 120 | Request timeout in seconds |
| `FLARECRAWL_PROXY` | - | Default proxy URL (http/https/socks5) |
| `JINA_API_KEY` | - | Jina API key for `search` command |

## Firecrawl Compatibility

Flarecrawl follows the same command structure as the `firecrawl` CLI:

| firecrawl command | flarecrawl equivalent | Notes |
|---|---|---|
| `firecrawl scrape URL` | `flarecrawl scrape URL` | Same flags |
| `firecrawl scrape URL1 URL2` | `flarecrawl scrape URL1 URL2` | Concurrent |
| `firecrawl crawl URL --wait` | `flarecrawl crawl URL --wait` | Same flags |
| `firecrawl map URL` | `flarecrawl map URL` | Same flags |
| `firecrawl download URL` | `flarecrawl download URL` | Saves to `.flarecrawl/` |
| `firecrawl agent PROMPT` | `flarecrawl extract PROMPT` | Uses Workers AI |
| `firecrawl credit-usage` | `flarecrawl usage` | Local tracking |
| `firecrawl search QUERY` | `flarecrawl search QUERY` | Via Jina Search API |
| `firecrawl --status` | `flarecrawl --status` | Same |

### What's different

- **`search` command** — uses Jina Search API (requires `JINA_API_KEY`), with `--scrape` to scrape results
- **`extract` instead of `agent`** — same concept, different name to avoid confusion
- **`favicon` command** — bonus: extract favicon/apple-touch-icon URLs from pages
- **`schema` command** — bonus: extract LD+JSON, OpenGraph, Twitter Cards
- **PDF command** — bonus: Cloudflare supports PDF rendering, Firecrawl doesn't
- **Output directory** — `.flarecrawl/` instead of `.firecrawl/`
- **`--only-main-content`** — client-side via selectolax with link-density gating (Firecrawl uses server-side extraction). Link-density ≥ 60% on a `<main>`/`<article>` candidate causes fallthrough to the next selector, preventing nav-soup from leaking into output.

## Output Format

All `--json` output follows a consistent envelope:

```json
{
  "data": { ... },
  "meta": { "format": "markdown", "count": 1 }
}
```

Errors:

```json
{
  "error": { "code": "AUTH_REQUIRED", "message": "Not authenticated..." }
}
```

### Exit Codes

| Code | Meaning | Action |
|------|---------|--------|
| 0 | Success | Continue |
| 1 | Error | Check stderr for details |
| 2 | Auth required | Run `flarecrawl auth login` |
| 3 | Not found | Check job ID |
| 4 | Validation | Fix arguments |
| 5 | Forbidden | Check token permissions |
| 7 | Rate limited | Wait and retry |

## Batch & Parallel

Commands that operate on multiple URLs support batch mode with configurable parallelism.

### Batch input (`--batch`)

```bash
# Plain text file (one URL per line, # comments supported)
flarecrawl scrape --batch urls.txt --workers 5

# JSON array
flarecrawl scrape --batch urls.json

# NDJSON (one JSON object per line)
flarecrawl extract "Get title" --batch urls.ndjson --workers 3
```

Input format is auto-detected: starts with `[` → JSON array, starts with `{` → NDJSON, otherwise plain text.

### Batch output

Batch mode outputs **NDJSON** (one JSON object per line) with index correlation:

```json
{"index": 0, "status": "ok", "data": {"url": "https://a.com", "content": "...", "elapsed": 1.2}}
{"index": 1, "status": "error", "error": {"code": "TIMEOUT", "message": "Request timed out..."}}
{"index": 2, "status": "ok", "data": {"url": "https://c.com", "content": "...", "elapsed": 0.8}}
```

Results are sorted by index. Failed URLs don't stop processing — errors are reported inline.

### Workers

| Flag | Default | Max | Notes |
|------|---------|-----|-------|
| `--workers` / `-w` | 3 | 10 | Matches CF paid tier concurrency limit |

```bash
# Conservative (free tier: 3 concurrent browsers)
flarecrawl scrape --batch urls.txt --workers 3

# Aggressive (paid tier: up to 10)
flarecrawl scrape --batch urls.txt --workers 10
```

### Supported commands

| Command | `--batch` | `--workers` | Notes |
|---------|-----------|-------------|-------|
| `scrape` | Yes | Yes | Also supports `--input` (alias) |
| `extract` | Yes | Yes | Supplements `--urls` |
| `crawl` | No | No | Has its own async job system |
| `screenshot` | No | No | Single URL |
| `pdf` | No | No | Single URL |

## Advanced Usage

### Raw JSON body passthrough

Every command supports `--body` to send a raw JSON payload directly to the CF API, bypassing all flag processing:

```bash
flarecrawl scrape --body '{
  "url": "https://example.com",
  "gotoOptions": {"waitUntil": "networkidle0", "timeout": 60000},
  "rejectResourceTypes": ["image", "media"]
}' --json
```

### Piping and chaining

```bash
# Map URLs then batch scrape them
flarecrawl map https://docs.example.com --json | \
  jq -r '.data[]' | head -10 > urls.txt
flarecrawl scrape --batch urls.txt --workers 5

# Crawl and extract just the markdown
flarecrawl crawl https://example.com --wait --limit 10 --json | \
  jq -r '.data.records[] | select(.status=="completed") | .markdown'

# Stream crawl results through jq
flarecrawl crawl JOB_ID --ndjson --fields url,markdown | \
  jq -r '.url + "\t" + (.markdown | length | tostring)'
```

### Retry behavior

Requests automatically retry up to 3 times on HTTP 429 (rate limited), 502, and 503 errors with exponential backoff. Timeouts also trigger retries.

## Agent Safety

Flarecrawl is often the perception layer for AI agent workflows - scraped
content flows directly into LLM context windows and RAG systems. The
`--agent-safe` flag sanitises content against adversarial attacks before it
reaches the consuming agent.

### Background

This implementation is informed by
[AI Agent Traps](https://ssrn.com/abstract=6372438) (Franklin, Tomasev,
Jacobs, Leibo, Osindero - Google DeepMind, March 2026), which identifies
six categories of adversarial attacks against autonomous AI agents navigating
the web:

| Category | Target | Flarecrawl Defence |
|----------|--------|-------------------|
| Content Injection | Agent perception | **Defended** - hidden text, comments, attributes, unicode tricks, iframes, hidden inputs, CSS class hiding, meta tags |
| Semantic Manipulation | Agent reasoning | **Defended** - urgency clusters and authority claims flagged (not removed) |
| Behavioural Control | Agent actions | **Defended** - prompt injection patterns detected and stripped |
| Cognitive State | Agent memory/RAG | Upstream - content provenance via `metadata.source` |
| Systemic | Multi-agent networks | Upstream - outside scraping layer |
| Human-in-the-Loop | Human supervisor | Upstream - outside scraping layer |

### Usage

```bash
# Sanitise content for safe agent consumption
flarecrawl scrape https://example.com --agent-safe --json

# Combine with extraction flags
flarecrawl scrape https://example.com --agent-safe --paywall --stealth --json

# Batch mode
flarecrawl scrape --batch urls.txt --agent-safe --workers 5

# All commands support --agent-safe
flarecrawl crawl https://example.com --wait --limit 50 --agent-safe
flarecrawl download https://docs.example.com --limit 50 --agent-safe
flarecrawl extract "Get products" --urls https://shop.example.com --agent-safe --json

# Stdin pipe (sanitises local HTML)
cat page.html | flarecrawl scrape --stdin --agent-safe --json
```

### Two-phase sanitisation pipeline

Content passes through 13 sanitisers in two phases:

**Phase 1 (HTML)** - 8 sanitisers run on the DOM before markdown conversion:

| Sanitiser | Attack Vector | Action |
|-----------|--------------|--------|
| Hidden text | CSS hiding (`display:none`, `opacity:0`, off-screen, `clip-path`, `color:transparent`, etc.) | Removed |
| HTML comments | Instruction-like content in `<!-- -->` comments | Removed |
| Suspicious attributes | Injection patterns in `data-*`, `aria-label`, `alt`, `title` | Removed |
| Unicode tricks | Zero-width characters, bidirectional text overrides | Removed |
| Hidden iframes | `<iframe src="...">` with external content | Removed |
| Hidden inputs | `<input type="hidden" value="...">` with instruction patterns | Removed |
| CSS class hiding | `.d-none`, `.hidden`, `[hidden]` attribute with injection content | Removed |
| Meta injection | Custom `<meta>` tags with adversarial content (standard tags preserved) | Removed |

**Phase 2 (Text)** - 5 sanitisers run on markdown after conversion:

| Sanitiser | Attack Vector | Action |
|-----------|--------------|--------|
| Prompt injection | "ignore previous instructions", "SYSTEM:", role-play, delimiter injection | Removed |
| Semantic manipulation | Urgency clusters, authority claims | **Flagged only** |
| Homoglyph evasion | Cyrillic/Greek lookalike characters bypassing patterns | Removed |
| Markdown exfiltration | Image URLs with suspicious query params (IP addresses, exfil params) | **Flagged only** |
| HTML entity evasion | `&#73;gnore` entity-encoded injection patterns | Removed |

### False positive prevention

| Technique | How it works |
|-----------|-------------|
| Short-line bias | Prompt injection only strips lines <200 chars - articles *about* injection are preserved |
| Accessibility preservation | `.sr-only`/`.visually-hidden` text kept unless it matches injection patterns |
| Standard meta allowlist | `description`, `og:*`, `twitter:*`, `charset` meta tags never touched |
| Mixed-script detection | Homoglyph normalisation only on lines with both Latin and Cyrillic/Greek chars |
| Threshold gates | Hidden elements need 20+ chars; hidden inputs need 50+ chars + pattern match |
| Flag vs remove | Semantic manipulation and exfiltration are flagged, never removed |

### JSON output

When `--json` is used, `metadata.agentSafety` reports what was detected:

```json
{
  "metadata": {
    "agentSafety": {
      "sanitised": true,
      "findings": [
        {"category": "content_injection", "severity": "high",
         "description": "Hidden text via CSS (2 elements)", "action": "removed", "count": 2},
        {"category": "prompt_injection", "severity": "high",
         "description": "Prompt injection patterns removed (1 lines)", "action": "removed", "count": 1},
        {"category": "semantic_manipulation", "severity": "medium",
         "description": "Urgency language clusters (1 lines)", "action": "flagged", "count": 1}
      ],
      "stats": {
        "removed": 3,
        "flagged": 1,
        "byCategory": {
          "content_injection": 2,
          "prompt_injection": 1,
          "semantic_manipulation": 1
        }
      }
    }
  }
}
```

### Extensibility

New attack vectors are added by writing a function and decorating it:

```python
from flarecrawl.sanitise import register_html, register_text, Finding

@register_html
def sanitise_new_vector(soup):
    """HTML-level sanitiser - mutates soup, returns findings."""
    # ... detection and removal logic ...
    return [Finding(category="content_injection", severity="high",
                    description="New vector (N elements)", action="removed", count=n)]

@register_text
def sanitise_new_text_vector(text):
    """Text-level sanitiser - returns (cleaned_text, findings)."""
    # ... detection logic ...
    return cleaned_text, findings
```

Add a corpus fixture to `tests/corpus/attacks/` and the parametrised test
suite picks it up automatically.

### Test corpus

The `tests/corpus/` directory contains 61 fixture files that serve as both
test data and a living catalogue of known attack vectors:

- **`attacks/`** (45 files) - adversarial fixtures across 13 categories
- **`benign/`** (16 files) - false-positive traps: security articles, responsive
  CSS, ARIA accessibility, admin UIs, medical urgency, i18n content, code
  tutorials, system docs, journalism

Parametrised tests validate that every attack file produces findings and
every benign file produces zero removals.

## Pricing Details

| Tier | Browser Time | Concurrent Browsers |
|------|-------------|-------------------|
| **Free** | 10 min/day | Up to 120 (default) |
| **Paid** | 10 hr/month included, then $0.09/hr | Up to 120 (default), higher on request |

Browser time is shared between REST API calls and Workers bindings. Track your usage with `flarecrawl usage`.

## Project Structure

```
flarecrawl/
├── pyproject.toml              # Package config
├── AGENTS.md                   # AI agent context (served by `flarecrawl guide`)
├── README.md                   # This file
├── src/flarecrawl/
│   # CLI (Typer) — one module per command family, assembled in cli/__init__.py
│   ├── cli/                    # __init__ builds `app`; scrape, crawl, fetch, media,
│   │                           #   techdetect, search, recipe, sessions, … + _common helpers
│   # MCP server (optional [mcp] extra) — exposes the CLI as an agent tool surface
│   ├── mcp_serve.py            # stdio transport entry point (only file importing `mcp`)
│   ├── mcp_tools/              # 36-tool registry: orientation / composite / curated / raw,
│   │                           #   in-process CLI exec, §30 error envelopes, capabilities()
│   # Core
│   ├── client.py               # CF Browser Run REST API client (httpx pooling, HTTP/2)
│   ├── config.py               # Usage tracking, env-var config, session storage
│   ├── credentials.py          # Credential chain: OS keyring / .env / legacy config
│   ├── guide.py                # `flarecrawl guide` — serves the packaged AGENTS.md
│   # Fetching & rendering
│   ├── fetch.py                # Content-type aware download — 4-branch routing: binary / JSON / raw text / HTML→CF
│   ├── negotiate.py            # Markdown content negotiation (Accept: text/markdown)
│   ├── cdp.py                  # CDP WebSocket client (persistent sessions, DevTools Protocol)
│   ├── local_browser.py        # Local Playwright Chromium backend (--browser local)
│   ├── humanize.py             # Synthesised mouse/scroll/idle input for headless evasion
│   ├── recipe.py               # YAML multi-step browser flows with resume journal
│   # Anti-bot
│   ├── stealth.py              # Stealth HTTP (curl_cffi TLS impersonation)
│   ├── paywall.py              # Paywall bypass cascade (SSR, Referer, Wayback, Jina)
│   ├── p6.py                   # Mint→replay primitive (local Chromium mint, curl_cffi replay)
│   ├── jarhealth.py            # Offline cookie-jar freshness verdicts (session inspect)
│   ├── blockdetect.py          # Machine-readable bot-wall verdicts (meta.blocked)
│   ├── cookies.py              # Cookie loading (Puppeteer/Netscape/Chrome), conversion, validation
│   ├── browser_cookies.py      # Grab cookies from local Chrome/Firefox (--browser-cookies)
│   ├── fingerprint.py          # Browser fingerprint helpers (stealth_init.js wiring)
│   # Crawling at scale
│   ├── authcrawl.py            # Authenticated BFS crawler (Frontier v2, resume, refresh-days)
│   ├── frontier_v2.py          # Persistent crawl frontier (SQLite, fair scheduling, circuit breakers)
│   ├── sitemap.py              # Sitemap/RSS/Atom URL discovery
│   ├── robots.py               # robots.txt compliance (protego, per-host crawl-delay)
│   ├── canon.py                # URL canonicalisation (strips utm_*, gclid, fbclid, …)
│   ├── ratelimit.py            # Adaptive politeness, Retry-After honouring
│   ├── dead_letter.py          # Per-URL retry budget + dead-letter inspection
│   ├── journal.py              # Crawl/recipe resume journals
│   ├── delta.py                # Change tracking (--diff)
│   ├── batch.py                # Batch processing (parse + parallel workers)
│   # Extraction & analysis
│   ├── extract.py              # HTML extraction (main content, images, schema, tags)
│   ├── sanitise.py             # Agent-safety sanitisation (hidden text, injection, manipulation)
│   ├── wappalyzer.py           # tech-detect engine (fingerprint matching + custom overlay)
│   ├── wappalyzer_data/        # ~7,500 upstream fingerprints + custom_fingerprints.json overlay
│   ├── design.py               # Design system extraction (tokens, coherence scoring, formatting)
│   ├── openapi.py              # OpenAPI/Swagger spec discovery and validation
│   ├── videos.py               # Video URL discovery (+ yt-dlp registry resolution)
│   ├── search.py               # Web search via Jina Search API
│   # Infrastructure
│   ├── cache.py                # File-based response cache
│   ├── rules.py                # Per-site YAML rulesets (load, merge, cache)
│   ├── default_rules.yaml      # Shipped per-site header rules
│   ├── telemetry.py            # OpenTelemetry tracing (--tracing)
│   ├── _http.py / _validate.py / _bloom_io.py / json_compat.py / shutdown.py
│   └── assets/                 # Vendored stealth_init.js
└── tests/
    ├── conftest.py             # Fixtures (incl. keyring isolation — tests never touch real creds)
    ├── test_*.py               # 60+ unit test modules, 1,500+ tests, no network
    ├── test_mcp_*.py           # MCP server tests (registry, capabilities, coverage audit, exec, stdio)
    ├── corpus.py               # Live feature corpus (80 tests x 8 sites, needs CF auth)
    ├── corpus/                 # Agent-safety fixture corpus (61 files)
    │   ├── attacks/            # Adversarial fixtures (45 files, 13 categories)
    │   └── benign/             # False-positive traps (16 files)
    └── validate_custom_overlay.py, bench*.py, compare_w3techs.py  # operator scripts (not pytest targets)
```

## Development

```bash
# Install dev dependencies
uv tool install --editable . --with pytest --with ruff

# Install CDP support (optional — needed for --cdp, interact, webmcp)
uv pip install websockets

# Install the MCP server (optional — needed for `flarecrawl mcp`)
uv tool install --editable '.[mcp]'

# Run unit tests (1,500+ tests, no network; MCP tests need the [mcp] extra)
pytest tests/ -v

# Run live tests against public sites (needs CF auth)
PYTHONPATH=src pytest tests/live/ -v -m live

# Run CDP live tests (needs CF auth + websockets)
PYTHONPATH=src pytest tests/live/ -v -m cdp

# Lint
ruff check src/

# Reinstall after changes
uv tool install --editable .
```

## Forma Protocol

This tool follows the [Forma Protocol](https://github.com/forma-tools/forma).

## License

MIT
