Turn any URL into clean markdown.

One-click install for AI agents

scrapefold ships a CLI (scrapefold) and an MCP server (scrapefold-mcp) with four tools — scrape_url, crawl_site, list_engines, classify_url. All four definitions cost your agent ≈750 tokens (budget-tested). Need one fact from a long page? scrape_url(focus="query") returns only the relevant blocks. Failures come back structured — never error-page HTML posing as content. One command registers it in your client:

$ pip install "scrapefold[mcp]"
$ scrapefold install claude   # Claude Code
$ scrapefold install codex    # Codex CLI
$ scrapefold install cursor   # Cursor
$ scrapefold install vscode   # VS Code

Any other client: paste the config JSON above into its MCP settings. Full agent instructions live at scrapefold.com/install.md — fetchable by any agent.

Why Scrapefold?

Every scraping vendor has trade-offs. Scrapefold lets you switch between them with one line — and escalates from free local engines to paid APIs only as far as a site forces it.

Search with Enrichfold. Fetch with Scrapefold.

scrapefold.search(query) calls Enrichfold, which merges ranked results from Exa, Parallel, You.com, Tavily, Linkup, Seltz, TinyFish, Nimble, Browserbase, Serper, and DuckDuckGo. Then scrapefold.scrape(url) retrieves a page through the URL engine you choose.

from scrapefold import SearchOptions, ScrapeOptions, search, scrape

hits = await search("example.com company", SearchOptions(engines=("parallel", "tavily")))
page = await scrape(hits[0].url, ScrapeOptions(engines=("firecrawl",)))

38 URL engines, one interface

Local engines are free and fast; SaaS engines add premium proxies and stealth. The router picks the cheapest tier that works. Ratings: ★★★ excellent · ★★☆ good · ★☆☆ basic.

How to choose

Or skip the decision entirely — call scrape(url) and let the router pick.

• Static blog or documentation siterequests — zero deps, sub-second
• JS-rendered SPA, no anti-botscrapling_fast (free) or Jina Reader (free tier)
• Cloudflare / Datadome / PerimeterXscrapling_stealth (free) → Firecrawl / ScrapingBee (paid)
• Site that emits clean markdown via APIJina Reader or Keenable — direct markdown, no parsing
• Visual layouts, tables, charts, or screenshotsPixelRAG — local pixelshot tiles + reader markdown / JSON
• LinkedIn / niche socialExa public people/company search + Apify (LinkedIn) actor fallback
• Blog posts by keyword, brand mentionsTwingly — blog-search index; query ("datatera lang:en") or blog URL
• Structured fields straight from a pageScraperAPI — AI Parser fills the json slot
• IP-geofenced targetsOxylabs — residential pool + geo_location
• Page is gone (404) or paywalled nowwayback — archive.org snapshot, marked source=archive.org
• Need an MCP server for AI agentsscrapefold-mcp — built-in; register with scrapefold install claude

Quickstart

Install one extra per vendor, or scrapefold[all] for everything.

import asyncio
from scrapefold import scrape, crawl_site, ScrapeOptions

async def main():
    # Single URL, auto-engine — router picks the cheapest tier that works
    result = await scrape("https://example.com")
    print(result.markdown)        # always populated
    print(result.engine)          # which engine actually fetched it

    # Cloudflare-protected site — same call, router auto-escalates
    result = await scrape(
        "https://protected.example.com",
        opts=ScrapeOptions(render_js=True, stealth=True),
    )

    # Whole-site crawl with disk cache
    crawl = await crawl_site(
        "https://docs.example.com",
        opts=ScrapeOptions(max_pages=50, max_depth=3),
        output="site.md",
    )

asyncio.run(main())

# CLI
$ scrapefold scrape https://example.com
$ scrapefold scrape https://example.com --focus "pricing"  # only relevant blocks — saves tokens
$ scrapefold crawl https://docs.example.com --max-pages 50 --output site.md
$ scrapefold list-engines
$ scrapefold doctor            # health check: engines, MCP extra
$ scrapefold update --check    # self-update via PyPI

Built by & ecosystem

Scrapefold is built and maintained by Mike Sadofyev (CEO, Datatera.ai) — the scraping engine behind Datatera — alongside a small ecosystem of AI-data tooling. Connect on LinkedIn, X, or GitHub.