crawler

A small self-hosted crawler for AI agents.

Edge case: sites that return a Cloudflare client challenge

When a site returns a client challenge, the crawler reports an honest error instead of scraping the challenge DOM.

Usage

crawler scrape https://docs.python.org/3/

Pages become Markdown, text, links and metadata for agents. It runs from the CLI, Python, Docker or as an MCP server, and keeps its cache, jobs and usage in SQLite.

Usage

crawler scrape https://docs.python.org/3/

Pages become Markdown, text, links and metadata for agents. It runs from the CLI, Python, Docker or as an MCP server, and keeps its cache, jobs and usage in SQLite.

Usage

crawler scrape https://docs.python.org/3/

Pages become Markdown, text, links and metadata for agents. It runs from the CLI, Python, Docker or as an MCP server, and keeps its cache, jobs and usage in SQLite.

Usage

crawler scrape https://docs.python.org/3/

Pages become Markdown, text, links and metadata for agents. It runs from the CLI, Python, Docker or as an MCP server, and keeps its cache, jobs and usage in SQLite.