# Supacrawl

> Zero-infrastructure CLI web scraper for LLMs and agents. Turns web pages into
> clean markdown (or JSON) and searches the web — no server, no database. One
> command, `supacrawl`, that prints to stdout or writes files so it drops into
> shell pipelines.

Install with `pip install supacrawl` (`[stealth]` adds anti-bot engines, `[api]`
adds the REST server). For the full agent onboarding guide — command selection,
flags, failure recovery — read SKILL.md, or run `supacrawl install-skill` to
register Supacrawl as a discoverable skill in your agent runtime.

## Pick a command

- scrape: a known URL → markdown/JSON. `supacrawl scrape <url> [-f markdown|json|structuredData|...]`
- search: a query → web results, optionally scraped. `supacrawl search "<q>" [--time-range week] [--topic news]`
- map: a site → its URLs only. `supacrawl map <url>`
- crawl: a site → scrape many pages to a folder (robots-aware). `supacrawl crawl <url> --output dir/`
- llm-extract: URLs → schema-shaped JSON via an LLM. `supacrawl llm-extract <url...> --schema s.json`
- agent: open-ended gathering. `supacrawl agent "<task>"`

## Things an agent should know

- HTTP-first is automatic: static pages skip the browser (fast); JS/bot-protected pages auto-escalate. `--no-http-first` forces the browser.
- `-f structuredData` extracts schema.org JSON-LD, `__NEXT_DATA__`, microdata, and OpenGraph deterministically — cheaper and more exact than LLM `json` for published facts.
- `--expect "<css|text|N words>"` makes "the data is actually present" a retryable condition instead of returning a half-loaded page.
- Errors are remediation-shaped: a failure ends with `[HINT: ...]`. Apply the hint and retry; never retry the same command unchanged.
- Search filters (`--time-range`, `--start-date`/`--end-date`, `--topic`, `--include-domain`/`--exclude-domain`) are pushed to the provider, not applied after.

## Docs

- [SKILL.md](src/supacrawl/resources/SKILL.md): concise agent onboarding — command selection, flags, failure modes.
- [README.md](README.md): overview, installation, links.
- [CLI reference](docs/cli-reference.md): every command and option.
- [API reference](docs/api-reference.md): the optional Firecrawl v2-compatible REST API.
