Metadata-Version: 2.5
Name: scrapper-tool
Version: 4.2.0
Summary: Reusable web-scraping toolkit — Pattern A/B/C/D ladder, TLS-impersonation fallback chain, deterministic fixture-replay testing, and an optional MCP server for LLM agents.
Project-URL: Homepage, https://github.com/ValeroK/scrapper-tool
Project-URL: Repository, https://github.com/ValeroK/scrapper-tool
Project-URL: Issues, https://github.com/ValeroK/scrapper-tool/issues
Project-URL: Changelog, https://github.com/ValeroK/scrapper-tool/blob/main/CHANGELOG.md
Author: ValeroK
License-Expression: MIT
License-File: LICENSE
Keywords: anti-bot,curl-cffi,extruct,html-parsing,llm-agents,mcp,web-scraping
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: curl-cffi>=0.16.2
Requires-Dist: extruct>=0.18
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.5
Requires-Dist: selectolax>=0.3.21
Provides-Extra: agent
Requires-Dist: mcp<3,>=2.1.1; extra == 'agent'
Provides-Extra: cookies
Requires-Dist: rookiepy>=0.5.6; extra == 'cookies'
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: numpy>=1.26; extra == 'dev'
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: pip-audit>=2.7; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Provides-Extra: full
Requires-Dist: browser-use<0.14,>=0.13; extra == 'full'
Requires-Dist: browserforge>=1.2; extra == 'full'
Requires-Dist: camoufox[geoip]<0.6,>=0.5; extra == 'full'
Requires-Dist: crawl4ai>=0.9; extra == 'full'
Requires-Dist: numpy>=1.26; extra == 'full'
Requires-Dist: patchright>=1.49; extra == 'full'
Requires-Dist: pillow>=10; extra == 'full'
Requires-Dist: pydantic-settings>=2.15; extra == 'full'
Requires-Dist: scrapling[fetchers]>=0.3; extra == 'full'
Requires-Dist: turnstile-solver>=0.1.4; extra == 'full'
Provides-Extra: hostile
Requires-Dist: scrapling[fetchers]>=0.3; extra == 'hostile'
Provides-Extra: http
Requires-Dist: fastapi>=0.115; extra == 'http'
Requires-Dist: prometheus-client>=0.20; extra == 'http'
Requires-Dist: pyyaml>=6.0; extra == 'http'
Requires-Dist: uvicorn[standard]>=0.32; extra == 'http'
Provides-Extra: llm-agent
Requires-Dist: browser-use<0.14,>=0.13; extra == 'llm-agent'
Requires-Dist: browserforge>=1.2; extra == 'llm-agent'
Requires-Dist: camoufox[geoip]<0.6,>=0.5; extra == 'llm-agent'
Requires-Dist: crawl4ai>=0.9; extra == 'llm-agent'
Requires-Dist: numpy>=1.26; extra == 'llm-agent'
Requires-Dist: patchright>=1.49; extra == 'llm-agent'
Requires-Dist: pillow>=10; extra == 'llm-agent'
Requires-Dist: pydantic-settings>=2.15; extra == 'llm-agent'
Provides-Extra: turnstile-solver
Requires-Dist: turnstile-solver>=0.1.4; extra == 'turnstile-solver'
Description-Content-Type: text/markdown

<div align="center">

# scrapper-tool

**A reusable Python web-scraping toolkit — production-grade primitives, anti-bot ladder, fixture-replay testing.**

Built from the scraping core behind [PartsPilot](https://github.com/ValeroK/affiliate-service), extracted as an open-source library so other projects (and LLM agents) can pick up the same patterns without redoing the reverse-engineering work.

<br />

[![CI](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml/badge.svg)](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
[![Python versions](https://img.shields.io/pypi/pyversions/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
[![Downloads](https://img.shields.io/pypi/dm/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://github.com/astral-sh/ruff)
[![Type-checked: mypy](https://img.shields.io/badge/type--checked-mypy-1f5082.svg)](https://mypy-lang.org/)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
[![GitHub Stars](https://img.shields.io/github/stars/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/stargazers)
[![GitHub Forks](https://img.shields.io/github/forks/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/network/members)

[**Quickstart**](docs/quickstart.md) · [**All docs**](docs/index.md) · [**Settings**](docs/SETTINGS.md) · [**MCP**](docs/mcp.md) · [**Docker**](docs/docker.md) · [**Changelog**](CHANGELOG.md)

</div>

---

> **Status (2026-08-29):** stable (`v3.1.1`). The public Python API and MCP tool surface are SemVer-stable.
>
> **`v3.1.0`** migrates the MCP server to the 2.x SDK (`FastMCP` — `MCPServer`),
> lifting the `mcp<2` cap deferred from 3.0.0. No public Python API or MCP tool
> changed; the breaking part is the dependency floor, `mcp>=2.1.1,<3` on the
> `[agent]` extra, which will not co-install with a `mcp` 1.x pin. It also fixes
> both docker-compose MCP services, which inherited the REST entrypoint and so
> never spoke MCP at all.
>
> **`v3.0.0`** adds a target URL guard that vets every URL before a request is issued — private, loopback and cloud-metadata targets are refused, and that is **on by default**, which is the breaking part. It also gives the captcha grid tier its own vision model, blocks page-initiated SSRF in the render tier, promotes the impersonation ladder to `chrome150`, and stops overriding the impersonated `User-Agent` (which had been advertising `scrapper-tool/0.1` beside a Chrome TLS handshake — a self-identifying mismatch).
>
> Read the [breaking-change table](CHANGELOG.md) before upgrading. Every number in the docs is measured, including the ones that did not work; see [`docs/TESTING.md`](docs/TESTING.md).

## What it does

Web scraping is mostly the same work every time: pick the extraction method the
site actually needs, survive the TLS fingerprint, retry sanely, and write tests
that do not break the moment the vendor ships CSS. `scrapper-tool` packages the
parts that do not change per vendor.

One call does the escalation for you — cheapest method first, climbing only when
the site forces it:

```python
from scrapper_tool import scrape

data = await scrape("https://vendor.example/product/123")
```

Behind that call: a TLS-impersonation ladder, a stealth browser, a local LLM, and
a captcha cascade — in that order, and only as far as the site makes necessary.

## Install

```bash
uv pip install "scrapper-tool[full,agent]"    # all five patterns + MCP server
camoufox fetch                                # ~300 MB, best-stealth browser
```

`pip` works too, but `[full]` needs `uv` — Scrapling and Crawl4AI pin
incompatible `lxml` ranges and only `uv` honours the override that reconciles
them. Lighter installs and the pip escape hatch: **[Install guide](docs/quickstart.md#install)**.

Check what actually works on your machine:

```bash
scrapper-tool doctor
```

It reports every tier as `ok` / `degraded` / `missing` with the exact command to
fix each one, and exits non-zero so it works as a CI or container healthcheck.

## The five patterns

Pick the one DevTools points at, or let `scrape()` choose.

| Pattern | When | Cost |
|---|---|---|
| **A** — JSON API | An XHR returns the data | Lowest |
| **B** — Embedded JSON | `ld+json`, `__NEXT_DATA__`, `__NUXT__` | Low |
| **C** — CSS / microdata | Price is in the HTML, no JSON | Medium |
| **D** — Hostile | Cloudflare Turnstile, Akamai, DataDome | High — real browser |
| **E** — LLM agent | D is still blocked, or the page needs interaction | Highest — local LLM |

Full guides: **[patterns A–E](docs/index.md#pattern-guides)**.

## Security: targets are vetted before they are fetched

Every surface checks a URL before issuing a request. Private, loopback,
link-local and cloud-metadata targets are refused, along with non-`http(s)`
schemes and hostnames that *resolve* into private space.

This is **on by default**, and it matters most if you run the REST sidecar:
without it, anything that can reach the sidecar can make it fetch
`169.254.169.254` and read your cloud credentials back.

To reach a legitimate internal target, allowlist it rather than turning the
guard off:

```bash
SCRAPPER_TOOL_URL_GUARD_ALLOW=127.0.0.1,10.0.0.0/8
```

What is covered, what is not, and the fully-closed `..._STRICT` mode:
**[Target URL guard](docs/SETTINGS.md#target-url-guard-ssrf-protection-v221)**.

## Run it as a service

| Mode | Command | Docs |
|---|---|---|
| **MCP server** (Claude, Cursor, any MCP client) | `scrapper-tool-mcp` | [docs/mcp.md](docs/mcp.md) |
| **REST sidecar** (any language, plain HTTP) | `scrapper-tool-serve` | [docs/http-sidecar.md](docs/http-sidecar.md) |
| **Docker** (all five patterns in one image) | `docker compose up` | [docs/docker.md](docs/docker.md) |

## Use it as an MCP server

Exposes the whole toolkit to any MCP client — Claude Code, Claude Desktop,
Cursor, mcp-use, AutoGen, LangChain. Needs the `[agent]` extra:

```bash
uv pip install "scrapper-tool[full,agent]"
```

### Wire it up

Add to your client's MCP config (`.mcp.json` for Claude Code,
`claude_desktop_config.json` for Claude Desktop):

```json
{
  "mcpServers": {
    "scrapper-tool": {
      "command": "scrapper-tool-mcp",
      "args": [],
      "env": {}
    }
  }
}
```

Restart the client and all nine tools appear, plus a `skill://scrapper-tool`
resource carrying the tool's own operating manual. That is the whole setup for the
default transport — stdio, which the client spawns and talks to over
stdin/stdout.

### The nine tools

| Tool | Use it for |
|---|---|
| `auto_scrape` | **Start here.** Escalates A/B/C → D → E1 → E2 by itself and reports which tier won. |
| `fetch_with_ladder` | One fetch through the TLS-impersonation ladder. `extract_structured=True` also parses JSON-LD. |
| `extract_product` | schema.org Product+Offer out of HTML you already have. |
| `extract_microdata_price` | `<meta itemprop="price">` anchors out of HTML you already have. |
| `map_site` | List a site's URLs from sitemaps + page links. No browser, no LLM, so it is cheap. |
| `crawl_site` | Breadth-first crawl running the full cascade per page. Honours robots.txt. |
| `agent_extract` | Pattern E1 — stealth render plus one LLM call. Needs `[llm-agent]`. |
| `agent_browse` | Pattern E2 — multi-step agent for logins, pagination, forms. Needs `[llm-agent]`. |
| `canary` | Which TLS fingerprint a site accepts. Diagnostics. |

`docs/mcp-tools.json` is the generated, CI-enforced copy of this list.

### Over HTTP instead of stdio

For a long-lived server that clients reach by URL:

```bash
scrapper-tool-mcp --transport streamable-http --host 0.0.0.0 --port 8000
```

Then point the client at `http://localhost:8000/mcp`. `--transport sse` is also
supported. Each flag has an env var (`SCRAPPER_TOOL_MCP_TRANSPORT`, `_HOST`,
`_PORT`).

In Docker, use the bundled service rather than the bare image — the image's
default entrypoint is the REST sidecar, and the compose service overrides it:

```bash
docker compose --profile http up -d scrapper-tool-mcp-http
```

For the stdio spawn pattern in Docker (`docker compose run --rm -T
scrapper-tool`), see **[docs/mcp.md](docs/mcp.md)**.

### Check it works

```bash
scrapper-tool doctor
```

Reports every tier as `ok` / `degraded` / `missing`. For a real end-to-end
session that opens a JSON-RPC connection and calls all nine tools:

```bash
uv run python scripts/e2e/test_mcp_session.py
```

Two things that will otherwise cost you an hour: `agent_browse` needs a
CDP-capable browser, so set `SCRAPPER_TOOL_AGENT_BROWSER=patchright` (the
Camoufox default is Firefox, which has no CDP and fails deliberately rather
than silently dropping stealth); and in Docker a host-local LLM URL must be
`host.docker.internal`, not `127.0.0.1`, which inside the container means the
container.

Full reference: **[docs/mcp.md](docs/mcp.md)**. Framework-specific wiring:
**[docs/agent-integration.md](docs/agent-integration.md)**.

## Settings

Every knob is an env var, a constructor argument, or a per-call keyword — in that
order of precedence. **[docs/SETTINGS.md](docs/SETTINGS.md)** is the canonical
reference: if a setting is not there, it is not a public knob.
**[`.env.example`](.env.example)** is a drop-in starter with every variable annotated.

## Architecture

```mermaid
flowchart TD
    A[Your scraper code or LLM agent] --> B[vendor_client / request_with_retry]
    B --> C{TLS-sensitive?}
    C -- no --> D[httpx]
    C -- yes --> E[curl_cffi ladder]
    E --> E1[chrome150] --> E2[chrome146] --> E3[safari2601] --> E4[firefox147]
    D --> F[Response]
    E4 --> F
    F --> G{Pattern}
    G -- A --> H[JSON API model]
    G -- B --> I[extruct: ld+json / next_data / nuxt]
    G -- C --> J[selectolax: microdata / CSS]
    G -- D --> K["Scrapling (Playwright + Turnstile)"]
    G -- "BlockedError + interactive" --> M["Pattern E: agent_extract / agent_browse"]
    M --> M1["Stealth browser (Camoufox / Patchright / Obscura)"]
    M1 --> M2["Local LLM (Ollama, qwen3-vl:8b)"]
    M2 --> M3["Captcha cascade (Camoufox auto → Theyka → paid)"]
    M3 --> L[Validated product data]
    H --> L
    I --> L
    J --> L
    K --> L
```

## Documentation

| | |
|---|---|
| **[Quickstart](docs/quickstart.md)** | 5-minute on-ramp. |
| **[Settings reference](docs/SETTINGS.md)** | Every env var, default, choice list. *(v1.0.0+)* |
| **[`.env.example`](.env.example)** | Drop-in starter file with every variable annotated. |
| **[E2E test plan](docs/E2E_TEST_PLAN.md)** | Operator-runnable end-to-end suite — library / Docker / MCP modes against LM Studio. *(v1.0.0+)* |
| **[`scripts/e2e/`](scripts/e2e/)** | Runnable test scripts referenced by the E2E plan. |
| **[Recon playbook](docs/recon.md)** | DevTools-driven reverse-engineering of a new vendor site. |
| **[Pattern A — JSON API](docs/patterns/a-json-api.md)** | Vendor exposes an XHR / JSON endpoint. |
| **[Pattern B — Embedded JSON](docs/patterns/b-embedded-json.md)** | `ld+json`, `__NEXT_DATA__`, `__NUXT__`, RSC payloads. |
| **[Pattern C — CSS / microdata](docs/patterns/c-css-microdata.md)** | `itemprop="price"`, fallback selectors. |
| **[Pattern D — Hostile](docs/patterns/d-hostile.md)** | Cloudflare Turnstile, Akamai EVA. |
| **[Pattern E — LLM agent](docs/patterns/e-llm-agent.md)** | Local-LLM-driven scraping for any protected site. *(v1.0.0+)* |
| **[Anti-bot ladder reference](docs/reference/ladder.md)** | How the ladder walks, when to bump the primary profile. |
| **[Test helpers](docs/reference/testing.md)** | `FakeCurlSession`, `replay_fixture`, golden-snapshot pattern. |
| **[Agent integration](docs/agent-integration.md)** | MCP wiring for Claude, OpenClaw, Hermes Agent, AutoGen, LangChain. *(v0.2.0+)* |
| **[2026-04-30 landscape research](docs/research/2026-04-30-landscape.md)** | Why these tools, sourced. |

## Why this exists

Most scrapers are written from scratch every time, even though 90% of the work is the same: pick the right extraction pattern, survive the TLS fingerprint, retry/backoff sanely, and write tests that don't drift the moment a site updates.

`scrapper-tool` packages the parts that don't change per vendor, so you only write the parts that do.

- **Pattern-first design.** Five named, documented extraction patterns (A–E) — pick the one DevTools points at, skip the rest.
- **Anti-bot ladder built in.** Auto-walks `chrome150 → chrome146 → safari2601 → firefox147 → chrome133a` when a profile gets fingerprinted.
- **Deterministic tests.** Fixture-replay (`FakeCurlSession`, `replay_fixture`, golden snapshots) — no live HTTP in CI.
- **Optional hostile mode.** Cloudflare Turnstile / Akamai EVA defeat path via [Scrapling](https://github.com/D4Vinci/Scrapling) — opt-in extra, no Playwright bloat by default.
- **LLM-agent ready.** `v0.2.0+` ships an MCP server so Claude, AutoGen, LangChain, etc. can drive the scraper directly.
- **Local-LLM scraping for any protected site (`v1.0.0+`).** Pattern E adds Camoufox + browser-use + Crawl4AI + Ollama — zero API cost, two modes (`agent_extract` for fast 1-call extraction, `agent_browse` for interactive multi-step tasks). Humanlike-behavior layer defeats DataDome.
- **Captchas solved on the way past (`v2.2.0+`).** Five tiers, cheapest first: settle → click the checkbox → align the slider (pure geometry, **no model**) → read the image grid with a local VLM → paid solver. Measured live: reCAPTCHA v2 grids **3/4–4/5** with a ~27B VLM, GeeTest sliders **~20%** with no model at all. reCAPTCHA v3 and AWS WAF are *not* solvable — they are risk scores, not puzzles, and the docs say so.
- **Clearance cookies are kept, not thrown away (`v2.2.0+`).** A solve costs ~70 s of local inference or a paid API call; the `cf_clearance` it buys now survives to the next tier, and to the next run via a persisted browser profile.
- **Boring stack.** `httpx`, `curl_cffi`, `selectolax`, `extruct`. No managed SaaS bundled — your code, your egress.

## Roadmap

- [x] **v0.1.0** — Core HTTP client, retry/backoff, anti-bot ladder, patterns A–D, fixture-replay test helpers.
- [x] **v0.2.0** — MCP server for LLM agents; canary CLI for nightly fingerprint-health probes.
- [x] **v1.0.0** — Pattern E: local-LLM-driven scraping (Camoufox + browser-use + Crawl4AI + Ollama), captcha cascade, humanlike-behavior layer, full Docker stack. Public API + MCP tool surface stable under SemVer.
- [ ] **v1.1.0** — Pluggable rate-limit / robots.txt policies; per-vendor profile presets; `agent_session()` warm-browser pooling; broader Pattern E backends.

See [`CHANGELOG.md`](CHANGELOG.md) for landed changes and [open issues](https://github.com/ValeroK/scrapper-tool/issues) for what's in flight.

## Contributing

PRs and issues are welcome. Every PR that meaningfully changes how we scrape lands a `CHANGELOG.md` row.

- Read **[`CONTRIBUTING.md`](CONTRIBUTING.md)** for the maintenance contract.
- Read **[`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)** before opening a discussion.
- Good first issues live under the [`good first issue`](https://github.com/ValeroK/scrapper-tool/labels/good%20first%20issue) label.

## Contributors

<a href="https://github.com/ValeroK/scrapper-tool/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=ValeroK/scrapper-tool" alt="Contributors" />
</a>

Want to see your avatar here? Check [CONTRIBUTING.md](CONTRIBUTING.md) and open a PR.

## Acknowledgements

`scrapper-tool` stands on the shoulders of these projects:

- [`httpx`](https://github.com/encode/httpx) — async HTTP client
- [`curl_cffi`](https://github.com/lexiforest/curl_cffi) — TLS / JA3 impersonation
- [`selectolax`](https://github.com/rushter/selectolax) — fast HTML parsing
- [`extruct`](https://github.com/scrapinghub/extruct) — `ld+json`, microdata, RDFa extraction
- [`Scrapling`](https://github.com/D4Vinci/Scrapling) — Playwright-based hostile-site backend

## License

[MIT](LICENSE) © scrapper-tool contributors.

<div align="center">

If `scrapper-tool` saves you time, consider [starring the repo](https://github.com/ValeroK/scrapper-tool) — it helps others find it.

</div>
