Metadata-Version: 2.4
Name: scrapy-calyprium
Version: 1.38.0
Summary: Anti-detection Scrapy middleware — proxy routing and browser rendering for web scraping
Author-email: Calyprium <hello@calyprium.com>
License: MIT
Project-URL: Homepage, https://calyprium.com
Project-URL: Documentation, https://docs.calyprium.com
Project-URL: Repository, https://github.com/Aarkc/scrapy-calyprium
Keywords: scrapy,anti-detection,proxy,browser,tls-fingerprint,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy>=2.11.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: brotli>=1.1.0
Requires-Dist: zstandard>=0.22.0
Provides-Extra: local
Requires-Dist: httpcloak>=0.4.0; platform_system == "Linux" and extra == "local"
Requires-Dist: curl_cffi>=0.7.0; extra == "local"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: responses; extra == "dev"
Dynamic: license-file

# scrapy-calyprium

Anti-detection [Scrapy](https://scrapy.org) middleware for web scraping — proxy routing and stealth browser rendering powered by [Calyprium](https://calyprium.com).

## Install

```bash
pip install scrapy-calyprium
```

## Quick Start

```python
# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")
```

This auto-configures:
- **VeilProxyMiddleware** — routes requests through rotating proxies with TLS fingerprinting
- **MimicBrowserMiddleware** — renders JavaScript pages with stealth browser instances
- **PaceMiddleware** / **PoolBusyMiddleware** — an optional pages-per-minute pace, and backpressure when the shared proxy pool is busy
- **S3 feed storage** — write spider output to Calyprium storage using Scrapy's built-in `S3FeedStorage`

## Usage

### Automatic Configuration (recommended)

```python
# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(
    api_key="clp_your_key_here",
    mimic_stealth_level="maximum",  # basic, moderate, maximum
)
```

### Manual Configuration

```python
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.VeilProxyMiddleware": 100,
    "scrapy_calyprium.MimicBrowserMiddleware": 200,
}

CALYPRIUM_API_KEY = "clp_your_key_here"
VEIL_USER_ID = "your-user-id"
```

### Request Tracing

`CalypriumTraceMiddleware` (added by `configure()`) gives every request one trace: the wait in Scrapy's queue,
each download attempt, the challenge solve if there is one, and the callback that parses the page. The SDK's calls
to Mimic and Tessera carry the trace (`traceparent`), so their steps join it. On the Calyprium platform the
trace of every request of a run is shown with its run. To set it up by hand, register it as both a downloader and
a spider middleware:

```python
DOWNLOADER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
SPIDER_MIDDLEWARES = {"scrapy_calyprium.CalypriumTraceMiddleware": 50}
```

### Failure Diagnosis and Retries

Every download attempt gets a diagnosis from the fetch layer's shared failure classifier (the same
one Mimic and Tessera use), in `response.meta["calyprium_diagnosis"]`:
`{"class", "number", "side", "move", "vendor"?, "status"?, "evidence"?, "confidence"?, "retry_after"?}`.
A transport error leaves it on `failure.request.meta` for an errback. When Mimic's own call failed
and the request fell back to a plain download, `meta["calyprium_fallback_diagnosis"]` and
`meta["calyprium_fetch_error"]` say why.

`MimicBrowserMiddleware` retries an attempt when its diagnosis's move is worth another try:
`retry_elsewhere` and `climb` at once, `retry_later` after `CALYPRIUM_RETRY_DELAY`, `slow_down`
after a doubling backoff (a `Retry-After` wins), `fix_ours` once, and never a `stop` (not found,
sign-in / geo / consent walls, policy). A retry carries `meta["calyprium_attempt_number"]` and
`meta["calyprium_prior_class"]`, which Mimic receives to resume its ladder. `meta["dont_retry"]`
and `meta["max_retry_times"]` work as with Scrapy's RetryMiddleware; with that middleware enabled,
the statuses in `RETRY_HTTP_CODES` are left to it. Stats: `calyprium/retry/<class>` per retry,
`calyprium/class/<class>` per final response.

### Pace and a Shared Proxy Pool

`configure()` adds two downloader middlewares from `scrapy_calyprium.middleware.portfolio`:

- **PaceMiddleware** (120) holds the whole spider, all domains together, to `CALYPRIUM_PACE_PPM` pages a
  minute (a token bucket of `CALYPRIUM_PACE_BURST` tokens). It paces every attempt, including the ones
  `MimicBrowserMiddleware` answers itself on the local route, where `DOWNLOAD_DELAY` and the per-domain
  concurrency do not apply. Without the setting it is not installed. Stats: `calyprium/pace/delayed`,
  `calyprium/pace/wait_seconds`.
- **PoolBusyMiddleware** (950) handles the Calyprium gateway's `pool_busy` answer. The gateway sends it when
  the shared proxy pool has no free connection: `503 Service Unavailable (pool_busy)` with
  `X-Calyprium-Proxy-Error: pool_busy`, which Scrapy reports as a `TunnelError` for HTTPS and the local route
  as `PoolBusyError`. It is not the site's answer, so the request goes back in the queue after a 2-10 s
  jittered backoff and keeps its retries (`retry_times` and the Calyprium retry's count). Nothing solves,
  falls back or blames a cookie slot, exit or domain, and the run's stats do not count the refusal as a
  failure. After `CALYPRIUM_POOL_BUSY_MAX_REQUEUES` refusals of one request, the usual error handling takes
  over. With `CALYPRIUM_POOL_AIMD` on, each refusal cuts `CONCURRENT_REQUESTS` to 3/4, at most once every
  5 s and never below 1. Every 20 responses in a row without a refusal add 1 back, up to the configured
  value. Stats: `calyprium/pool_busy`, `calyprium/pool_busy/requeued`,
  `calyprium/pool_busy/max_reached`, `calyprium/pool_busy/wait_seconds`,
  `calyprium/pool_busy/concurrency`.

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_calyprium.PaceMiddleware": 120,
    "scrapy_calyprium.PoolBusyMiddleware": 950,
}
CALYPRIUM_PACE_PPM = 30  # optional: one page every 2 s
```

### Saving Output to Calyprium Storage

Spider output is saved to Calyprium's S3-compatible storage using Scrapy's built-in feed export:

```python
# settings.py
import scrapy_calyprium

scrapy_calyprium.configure(api_key="clp_your_key_here")

FEEDS = {
    "s3://calyprium/my-spider/%(time)s.jl": {
        "format": "jsonlines",
    },
}
```

The S3 credentials are auto-configured by `configure()` — no additional setup needed.

### Browser Rendering

Mark requests that need JavaScript rendering:

```python
import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        # Regular request (proxy only)
        yield scrapy.Request("https://example.com")

        # Browser-rendered request
        yield scrapy.Request(
            "https://example.com/spa",
            meta={"mimic": True},
        )
```

### Browser-Made Form Submits

Some sites refuse a form POST that their own page did not make (F5 BIG-IP's bot defence adds per-request
tokens in the page just before its search postback). Mark such a request and a real browser page submits it:
Mimic loads the page on the run's exit, types the fields, clicks the control that submits, and the spider gets
the page after the submit as a normal response. The page is kept for the run, so a follow-up postback (the next
result page) acts on the results the last one left:

```python
yield scrapy.FormRequest(url, formdata=fields, callback=self.parse_results,
                         meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$cmdSearch"}})
yield scrapy.Request(url, callback=self.parse_results, dont_filter=True,
                     meta={"mimic_browser_submit": {"submit": "ctl00$MainContent$lbtnNextHeader",
                                                    "navigate": "never"}})
```

A captcha or challenge page comes back as it is (Mimic closes that browser and never solves it), a submit the
page did not apply comes back empty, and a browser submit is never re-sent by the middleware.
`meta["mimic_submit"]` says how it went. Options and the `MIMIC_SUBMIT_*` settings:
`scrapy_calyprium/_browser_submit.py`.

## Authentication

All middleware requires a valid API key. Set it via:

1. `scrapy_calyprium.configure(api_key="clp_...")`
2. `CALYPRIUM_API_KEY` environment variable

## Settings Reference

| Setting | Description | Default |
|---------|-------------|---------|
| `CALYPRIUM_API_KEY` | API key for all services | — |
| `VEIL_GATEWAY_URL` | Proxy gateway URL | `https://proxy.calyprium.com` |
| `VEIL_USER_ID` | User ID for proxy routing | — |
| `VEIL_PROFILE` | Proxy routing profile | — |
| `VEIL_PROXY_TYPE` | `datacenter`, `residential`, `residential_rotating` | — |
| `MIMIC_SERVICE_URL` | Mimic browser service URL | `https://mimic.calyprium.com` |
| `MIMIC_STEALTH_LEVEL` | `basic`, `moderate`, `maximum` | `moderate` |
| `MIMIC_BROWSER_ENGINE` | Specific browser engine | auto |
| `MIMIC_USE_PROXY` | Route browser through proxy | `False` |
| `MIMIC_ALL_REQUESTS` | Render all requests via browser | `False` |
| `MIMIC_USE_SPECTRE` | Use device fingerprints | `True` |
| `MIMIC_SOLVE_DEADLINE_S` | How long Mimic's `/api/solve` may run (Mimic's `SOLVE_DEADLINE_S`); local-first solves wait this plus 30 s | `240` |
| `CALYPRIUM_RETRY_ENABLED` | Retry attempts whose diagnosis says so | `True` |
| `CALYPRIUM_RETRY_TIMES` | Retries per request | `2` |
| `CALYPRIUM_RETRY_DELAY` | Seconds before a `retry_later`; base of the `slow_down` backoff | `5` |
| `CALYPRIUM_RETRY_MAX_DELAY` | Cap on any retry delay (seconds) | `60` |
| `CALYPRIUM_PACE_PPM` | Spider-wide pages per minute, all domains together (`0` = off) | off |
| `CALYPRIUM_PACE_BURST` | Pages the pace lets go at once after an idle spell | `1` |
| `CALYPRIUM_POOL_BUSY_MAX_REQUEUES` | Requeues of one request for the gateway's `pool_busy` before the usual error handling | `30` |
| `CALYPRIUM_POOL_BUSY_DELAY` | Shortest backoff before a requeued request goes again (seconds) | `2` |
| `CALYPRIUM_POOL_BUSY_MAX_DELAY` | Longest backoff; the ceiling doubles with each refusal up to this (seconds) | `10` |
| `CALYPRIUM_POOL_AIMD` | Lower `CONCURRENT_REQUESTS` on `pool_busy`, raise it back after successes | `True` |
| `PRISM_WAIT_FOR_SCAN_S` | `PrismSitemapSpider`: how long to wait for Prism to index an unknown domain before closing as `prism_no_urls` (ends 30 s before `CLOSESPIDER_TIMEOUT`) | `900` |

## Development

`src/scrapy_calyprium/routing/diagnosis.py` is **vendored**: it is Mimic's failure classifier,
copied verbatim from the Calyprium monorepo (`stealth/mimic/mimic/routing/diagnosis.py`). Never
edit it here; when Mimic changes it, copy the file again (its `VERSION` constant says which
contract it is) and run the tests. SDK-side adaptations live in `routing/block_detect.py`.

## License

MIT
