Metadata-Version: 2.4
Name: scrapy-crawio
Version: 1.1.0
Summary: Scrapy downloader middleware for Crawio: route requests through the Crawio API, with proxies and anti-bot handling included.
Author-email: Crawio <support@crawio.com>
License-Expression: MIT
Project-URL: Homepage, https://crawio.com
Project-URL: Documentation, https://crawio.com/docs
Keywords: scrapy,scraping,proxy,crawler,anti-bot,crawio
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy>=2.6
Requires-Dist: requests>=2.25
Dynamic: license-file

# scrapy-crawio

Scrapy downloader middleware for [Crawio](https://crawio.com).

Your spiders make normal requests; this middleware routes them through the Crawio API, which
fetches each one from a healthy exit — using the cheapest route that works, up to a full browser
when the site's anti-bot needs one — and returns the site's own response. No spider changes, no
per-request proxy juggling. Only successful requests are billed.

## Install

```bash
pip install scrapy-crawio
```

## Configure (`settings.py`)

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_crawio.CrawioMiddleware": 1000,
}

CRAWIO_API_KEY = "rk-..."                   # required — dashboard -> API Key
CRAWIO_DOMAINS = ["example.com"]            # hosts to route; omit to route every request
CRAWIO_API_URL = "https://api.crawio.com"   # optional
CRAWIO_TIMEOUT = 120                        # seconds, optional
CRAWIO_SESSION = "job-1"                    # optional: default sticky session id

# Recommended: let Scrapy retry the transient answers.
RETRY_ENABLED = True
RETRY_HTTP_CODES = [429, 502, 503, 504]
```

## Use

Write spiders exactly as normal — the matched hosts go through Crawio:

```python
import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"

    def start_requests(self):
        yield scrapy.Request("https://example.com/catalogue/page-1.html", callback=self.parse)

    def parse(self, response):
        self.logger.info("fetched in %s ms", response.headers.get("X-Crawio-Ms", b"").decode())
        for href in response.css("h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item)
```

A page or JSON arrives as a `TextResponse`. A file (PDF, image, archive) arrives as a plain
`scrapy.http.Response` with its original bytes in `response.body`.

## Response headers

Every response through Crawio carries these, alongside the site's own headers:

| Header | Meaning |
|---|---|
| `X-Crawio-Request-Id` | The request's reference (`req_...`). Quote it when you contact support. |
| `X-Crawio-Cost` | Credits this request used: `1`, or `0` when it was not billed (blocks, errors, timeouts). |
| `X-Crawio-Ms` | Crawio's own time for the fetch, in milliseconds. |

Error responses (429, 502, 503, 504) carry `X-Crawio-Request-Id` and `X-Crawio-Cost` too, and a 429
carries `Retry-After`.

## Per-request overrides

```python
scrapy.Request(url, meta={"crawio_skip": True})       # download this one directly, not through us
scrapy.Request(url, meta={"crawio_session": "job-2"}) # pin to a sticky session
```

A sticky session keeps the same exit and the same cookies across requests: sign in once, reuse the
id, and the following requests stay signed in.

## Retries and billing

Every request carries an `Idempotency-Key`, so a retry reattaches to the same job instead of
starting — and billing — a second one. The key is stamped on the `Request`, so Scrapy's own
`RetryMiddleware` reuses it.

## What Crawio answers

| code | meaning |
|------|---------|
| 200 | success — the site's response is in the body |
| 401 | invalid API key; the request is dropped, not retried |
| 403 | account suspended; fix the account, do not retry |
| 410 | a strict sticky session lost its exit — sign in again on a new session |
| 429 | a plan limit: the body names which (`rate_limited`, `concurrency_limit`, `daily_quota_exceeded`, `monthly_quota_exceeded`, `credits_used_up`) and `Retry-After` says when, where the wait is known |
| 502 | the site blocked the request behind its anti-bot |
| 503 | no capacity right now; retry |
| 504 | the fetch timed out |

Everything except 401 comes back as a real `Response`, so `RetryMiddleware` decides what to do with
it and the body tells you why.

## License

MIT
