Metadata-Version: 2.5
Name: scrapeunblocker-scrapy-middleware
Version: 0.1.0
Summary: Scrapy downloader middleware that routes your spider's requests through the ScrapeUnblocker web scraping API - real-browser rendering that gets past Cloudflare, DataDome, PerimeterX and Akamai, with no spider changes.
Project-URL: Homepage, https://scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware
Project-URL: Documentation, https://docs.scrapeunblocker.com/sdks/scrapy?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware
Project-URL: Source, https://github.com/ScrapeUnblocker/scrapeunblocker-scrapy-middleware
Project-URL: Changelog, https://github.com/ScrapeUnblocker/scrapeunblocker-scrapy-middleware/blob/main/CHANGELOG.md
Author-email: ScrapeUnblocker <support@scrapeunblocker.com>
License-Expression: MIT
License-File: LICENSE
Keywords: akamai,anti-bot,cloudflare,crawler,datadome,javascript rendering,perimeterx,proxy,scraper,scraping api,scrapy,scrapy downloader middleware,scrapy middleware,web scraping
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: scrapy>=2.8
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# ScrapeUnblocker Scrapy middleware

A Scrapy downloader middleware that routes your spider's requests through the [ScrapeUnblocker](https://scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware) web scraping API. Keep your spiders as they are: every page is fetched in a real browser behind ScrapeUnblocker's anti-bot bypass (Cloudflare, DataDome, PerimeterX, Akamai and others), and your callbacks receive the HTML as if the spider had downloaded the page itself.

- `response.url`, `response.follow()`, relative links and selectors work against the target site, not the API
- Per-request options through `Request.meta`, defaults in `settings.py`
- The target site's own `404` / `410` reaches your spider as a normal `404` / `410`
- API errors keep their status and are explained in the log; `429` and `5xx` go through Scrapy's `RetryMiddleware`

## Install

```bash
pip install scrapeunblocker-scrapy-middleware
```

Requires Python 3.9+ and Scrapy 2.8+.

Get your API key at [app.scrapeunblocker.com](https://app.scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware). The free trial does not require a credit card.

## Enable it

In your project's `settings.py`:

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapeunblocker_middleware.ScrapeUnblockerMiddleware": 543,
}

SCRAPEUNBLOCKER_API_KEY = "YOUR_API_KEY"  # or set the SCRAPEUNBLOCKER_API_KEY env var

# Optional: API options applied to every request
SCRAPEUNBLOCKER_DEFAULTS = {
    "proxy_country": "us",
}

# Recommended (see "Recommended settings" below)
CONCURRENT_REQUESTS = 20  # at most your plan's concurrent requests
CONCURRENT_REQUESTS_PER_DOMAIN = 20
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 523, 524, 408, 429]
```

That is all. Run your spider as usual.

## What the middleware does

For every request it routes, the middleware:

1. Swaps the request for `POST https://api.scrapeunblocker.com/getPageSource?url=<original URL>` with your options in the query string. Booleans are sent as `true` / `false`, `steps` as a JSON array.
2. Sends your key in the `x-scrapeunblocker-key` header. The target site's own headers and cookies are not sent to the API.
3. Keeps the request's callback, errback, `cb_kwargs`, `meta` and priority.
4. When the answer arrives, gives the response the original URL and an original-looking `response.request`, so selectors, `response.urljoin()`, `response.follow()` and the referer of follow-up requests all point at the target site.
5. Reports the target's own status from the `X-Origin-Status` header as `response.status` (see [Status codes](#status-codes-and-errors)).

Requests that are already addressed to the API host are never wrapped again, so retries are safe. The API call is rescheduled with `dont_filter=True`, so the dupefilter and the offsite filter (which already passed the original request) do not drop it, and it keeps the target site as its download slot, so `CONCURRENT_REQUESTS_PER_DOMAIN`, `DOWNLOAD_DELAY` and `DOWNLOAD_SLOTS` still apply per target site.

## Settings

| Setting | Default | Description |
| --- | --- | --- |
| `SCRAPEUNBLOCKER_API_KEY` | - | Your API key. Falls back to the `SCRAPEUNBLOCKER_API_KEY` and then the `SCRAPEUNBLOCKER_KEY` environment variable. Without a key the middleware disables itself (Scrapy logs `Disabled ScrapeUnblockerMiddleware: no ScrapeUnblocker API key...`) and requests go out directly. |
| `SCRAPEUNBLOCKER_DEFAULTS` | `{}` | API options sent with every routed request. A dict, or a JSON string (handy with `-s` on the command line). |
| `SCRAPEUNBLOCKER_API_URL` | `https://api.scrapeunblocker.com/getPageSource` | The endpoint to call. |
| `SCRAPEUNBLOCKER_ENABLED` | `True` | Set to `False` to turn the middleware off without removing it. |
| `SCRAPEUNBLOCKER_ORIGIN_STATUS` | `True` | Use the target site's own status (`X-Origin-Status`) as `response.status`. With `False`, `response.status` is always the API's status. |

Settings can also be set per spider with `custom_settings`.

## API options

These are the `/getPageSource` options, set in `SCRAPEUNBLOCKER_DEFAULTS` or per request:

| Option | Type | Description |
| --- | --- | --- |
| `proxy_country` | `str` | Two-letter country code (`"us"`, `"de"`, ...) for the exit IP. An unsupported code answers `400` with the list of available countries. |
| `parsed_data` | `bool` | Return AI-parsed JSON instead of HTML. See [Parsed data](#parsed-data). |
| `get_cookies` | `bool` | Return JSON with the HTML and the cookies set during the visit. See [Cookies](#cookies). |
| `steps` | `list` of dicts | Browser actions to run after the page loads (click, type, scroll, wait). See [Browser steps](#browser-steps). |
| `list_elements` | `bool` | Return the page's interactive elements with ready-made selectors (JSON) instead of HTML, to help you build `steps`. |

Browser rendering always happens; there is no option for it. Any other key is sent as-is, so an option added to the API later works without upgrading this package, but a name the API does not know has no effect, so the middleware logs a warning for it once (for example `country` -> use `proxy_country`). `url` cannot be set as an option: the request's own URL is sent.

## Per-request options

Put a dict in `Request.meta["scrapeunblocker"]`. It is merged over `SCRAPEUNBLOCKER_DEFAULTS` for that request only; a value of `None` removes a default.

```python
import scrapy


class PriceSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/product/123"]

    def parse(self, response):
        yield {"price": response.css(".price::text").get(default="").strip()}

        yield scrapy.Request(
            "https://example.com/product/124",
            meta={"scrapeunblocker": {"proxy_country": "de"}},
            callback=self.parse,
        )
```

Options are not inherited: a request made with `response.follow(href)` gets the defaults again unless you pass `meta` yourself.

## Sending a request directly

To fetch a request without ScrapeUnblocker, set either of:

```python
yield scrapy.Request(url, meta={"scrapeunblocker": False})
yield scrapy.Request(url, meta={"dont_scrapeunblocker": True})
```

Some requests are always sent directly, because `/getPageSource` cannot serve them:

- **Requests that are not `GET`** (for example `FormRequest` posts). The API loads a URL in a browser; it cannot replay a form submission. They go out directly and the middleware logs a warning once per method. To submit a form through ScrapeUnblocker, load the page with `steps` that fill in and submit the form.
- **Requests Scrapy components make for themselves** (`callback=NO_CALLBACK`): `robots.txt` when `ROBOTSTXT_OBEY` is on, and `FilesPipeline` / `ImagesPipeline` downloads. `/getPageSource` returns HTML documents only (a file or image answers `415`), and routing `robots.txt` would cost a call per site. Setting `meta["scrapeunblocker"]` to a dict or `True` routes such a request anyway.
- **URLs that are not `http` / `https`** (`data:`, `file:`, ...).

## Parsed data

With `parsed_data` the body is JSON, and `response.json()` works as usual:

```python
def parse(self, response):
    payload = response.json()
    product = payload["data"]["data"]
    yield {"title": product.get("title"), "price": product.get("price")}
```

The shape, exactly as the API returns it:

```json
{
  "data": {
    "page_type": "product_detail",
    "data": {
      "title": "A Light in the Attic",
      "price_incl_tax": "...",
      "images": ["https://..."]
    }
  }
}
```

- `data.page_type` is the detected page type (`product_detail`, `listing_detail`, `article_detail`, `search_list`, `homepage`, ..., `not_found` or `unknown`).
- `data.data` holds the extracted fields; list pages put results in `items` with `total` and `next_page`.
- When the page rendered but nothing could be extracted, the answer is still `200`, `data.data` is empty, `data_extracted` is `false` and `html` carries the rendered page so you can parse it yourself.
- A dead URL comes back as `404` / `410` with `{"data": {"page_type": "not_found", "data": {}}}`.

To know in a callback which options a response was fetched with (defaults included), read `response.meta["scrapeunblocker_params"]`:

```python
def parse(self, response):
    if response.meta["scrapeunblocker_params"].get("parsed_data"):
        yield response.json()["data"]["data"]
    else:
        yield {"title": response.css("h1::text").get(default="").strip()}
```

Field reference: [parsed data guide](https://docs.scrapeunblocker.com/guides/parsed-data?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware).

## Browser steps

`steps` runs actions in a real browser after the page loads, then returns the resulting HTML. Use it to wait for content that appears late, click "load more", scroll, or search:

```python
yield scrapy.Request(
    "https://example.com/search",
    meta={
        "scrapeunblocker": {
            "steps": [
                {"action": "type", "selector": "input[name=q]", "value": "headphones"},
                {"action": "press_key", "value": "Enter"},
                {"action": "wait_for", "selector": ".results"},
            ]
        }
    },
)
```

Actions: `wait_for`, `wait_for_text`, `wait`, `click`, `type`, `select`, `press_key`, `scroll`. A step that cannot run answers `422` with a JSON body naming the step; the middleware logs it as `error=step_failed, reason=..., step_index=..., action=...`. A request with `steps` runs its actions every time it is sent, retries included. Full reference: [browser steps](https://docs.scrapeunblocker.com/guides/browser-steps?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware).

## Cookies

With `get_cookies` the body is JSON: `html` (the page), `cookies` (every cookie set during the visit) and `proxy_address`. Read it with `response.json()`:

```python
def parse(self, response):
    payload = response.json()
    selector = scrapy.Selector(text=payload["html"])
    cookies = {c["name"]: c["value"] for c in payload["cookies"]}
```

## What your callback receives

- `response.url` is the URL you requested. The API follows the target's redirects itself and does not report the final URL, so on a page reached through a redirect, relative links resolve against the URL you requested.
- `response.request` is a `GET` for that URL with your callback, headers, cookies and `meta`.
- `response.headers` are the API's headers, including `X-Origin-Status` when the target answered with an error of its own.
- `response.flags` contains `"scrapeunblocker"`, so the `Crawled (200) <GET ...>` log line shows which pages came through the API.
- The HTML is re-indented by the API, so text nodes carry surrounding whitespace and newlines. Strip them: `response.css("h1::text").get(default="").strip()`, `normalize-space()` in XPath, or `MapCompose(str.strip)` in an item loader.

`response.meta` gains these keys:

| Key | Value |
| --- | --- |
| `scrapeunblocker_original_url` | The URL you requested. Also set on the API request, so an errback can read it from `failure.request.meta`. |
| `scrapeunblocker_params` | The options sent (defaults merged with `meta["scrapeunblocker"]`). |
| `scrapeunblocker_api_status` | The status the API itself answered with. |

## Status codes and errors

**The target site's own answer.** When the API reached the site and the site itself answered with an error, the response carries `X-Origin-Status`. The middleware makes that the response status, so Scrapy treats it the way it would treat the site's answer:

| API answer | `response.status` | Meaning |
| --- | --- | --- |
| `404` / `410` with `X-Origin-Status` | `404` / `410` | The page does not exist on the target site. |
| `200` with `X-Origin-Status: 5xx` | that `5xx` | The target's origin server returned its own error page. |
| `200` with `X-Origin-Status: 401` / `451` / `500` | that status | The target's own refusal (for example a login wall), not a problem with your API key. |

These calls are billed and retrying returns the same answer, so the middleware marks them `dont_retry`. `HttpErrorMiddleware` drops them unless you allow the status, for example `meta={"handle_httpstatus_list": [404, 410]}`. The first one of each status is logged at INFO. Set `SCRAPEUNBLOCKER_ORIGIN_STATUS = False` to keep the API's status instead.

**API errors** keep their status and are logged with the API's message and a hint (first occurrence of each at WARNING, or ERROR for `401` / `402`; repeats at DEBUG):

| Status | Meaning | Retried by Scrapy's defaults |
| --- | --- | --- |
| `400` | Bad URL, unsupported `proxy_country`, or the domain does not exist | No |
| `401` | API key not recognised, or the account has no active plan | No |
| `402` | Quota, credit limit or budget limit reached, or a failed payment | No |
| `403` | Blocked by bot protection on every bypass path | No (add `403` to `RETRY_HTTP_CODES` to retry it) |
| `415` | The URL is not an HTML document (send it directly instead) | No |
| `422` | Invalid option, or a browser step failed | No |
| `429` | More requests in flight than your plan's concurrent requests limit | Yes |
| `500`, `502`, `503`, `504` | Temporary problem on the API side, or the call reached its time cap | Yes |
| `523` | The target site is unreachable right now | Only if you add `523` to `RETRY_HTTP_CODES` |

Failed calls are not billed, so retrying them costs nothing. The exceptions are `400 Hostname cannot be resolved` (the domain does not exist) and a `422` from a failed browser step (the page was rendered). Full list: [errors](https://docs.scrapeunblocker.com/errors?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware).

## Recommended settings

```python
# Your plan's concurrent requests limit, not more: requests over it answer 429.
CONCURRENT_REQUESTS = 20
# The API call keeps the target site as its download slot, so this caps how
# many calls run at once for one site. Raise it to use your plan's limit on a
# single-site crawl (Scrapy's default is 8).
CONCURRENT_REQUESTS_PER_DOMAIN = 20

# A call can take its render time cap plus queueing time when the API is busy.
# Keep Scrapy's default; lower values cancel calls that would have succeeded.
DOWNLOAD_TIMEOUT = 180

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 523, 524, 408, 429]

# Calls take seconds, so AutoThrottle (which adapts to latency) slows a crawl
# down a lot. Cap throughput with the concurrency settings above instead.
AUTOTHROTTLE_ENABLED = False
```

Concurrency limits per plan: [concurrency limits](https://docs.scrapeunblocker.com/rate-limits?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware).

About the priority: give the middleware a number higher than `RobotsTxtMiddleware`'s `100`, so `robots.txt` rules are checked against the target site before the request is rewritten (the middleware warns at startup otherwise). `543`, just below `RetryMiddleware` (`550`), also lets `RetryMiddleware` see the API's own status before the target's status is applied.

## Stats

The middleware adds these counters to Scrapy's stats:

| Stat | Counts |
| --- | --- |
| `scrapeunblocker/requests` | Requests routed through the API (retries not included) |
| `scrapeunblocker/response_count` | API responses, retries included |
| `scrapeunblocker/response_status_count/<status>` | API responses by the API's status |
| `scrapeunblocker/origin_status_count/<status>` | Responses that carried `X-Origin-Status` |
| `scrapeunblocker/skipped/<reason>` | Requests sent directly: `meta`, `method`, `no_callback`, `scheme` |
| `scrapeunblocker/exception_count/<name>` | API calls that failed with a network error or timeout |

## Calling other endpoints

A request you address to the API host yourself is passed through untouched, and the middleware adds the API key header if it is missing. That lets a spider call other endpoints, for example Google results from `/serpApi`:

```python
yield scrapy.Request(
    "https://api.scrapeunblocker.com/serpApi?keyword=web+scraping",
    method="POST",
    callback=self.parse_serp,
)
```

## Links

- [ScrapeUnblocker](https://scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware)
- [Dashboard and API key](https://app.scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware)
- [Documentation](https://docs.scrapeunblocker.com?utm_source=pypi&utm_medium=integration&utm_campaign=scrapy-middleware)
- [Python SDK](https://pypi.org/project/scrapeunblocker/) for calling the API outside Scrapy

## License

MIT - see [LICENSE](LICENSE).
