Metadata-Version: 2.4
Name: scrapy-stealth
Version: 0.1.0
Summary: A pluggable stealth and anti-bot framework for Scrapy
Author-email: Fawad <fawadstar6@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Fawad
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/fawadss1/scrapy-stealth
Project-URL: Repository, https://github.com/fawadss1/scrapy-stealth
Project-URL: Issues, https://github.com/fawadss1/scrapy-stealth/issues
Project-URL: Changelog, https://github.com/fawadss1/scrapy-stealth/releases
Keywords: scrapy,web scraping,stealth,anti-bot,crawler,browser impersonation,proxy rotation,fingerprint,cloudflare bypass
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Security
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rnet>=2.4.2
Dynamic: license-file

# scrapy-stealth

A pluggable anti-bot and stealth framework for Scrapy.

[![Changelog](https://img.shields.io/badge/changelog-releases-blue)](https://github.com/fawadss1/scrapy-stealth/releases)

`scrapy-stealth` extends Scrapy with browser impersonation, proxy rotation, fingerprint cycling, and intelligent retry strategies — built for large-scale, production-grade crawling.

---

## Features

- Pluggable engine system (`scrapy`, `stealth`, or custom)
- Browser impersonation (Chrome, Firefox, Safari, Edge, Opera — latest versions)
- Per-request engine selection via `request.meta`
- Proxy support and rotation
- Browser fingerprint rotation
- Smart retry logic that auto-escalates to stealth engine on block
- Anti-bot detection (403/429 status codes + content keyword matching)
- Thread-safe async integration

---

## Installation

```bash
pip install scrapy-stealth
```

> Requires Python 3.10+ and Scrapy 2.15+

---

## Quick Start

### 1. Enable the middleware in `settings.py`

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_stealth.middlewares.stealth.StealthDownloaderMiddleware": 950,
}
```

### 2. Use it in your spider

By default, requests go through the standard Scrapy engine. To use the stealth engine, set `engine` in `request.meta`:

```python
import scrapy

class MySpider(scrapy.Spider):
    name = "example"
    start_urls = ["https://example.com"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"engine": "stealth"},
            )

    def parse(self, response):
        self.logger.info(f"Status: {response.status}")
        yield {"title": response.css("title::text").get()}
```

---

## Per-Request Configuration

All stealth options are set via `request.meta`:

| Key | Type | Description |
|---|---|---|
| `engine` | `str` | Engine to use: `"scrapy"` (default) or `"stealth"` |
| `impersonate` | `str` | Browser to impersonate: `"chrome_137"`, `"firefox_139"`, `"safari_18_5"`, etc. |
| `proxy` | `str` | Proxy URL, e.g. `"http://user:pass@host:port"` |

### Example with all options

```python
yield scrapy.Request(
    url="https://example.com",
    meta={
        "engine": "stealth",
        "impersonate": "firefox_139",
        "proxy": "http://user:pass@proxy-host:8080",
    },
)
```

---

## Strategies

### Proxy Rotation

Use `ProxyRotator` to randomly rotate proxies across requests:

```python
from scrapy_stealth.strategies.proxy import ProxyRotator

proxy_strategy = ProxyRotator(proxies=[
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080",
])

yield scrapy.Request(
    url="https://example.com",
    meta={
        "engine": "stealth",
        "proxy": proxy_strategy.get(),
    },
)
```

### Fingerprint Rotation

Use `ProfileRotator` to randomly rotate the browser fingerprint:

```python
from scrapy_stealth.strategies.fingerprint import ProfileRotator

fp = ProfileRotator()

yield scrapy.Request(
    url="https://example.com",
    meta={
        "engine": "stealth",
        "impersonate": fp.get(),  # randomly picks from latest Chrome, Firefox, Safari, Edge, Opera
    },
)
```

### Intelligent Retry

Use `RetryHandler` in your spider or middleware to retry blocked responses with automatic engine escalation:

```python
from scrapy_stealth.strategies.retry import RetryHandler

retry = RetryHandler()

def parse(self, response):
    if retry.should_retry(response):  # triggers on 403, 429, 503
        yield retry.build(response.request)  # retries via stealth engine
        return
    # normal parsing ...
```

`build` automatically:
- Increments `retry_times` in meta
- Switches `engine` to `"stealth"`
- Sets `dont_filter=True` to bypass Scrapy's duplicate filter

---

## Anti-Bot Detection

Use `AntiBotDetector` to classify responses as blocked:

```python
from scrapy_stealth.detectors.antibot import AntiBotDetector

detector = AntiBotDetector()

def parse(self, response):
    if detector.is_blocked(response):
        self.logger.warning("Blocked! Retrying...")
        # handle retry ...
        return
    # normal parsing ...
```

Detects blocks via:
- HTTP status codes: `403`, `429`
- Body keywords: `"captcha"`, `"access denied"`, `"verify you are human"`

---

## Full Example Spider

```python
import scrapy
from scrapy_stealth.strategies.proxy import ProxyRotator
from scrapy_stealth.strategies.fingerprint import ProfileRotator
from scrapy_stealth.strategies.retry import RetryHandler
from scrapy_stealth.detectors.antibot import AntiBotDetector

proxy_rotator = ProxyRotator(proxies=[
    "http://proxy1:8080",
    "http://proxy2:8080",
])
fp_rotator = ProfileRotator()
retry_handler = RetryHandler()
detector = AntiBotDetector()


class StealthSpider(scrapy.Spider):
    name = "stealth_example"
    start_urls = ["https://example.com"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "engine": "stealth",
                    "impersonate": fp_rotator.get(),
                    "proxy": proxy_rotator.get(),
                },
            )

    def parse(self, response):
        if detector.is_blocked(response):
            self.logger.warning("Blocked response detected, retrying...")
            yield retry_handler.build(response.request)
            return

        yield {"title": response.css("title::text").get(), "url": response.url}
```

---

## Supported Browsers for Impersonation

| Value | Browser |
|---|---|
| `chrome_137` | Chrome 137 (default) |
| `chrome_136` | Chrome 136 |
| `chrome_135` | Chrome 135 |
| `chrome_134` | Chrome 134 |
| `chrome_133` | Chrome 133 |
| `chrome_132` | Chrome 132 |
| `chrome_131` | Chrome 131 |
| `chrome_130` | Chrome 130 |
| `chrome_129` | Chrome 129 |
| `firefox_139` | Firefox 139 |
| `firefox_136` | Firefox 136 |
| `firefox_135` | Firefox 135 |
| `firefox_133` | Firefox 133 |
| `firefox_private_136` | Firefox 136 Private/Incognito |
| `firefox_private_135` | Firefox 135 Private/Incognito |
| `firefox_android_135` | Firefox Android 135 |
| `safari_18_5` | Safari 18.5 |
| `safari_18_3_1` | Safari 18.3.1 |
| `safari_18_3` | Safari 18.3 |
| `safari_18_2` | Safari 18.2 |
| `safari_18` | Safari 18 |
| `safari_ios_18_1_1` | Safari iOS 18.1.1 |
| `safari_ios_17_4_1` | Safari iOS 17.4.1 |
| `safari_ios_17_2` | Safari iOS 17.2 |
| `safari_ipad_18` | Safari iPad 18 |
| `edge_134` | Edge 134 |
| `edge_131` | Edge 131 |
| `edge_127` | Edge 127 |
| `edge_122` | Edge 122 |
| `opera_119` | Opera 119 |
| `opera_118` | Opera 118 |
| `opera_117` | Opera 117 |
| `opera_116` | Opera 116 |
| `okhttp_5` | OkHttp 5 (Android app) |
| `okhttp_4_12` | OkHttp 4.12 (Android app) |
| `okhttp_4_10` | OkHttp 4.10 (Android app) |
| `okhttp_4_9` | OkHttp 4.9 (Android app) |
| `okhttp_3_14` | OkHttp 3.14 (Android app) |
| `okhttp_3_13` | OkHttp 3.13 (Android app) |
| `okhttp_3_11` | OkHttp 3.11 (Android app) |
| `okhttp_3_9` | OkHttp 3.9 (Android app) |

---

## Requirements

- Python 3.10+
- `scrapy >= 2.15.0`

---

## Contributing

Contributions are welcome! This is an open source project and all help is appreciated.

1. Fork the repository on [GitHub](https://github.com/fawadss1/scrapy-stealth)
2. Create a feature branch: `git checkout -b feature/my-feature`
3. Make your changes and add tests if applicable
4. Open a pull request describing what you changed and why

**Ways to contribute:**
- Report bugs via [GitHub Issues](https://github.com/fawadss1/scrapy-stealth/issues)
- Suggest new engines, strategies, or detectors
- Improve documentation or examples
- Add support for new browser fingerprints

---

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for a full history of changes, or browse [GitHub Releases](https://github.com/fawadss1/scrapy-stealth/releases).

---

## License

This project is licensed under the **MIT License** — free to use, modify, and distribute.
See [LICENSE](LICENSE) for the full text.
