Metadata-Version: 2.4
Name: scrapy-ua-rotator
Version: 1.0.1
Summary: Flexible and modern User-Agent rotator middleware for Scrapy, supporting Faker, fake-useragent, and custom providers.
Home-page: https://github.com/geeone/scrapy-ua-rotator
Author: Sergei Denisenko
Author-email: sergei.denisenko@ieee.org
License: MIT
Project-URL: Documentation, https://github.com/geeone/scrapy-ua-rotator
Project-URL: Source, https://github.com/geeone/scrapy-ua-rotator
Project-URL: Bug Tracker, https://github.com/geeone/scrapy-ua-rotator/issues
Keywords: scrapy user-agent middleware rotation fake-useragent faker providers proxy web-scraping
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Framework :: Scrapy
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Faker>=36.0.0
Requires-Dist: fake-useragent>=2.0.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary



# scrapy-ua-rotator

[![PyPI](https://img.shields.io/pypi/v/scrapy-ua-rotator)](https://pypi.org/project/scrapy-ua-rotator/)
[![Python](https://img.shields.io/badge/Python-3.9%20|%203.10%20|%203.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/scrapy-ua-rotator/)
[![License](https://img.shields.io/github/license/geeone/scrapy-ua-rotator)](LICENSE)
[![Build Status](https://github.com/geeone/scrapy-ua-rotator/actions/workflows/build.yml/badge.svg)](https://github.com/geeone/scrapy-ua-rotator/actions/workflows/build.yml)
[![Codecov](https://codecov.io/gh/geeone/scrapy-ua-rotator/branch/main/graph/badge.svg)](https://codecov.io/gh/geeone/scrapy-ua-rotator)

A modern, pluggable User-Agent rotator middleware for the Scrapy framework.

Supports rotation via:
- [`fake-useragent`](https://pypi.org/project/fake-useragent/)
- [`Faker`](https://faker.readthedocs.io/en/stable/providers/faker.providers.user_agent.html)
- Scrapy’s built-in `USER_AGENT` setting

Also supports per-proxy rotation and easy extensibility with custom providers.

---

## 📋 Requirements

- Python 3.9+
- `Faker >= 18.0.0`
- `fake-useragent >= 1.5.0`

> ✅ **Tested with**: Scrapy 2.9, 2.10, 2.11, and 2.12  

---

## 📦 Installation

```bash
pip install scrapy-ua-rotator
```

---

## ⚙️ Configuration

Disable Scrapy’s default middleware and enable ours:

```python
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
    'scrapy_ua_rotator.middleware.RandomUserAgentMiddleware': 400,
    'scrapy_ua_rotator.middleware.RetryUserAgentMiddleware': 550,
}
```

Recommended provider order:

```python
USERAGENT_PROVIDERS = [
    'scrapy_ua_rotator.providers.FakeUserAgentProvider',  # Primary provider using the fake-useragent library
    'scrapy_ua_rotator.providers.FakerProvider',          # Fallback provider that generates synthetic UAs via Faker
    'scrapy_ua_rotator.providers.FixedUserAgentProvider', # Final fallback: uses the static USER_AGENT setting
]

# Static user-agent string to be used if all providers fail to return a valid value
USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64)..."
```

---

## 🧩 Provider Details

### FakeUserAgentProvider

Assigns a new user-agent using [`fake-useragent`](https://github.com/fake-useragent/fake-useragent).  
Supports fine-tuned filtering via:

```python
FAKE_USERAGENT_UA_TYPE = 'Chrome Mobile iOS'           # str; browser to prioritize (default: 'random')
FAKE_USERAGENT_OS = ['Linux']                          # str or list[str]; OS filter (default: None — all OSes)
FAKE_USERAGENT_PLATFORMS = ['mobile']                  # str or list[str]; platform filter (default: None — all platforms)
FAKE_USERAGENT_FALLBACK = 'Mozilla/5.0 (...)'          # str; fallback UA string (default: internal fallback)
```

> 💡 **Note:** See [docs](https://github.com/fake-useragent/fake-useragent/blob/main/README.md) for supported options and advanced usage.

### FakerProvider

Uses [`Faker`](https://faker.readthedocs.io/en/stable/providers/faker.providers.user_agent.html) to generate synthetic UA strings.

```python
FAKER_UA_TYPE = 'chrome'  # or 'firefox', 'safari', etc. (default: 'user_agent' — random web browser)
```

> 💡 **Note:** See [docs](https://faker.readthedocs.io/en/stable/providers/faker.providers.user_agent.html) for supported options and advanced usage.

### FixedUserAgentProvider

Simply uses the provided `USER_AGENT` setting without rotation.  
Useful as a fallback if other providers fail.

```python
USER_AGENT = "Mozilla/5.0 ..."
```

---

## 🔀 Proxy-Aware Mode

If you’re using rotating proxies (e.g., via `scrapy-proxies`), enable per-proxy UA assignment:

```python
RANDOM_UA_PER_PROXY = True
```

Make sure `RandomUserAgentMiddleware` has higher priority than your proxy middleware.

---

## 🧪 Example Output

To verify it’s working, log your request headers in your spider:

```python
def parse(self, response):
    self.logger.info("Using UA: %s", response.request.headers.get('User-Agent'))
```

---

## 🔧 Extending with Custom Providers

Add your own class:

```python
USERAGENT_PROVIDERS = [
    'your_project.providers.MyCustomProvider',
    ...
]
```

Just inherit from `BaseProvider` and implement `get_random_ua()`.

---

## 🤝 Contributing

Contributions, suggestions, and issues are welcome!  
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## 📄 License

MIT © [Sergei Denisenko](https://github.com/geeone)  
See [LICENSE](https://github.com/geeone/scrapy-ua-rotator/blob/main/LICENSE)
