Metadata-Version: 2.4
Name: intelliscrape
Version: 2.5.0
Summary: Advanced web scraping library with anti-detection, TLS impersonation, and stealth browsing. Scrapes 98% of websites.
Home-page: https://github.com/GuixJoy/IntelliScrape
Author: GuixJoy
Author-email: GuixJoy <guixjoy@users.noreply.github.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/GuixJoy/IntelliScrape
Project-URL: Documentation, https://github.com/GuixJoy/IntelliScrape#readme
Project-URL: Repository, https://github.com/GuixJoy/IntelliScrape
Project-URL: Issues, https://github.com/GuixJoy/IntelliScrape/issues
Keywords: web scraping,anti-detection,stealth browser,tls fingerprint,captcha solving,proxy rotation,python scraper,playwright,beautifulsoup,data extraction,cloudflare bypass,nodriver,camoufox
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: beautifulsoup4
Requires-Dist: lxml
Requires-Dist: curl_cffi>=0.7.0
Requires-Dist: playwright>=1.40.0
Requires-Dist: rich
Provides-Extra: stealth
Requires-Dist: nodriver>=0.30; extra == "stealth"
Provides-Extra: camoufox
Requires-Dist: camoufox>=0.1.0; extra == "camoufox"
Provides-Extra: captcha
Requires-Dist: capsolver>=1.0.0; extra == "captcha"
Provides-Extra: async
Requires-Dist: aiohttp>=3.9.0; extra == "async"
Provides-Extra: all
Requires-Dist: nodriver>=0.30; extra == "all"
Requires-Dist: camoufox>=0.1.0; extra == "all"
Requires-Dist: capsolver>=1.0.0; extra == "all"
Requires-Dist: playwright; extra == "all"
Requires-Dist: aiohttp>=3.9.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# IntelliScrape

[![PyPI version](https://img.shields.io/pypi/v/intelliscrape.svg)](https://pypi.org/project/intelliscrape/)
[![Python](https://img.shields.io/pypi/pyversions/intelliscrape.svg)](https://pypi.org/project/intelliscrape/)
[![Downloads](https://img.shields.io/pypi/dm/intelliscrape.svg)](https://pypi.org/project/intelliscrape/)
[![License](https://img.shields.io/pypi/l/intelliscrape.svg)](https://github.com/GuixJoy/IntelliScrape/blob/main/LICENSE)

**Scrape anything. Nothing scrapes back.**

IntelliScrape is a Python web scraping library with anti-detection, TLS fingerprint impersonation, and stealth browsing. It uses a 4-tier engine system that automatically escalates from fast HTTP requests to full browser automation — so you get the cheapest, fastest method that works, and heavier weapons only when needed.

---

## Table of Contents

- [Installation](#installation)
- [Quick Start](#quick-start)
- [CLI Reference](#cli-reference)
- [Library API Reference](#library-api-reference)
- [Engine System](#engine-system-4-tier-escalation)
- [Intelligent Mode](#intelligent-mode)
- [Anti-Detection](#anti-detection)
- [CAPTCHA Solving](#captcha-solving)
- [Manual CAPTCHA Solving](#manual-captcha-solving)
- [Proxy Configuration](#proxy-configuration)
- [Export Formats](#export-formats)
- [Async Support](#async-support)
- [Advanced Usage](#advanced-usage)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [License](#license)

---

## Installation

```bash
pip install intelliscrape
```

### Optional Extras

```bash
# Stealth browsing (nodriver engine)
pip install intelliscrape[stealth]

# Camoufox engine (maximum stealth, Firefox-based)
pip install intelliscrape[camoufox]

# CAPTCHA solving (2Captcha, CapSolver)
pip install intelliscrape[captcha]

# Async support (concurrent scraping)
pip install intelliscrape[async]

# Everything
pip install intelliscrape[all]

# Development
pip install intelliscrape[dev]
```

### System Requirements

- Python 3.9+
- For browser engines: `playwright install chromium` (or `camoufox install` for Camoufox)

---

## Quick Start

### CLI

```bash
# Scrape any website
intelliscrape https://example.com

# Save output to file
intelliscrape https://example.com -o output.txt

# Get structured JSON (title, description, meta tags)
intelliscrape https://example.com --json

# Analyze a site (see what approach IntelliScrape would use)
intelliscrape https://amazon.com --analyze

# Crawl entire website
intelliscrape https://docs.python.org --crawl --max-pages 50
```

### Python — One-liner

```python
from intelliscrape import scrape

text = scrape("https://news.ycombinator.com")
print(text[:500])
```

### Python — Full-featured class

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result)
```

### Get structured data

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape()
data = scraper.get_structured("https://github.com")

print(data.title)          # Page title
print(data.description)    # Meta description
print(data.og_data)        # OpenGraph tags
print(data.json_ld)        # JSON-LD structured data
```

### Crawl entire website

```python
from intelliscrape import crawl

result = crawl("https://docs.python.org", max_pages=100)
print(f"Scraped {result.total_pages} pages, {result.total_failed} failed")

for page in result.pages:
    print(f"  {page.url}: {len(page.content)} chars")
```

---

## CLI Reference

```
intelliscrape [URL] [OPTIONS]
```

### Output Options

| Flag | Description |
|---|---|
| `-o, --output FILE` | Save output to file |
| `--json` | Output structured JSON (title, description, meta tags, etc.) |
| `--raw` | Output raw HTML instead of extracted text |

### Intelligent Mode

| Flag | Description |
|---|---|
| `--analyze` | Analyze site and show recommendations (no scraping) |
| `--no-intelligent` | Disable intelligent auto-detection mode |

### Engine Control

| Flag | Description |
|---|---|
| `--force-browser` | Force browser engine for JS-heavy sites |
| `--manual-captcha` | Open a visible browser when CAPTCHA detected, wait for user to solve |

### Proxy Options

| Flag | Description |
|---|---|
| `--use-free-proxies` | Use free proxies automatically |
| `--no-free-proxies` | Disable free proxy finder |
| `--find-proxies` | Find and test free proxies (no URL needed) |
| `--brightdata-key KEY` | Bright Data API key for residential proxies |
| `--scraperapi-key KEY` | ScraperAPI key |
| `--oxylabs-key KEY` | Oxylabs API key |
| `--smartproxy-key KEY` | Smartproxy API key |

### Authentication

| Flag | Description |
|---|---|
| `--login` | Login to site before scraping |
| `--username USER` | Login username/email |
| `--password PASS` | Login password |
| `--login-url URL` | Explicit login URL |

### Cookies

| Flag | Description |
|---|---|
| `--save-cookies FILE` | Save cookies to JSON file |
| `--load-cookies FILE` | Load cookies from JSON file |

### Request Modification

| Flag | Description |
|---|---|
| `--block PATTERNS` | Block URLs (comma-separated patterns) |
| `--header "Key: Value"` | Add custom header (repeatable) |

### Pagination & Search

| Flag | Description |
|---|---|
| `--paginate` | Auto-follow pagination links |
| `--max-pages N` | Max pages for crawl/pagination (default: 50) |
| `--search QUERY` | Submit search query on the page |

### Crawl

| Flag | Description |
|---|---|
| `--crawl` | Crawl entire website |

### Downloads

| Flag | Description |
|---|---|
| `--download` | Download linked files from page |
| `--download-images` | Download all images from page |
| `--download-dir DIR` | Download directory (default: "downloads") |

### Export

| Flag | Description |
|---|---|
| `--export FORMAT` | Export format: `json`, `csv`, `excel`, `sqlite`, `text`, `markdown` |

### CLI Examples

```bash
# Basic scraping
intelliscrape https://example.com
intelliscrape https://example.com -o output.txt
intelliscrape https://example.com --json

# Analyze site protection
intelliscrape https://amazon.com --analyze

# Find free proxies
intelliscrape --find-proxies

# Use free proxies
intelliscrape https://amazon.com --use-free-proxies

# Login and scrape
intelliscrape https://site.com --login --username user --password pass

# Save/load cookies
intelliscrape https://site.com --save-cookies cookies.json
intelliscrape https://site.com --load-cookies cookies.json

# Custom headers
intelliscrape https://site.com --header "Authorization: Bearer xxx"

# Pagination
intelliscrape https://example.com/products --paginate --max-pages 10

# Search
intelliscrape https://google.com --search "python scraping"

# Download files
intelliscrape https://example.com --download
intelliscrape https://example.com --download-images

# Export formats
intelliscrape https://example.com --export csv -o data.csv
intelliscrape https://example.com --export json -o data.json

# Crawl entire site
intelliscrape https://docs.python.org --crawl --max-pages 50

# Force browser for JS-heavy sites
intelliscrape https://react-app.com --force-browser

# Manual CAPTCHA solving
intelliscrape https://protected-site.com --manual-captcha

# With residential proxy
intelliscrape https://amazon.com --brightdata-key YOUR_KEY
```

---

## Library API Reference

### `scrape()` — Quick One-liner

```python
from intelliscrape import scrape

text = scrape(url, **kwargs)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | str | required | Target URL |
| `engine` | str | None | Force engine: `"static"`, `"playwright_stealth"`, `"camoufox"`, `"nodriver"` |
| `extract` | bool | True | Extract text from HTML |
| `clean` | bool | True | Clean extracted text |
| `return_raw` | bool | False | Return raw HTML |
| `return_structured` | bool | False | Return `StructuredData` object |
| `handle_consent` | bool | True | Handle cookie consent banners |
| `force_browser` | bool | False | Force browser engine |

---

### `IntelliScrape` — Main Class

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape(**kwargs)
```

#### Constructor Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `proxy` | str, ProxyConfig, or list | None | Single proxy or list of proxies |
| `proxies` | list of str | None | Proxy strings (`host:port` or `user:pass@host:port`) |
| `brightdata_key` | str | None | Bright Data API key |
| `scraperapi_key` | str | None | ScraperAPI key |
| `oxylabs_key` | str | None | Oxylabs API key |
| `smartproxy_key` | str | None | Smartproxy API key |
| `prefer_residential` | bool | True | Prefer residential proxies |
| `use_free_proxies` | bool | True | Auto-find free proxies if none provided |
| `api_key` | str | None | CAPTCHA solving API key (2Captcha/CapSolver) |
| `captcha_provider` | str | None | `"2captcha"` or `"capsolver"` |
| `headless` | bool | True | Run browser in headless mode |
| `simulate_behavior` | bool | True | Enable human-like behavioral simulation |
| `manual_captcha` | bool | False | Open visible browser for manual CAPTCHA solving |
| `tls_profile` | str | `"chrome131"` | TLS fingerprint profile to impersonate |
| `session_profile` | str | None | Persistent session profile name |
| `max_retries` | int | 3 | Maximum retry attempts |
| `min_delay` | float | 0.5 | Minimum delay between requests (seconds) |
| `max_delay` | float | 3.0 | Maximum delay between requests (seconds) |
| `requests_per_minute` | int | None | Rate limit (requests per minute) |
| `intelligent` | bool | True | Enable intelligent auto-detection |
| `log_level` | str | `"WARNING"` | Logging level |

#### Methods

##### `scrape(url, **kwargs)`

Scrape a URL and return text content.

```python
result = scraper.scrape(
    url="https://example.com",
    engine=None,            # Force specific engine
    extract=True,           # Extract text
    clean=True,             # Clean text
    return_raw=False,       # Return raw HTML
    return_structured=False, # Return StructuredData
    handle_consent=True,    # Handle cookie consent
    force_browser=False,    # Force browser engine
    intelligent=None,       # Override intelligent mode
)
```

##### `get_structured(url, **kwargs)`

Get structured data (title, description, meta tags, JSON-LD).

```python
data = scraper.get_structured("https://github.com")
print(data.title)
print(data.description)
print(data.og_data)
print(data.json_ld)
```

##### `analyze(url)`

Analyze a site and return recommendations.

```python
analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type)           # "ecommerce"
print(analysis.protection_level)    # "high"
print(analysis.recommended_engine)  # "playwright_stealth"
print(analysis.recommended_delay)   # 3.0
```

##### `scrape_many(urls, **kwargs)`

Scrape multiple URLs with rate limiting.

```python
results = scraper.scrape_many([
    "https://example.com/page1",
    "https://example.com/page2",
])
# Returns: [{"url": ..., "content": ..., "success": ..., "error": ...}, ...]
```

##### `check_captcha(url)`

Check if a URL has a CAPTCHA.

```python
captcha = scraper.check_captcha("https://site.com")
if captcha:
    print(captcha.captcha_type)  # "recaptcha_v2", "hcaptcha", etc.
    print(captcha.site_key)
```

##### `check_antibot(url)`

Check anti-bot protection on a URL.

```python
info = scraper.check_antibot("https://site.com")
if info:
    print(info.vendor)       # "cloudflare", "akamai", etc.
    print(info.confidence)   # 0.95
```

##### `find_free_proxies(test=True)`

Find and test free proxies.

```python
proxies = scraper.find_free_proxies(test=True)
for p in proxies:
    print(f"{p['url']} - speed: {p['speed']:.2f}s")
```

##### `get_proxy_status()`

Get proxy manager status.

```python
status = scraper.get_proxy_status()
print(status['user_proxies'])
print(status['healthy_proxies'])
print(status['providers_available'])
```

---

### `crawl()` — Website Crawler

```python
from intelliscrape import crawl

result = crawl(
    url="https://docs.python.org",
    max_pages=50,
    delay=0.5,
    on_page=None,  # Callback: on_page(done, failed)
)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | str | required | Starting URL |
| `max_pages` | int | 50 | Maximum pages to crawl |
| `delay` | float | 0.5 | Delay between requests (seconds) |
| `on_page` | callable | None | Progress callback `on_page(done, failed)` |

Returns `CrawlResult` with:
- `result.pages` — list of `ScrapeResult` (url, content, status)
- `result.failed` — list of failed pages
- `result.total_pages` — total scraped
- `result.total_failed` — total failed
- `result.to_text()` — all content as single text string

---

### `AsyncIntelliScrape` — Async Scraping

```python
import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape() as scraper:
        urls = [
            "https://example.com",
            "https://python.org",
            "https://github.com",
        ]
        results = await scraper.scrape_many(urls, max_concurrent=5)
        for r in results:
            print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())
```

Also available as standalone functions:

```python
from intelliscrape import scrape_async, scrape_many_async

# Single URL
result = await scrape_async("https://example.com")

# Multiple URLs
results = await scrape_many_async(urls, max_concurrent=10)
```

---

### `DataExporter` — Export Formats

```python
from intelliscrape import DataExporter

# JSON
DataExporter.to_json(data, file="output.json")

# CSV
DataExporter.to_csv(data, file="output.csv")

# Excel
DataExporter.to_excel(data, file="output.xlsx")

# SQLite
DataExporter.to_sqlite(data, file="output.db", table="scraped_data")

# Text
DataExporter.to_text(data, file="output.txt")

# Markdown
DataExporter.to_markdown(data, file="output.md")

# Generic export
DataExporter.export(data, format="json", file="output.json")
```

---

### `Downloader` — File Downloads

```python
from intelliscrape import Downloader

downloader = Downloader()

# Download all linked files
results = downloader.download_links(html, base_url, "downloads/")

# Download all images
results = downloader.download_images(html, base_url, "downloads/images/")

for r in results:
    print(f"{'OK' if r.success else 'FAIL'}: {r.url}")
```

---

### `Authenticator` — Login & Sessions

```python
from intelliscrape import Authenticator, LoginCredentials

auth = Authenticator()
credentials = LoginCredentials(
    username="user@example.com",
    password="secret",
)

success = auth.login(
    "https://site.com/login",
    credentials,
    login_url="https://site.com/login",  # optional
)
```

---

### `FormSubmitter` — Form Interaction

```python
from intelliscrape import FormSubmitter

form_submitter = FormSubmitter()

# Find forms
forms = form_submitter.find_forms(html, base_url="https://site.com")

# Submit search
result_html = form_submitter.search(html, "python scraping", base_url="https://site.com")
```

---

### `Paginator` — Auto-pagination

```python
from intelliscrape import Paginator

paginator = Paginator()

# Find next page link
next_url = paginator.find_next_page(html, current_url, current_page)
```

---

### `RequestInterceptor` — Request/Response Modification

```python
from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()

# Block analytics URLs
interceptor.block_urls(["analytics", "tracking"])

# Add custom headers
interceptor.modify_headers({"X-Custom": "value"})

# Add response handler
def my_handler(response):
    response.body = response.body.replace("old", "new")
    return response

interceptor.add_response_handler(my_handler)
```

---

### `CookieManager` — Cookie Persistence

```python
from intelliscrape import CookieManager

cookie_mgr = CookieManager()

# Save cookies
cookie_mgr.save_cookies("https://site.com", {"session": "abc123"})

# Load cookies
cookies = cookie_mgr.load_cookies("https://site.com")
```

---

### `CaptchaDetector` & `CaptchaSolver` — CAPTCHA Handling

```python
from intelliscrape import CaptchaDetector, CaptchaSolver

# Detect CAPTCHA
captcha = CaptchaDetector.detect(html, url="https://site.com")
if captcha:
    print(captcha.captcha_type)  # CaptchaType.RECAPTCHA_V2
    print(captcha.site_key)

# Solve CAPTCHA (requires API key)
solver = CaptchaSolver(provider="capsolver", api_key="YOUR_KEY")
token = solver.solve_recaptcha_v2(site_key, page_url)
token = solver.solve_hcaptcha(site_key, page_url)
token = solver.solve_turnstile(site_key, page_url)
```

---

### `AntiBotDetector` — Anti-bot Vendor Detection

```python
from intelliscrape import AntiBotDetector

info = AntiBotDetector.detect(html=html, headers=headers, cookies=cookies)
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95
    print(info.indicators)   # ["cf-browser-verification", ...]
```

---

### Anti-bot Bypass Classes

```python
from intelliscrape import (
    CloudflareTurnstileBypass,
    DataDomeBypass,
    PerimeterXBypass,
    AkamaiBypass,
)

# Each bypass class provides:
# - Detection of the anti-bot vendor
# - Recommended engine, proxy, and behavior settings
# - Automated token solving (where possible)
```

---

## Engine System — 4-Tier Escalation

IntelliScrape uses a tiered engine system. It tries the cheapest, fastest method first and escalates only when needed.

```
scrape(url)
    |
    v
+---------------------------+
| Tier 1: Static (curl_cffi)|
| TLS impersonation         |
| Sub-second                |
+---------------------------+
    | if JS-only content
    v
+-------------------------------+
| Tier 2: Playwright Stealth    |
| Headless Chromium + patches   |
| JS rendering                  |
+-------------------------------+
    | if still blocked
    v
+-------------------------------+
| Tier 3: Camoufox             |
| Custom Firefox (C++ patches) |
| Maximum stealth              |
+-------------------------------+
    | if still blocked
    v
+-------------------------------+
| Tier 4: nodriver             |
| Raw CDP, no WebDriver traces |
+-------------------------------+
```

| Tier | Engine | Speed | Stealth | Best For |
|---|---|---|---|---|
| 1 | `static` (curl_cffi) | Sub-second | Low | Static sites, APIs |
| 2 | `playwright_stealth` | 2-5s | Medium | JS-heavy sites, basic bot detection |
| 3 | `camoufox` | 3-8s | High | Protected sites, fingerprint detection |
| 4 | `nodriver` | 5-15s | Maximum | DataDome, PerimeterX, Akamai |

```python
# Auto-detect (default)
text = scraper.scrape("https://site.com")

# Force specific engine
text = scraper.scrape("https://site.com", engine="playwright_stealth")

# Force browser for known JS-heavy sites
text = scraper.scrape("https://react-app.com", force_browser=True)
```

---

## Intelligent Mode

Enabled by default (`intelligent=True`). Before scraping, IntelliScrape analyzes the URL to determine:

- **Site type** (ecommerce, social, news, tech, education, etc.)
- **Protection level** (none, basic, moderate, high, extreme)
- **Recommended engine** (which tier to start with)
- **Recommended delay** (slower for protected sites)
- **Whether residential proxy is needed**

```python
# Analyze a site
analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type.value)         # "ecommerce"
print(analysis.protection_level.value)  # "high"
print(analysis.recommended_engine)      # "playwright_stealth"
print(analysis.recommended_delay)       # 3.0
print(analysis.requires_residential_proxy)  # True

# Disable intelligent mode
text = scraper.scrape("https://site.com", intelligent=False)
```

---

## Anti-Detection

IntelliScrape includes multiple layers of anti-detection:

| Feature | Description |
|---|---|
| **TLS Fingerprinting** | Impersonates Chrome, Firefox, Safari TLS fingerprints (JA3/JA4) |
| **Header Rotation** | Randomizes HTTP headers to avoid fingerprinting |
| **Browser Fingerprinting** | Randomizes viewport, timezone, language, WebGL, canvas |
| **Human Simulation** | Bezier curve mouse movements, natural scroll patterns, realistic delays |
| **Cookie Consent** | Auto-detects and handles cookie consent banners |
| **Rate Limiting** | Smart delays based on site protection level |
| **Retry with Backoff** | Exponential backoff with jitter on failures |

```python
# Disable behavior simulation
scraper = IntelliScrape(simulate_behavior=False)

# Custom TLS profile
scraper = IntelliScrape(tls_profile="firefox120")

# Custom rate limiting
scraper = IntelliScrape(
    min_delay=1.0,
    max_delay=5.0,
    requests_per_minute=20,
)
```

---

## CAPTCHA Solving

### Automated (via API)

Requires an API key from [2Captcha](https://2captcha.com) or [CapSolver](https://capsolver.com).

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape(
    api_key="YOUR_API_KEY",
    captcha_provider="capsolver",  # or "2captcha"
)

# CAPTCHA solving is triggered automatically when detected
result = scraper.scrape("https://protected-site.com")
```

**Supported CAPTCHA types:**

| Type | 2Captcha | CapSolver |
|---|---|---|
| reCAPTCHA v2 | Yes | Yes |
| reCAPTCHA v3 | No | Yes |
| hCaptcha | Yes | Yes |
| Cloudflare Turnstile | No | Yes |
| FunCaptcha | No | No |

### Manual CAPTCHA Solving

When `manual_captcha=True`, IntelliScrape opens a **visible browser window** if a CAPTCHA is detected, waits for you to solve it, then continues scraping.

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://site-with-captcha.com")
# A browser window opens -> solve CAPTCHA -> press Enter in terminal
```

```bash
# CLI
intelliscrape https://site-with-captcha.com --manual-captcha
```

---

## Proxy Configuration

### Single Proxy

```python
scraper = IntelliScrape(proxy="user:pass@proxy:8080")
```

### Multiple Proxies

```python
scraper = IntelliScrape(proxies=[
    "user:pass@proxy1:8080",
    "user:pass@proxy2:8080",
])
```

### Residential Proxy Providers

```python
scraper = IntelliScrape(
    brightdata_key="YOUR_BRIGHTDATA_KEY",
    # scraperapi_key="YOUR_KEY",
    # oxylabs_key="YOUR_KEY",
    # smartproxy_key="YOUR_KEY",
    prefer_residential=True,
)
```

### Free Proxies (Automatic)

```python
scraper = IntelliScrape(use_free_proxies=True)  # default
text = scraper.scrape("https://site.com")
```

```bash
# CLI
intelliscrape https://site.com --use-free-proxies
intelliscrape --find-proxies  # Just find proxies, no scraping
```

---

## Export Formats

### CLI

```bash
intelliscrape https://site.com --export json -o data.json
intelliscrape https://site.com --export csv -o data.csv
intelliscrape https://site.com --export excel -o data.xlsx
intelliscrape https://site.com --export sqlite -o data.db
intelliscrape https://site.com --export text -o data.txt
intelliscrape https://site.com --export markdown -o data.md
```

### Python

```python
from intelliscrape import DataExporter

data = [
    {"url": "https://example.com", "title": "Example", "content": "..."},
    {"url": "https://python.org", "title": "Python", "content": "..."},
]

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db", table="pages")
DataExporter.to_markdown(data, file="output.md")
```

---

## Async Support

```python
import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape(
        proxy="user:pass@proxy:8080",
        headless=True,
        max_concurrent=10,
    ) as scraper:
        urls = [f"https://example.com/page/{i}" for i in range(20)]
        results = await scraper.scrape_many(urls)
        for r in results:
            if r["success"]:
                print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())
```

---

## Advanced Usage

### Scrape with Login

```python
from intelliscrape import IntelliScrape, Authenticator, LoginCredentials

scraper = IntelliScrape()
auth = Authenticator(scraper.session_manager.session)

# Login
credentials = LoginCredentials(username="user@email.com", password="pass")
auth.login("https://site.com/login", credentials)

# Now scrape authenticated pages
result = scraper.scrape("https://site.com/dashboard")
```

### Scrape with Custom Headers

```python
scraper = IntelliScrape()
result = scraper.scrape(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123", "X-Custom": "value"},
)
```

### Block URLs

```python
from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()
interceptor.block_urls(["analytics", "tracking", "ads"])

result = scraper.scrape("https://site.com", interceptor=interceptor)
```

### Scrape React/Vue/Angular SPAs

```python
# Force browser engine for JavaScript-heavy SPAs
result = scraper.scrape("https://react-app.com", force_browser=True)

# Or force specific engine
result = scraper.scrape("https://vue-app.com", engine="playwright_stealth")
```

### Persistent Sessions

```python
scraper = IntelliScrape(session_profile="my_session")

# First run: creates session
scraper.scrape("https://site.com")

# Subsequent runs: reuses session cookies
scraper.scrape("https://site.com/dashboard")
```

### Download Files

```python
from intelliscrape import Downloader

downloader = Downloader()

# Download linked PDFs, ZIPs, etc.
html = scraper.scrape("https://example.com/downloads", return_raw=True)
results = downloader.download_links(html, "https://example.com", "downloads/")

# Download all images
results = downloader.download_images(html, "https://example.com", "downloads/images/")
```

---

## Troubleshooting

### Site returns empty or accessibility widget text

The site is likely a JavaScript SPA. Force browser mode:

```python
result = scraper.scrape(url, force_browser=True)
```

Or via CLI:

```bash
intelliscrape https://site.com --force-browser
```

### CAPTCHA blocking scraping

Use manual CAPTCHA solving:

```python
scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://protected-site.com")
```

Or automated solving:

```python
scraper = IntelliScrape(api_key="YOUR_KEY", captcha_provider="capsolver")
result = scraper.scrape("https://protected-site.com")
```

### Getting blocked by anti-bot

Try escalating engines:

```python
# Try with maximum stealth
result = scraper.scrape(url, engine="camoufox")

# With residential proxy
scraper = IntelliScrape(brightdata_key="YOUR_KEY")
result = scraper.scrape(url)
```

### Playwright not installed

```bash
pip install playwright
playwright install chromium
```

### Camoufox not installed

```bash
pip install camoufox
camoufox install
```

### nodriver not installed

```bash
pip install nodriver
```

---

## Project Structure

```
intelliscrape/
    __init__.py             # Public API exports
    __main__.py             # Entry point for `python -m intelliscrape`
    core.py                 # IntelliScrape class — main orchestrator
    cli.py                  # CLI (argparse + rich output)
    async_scraper.py        # AsyncIntelliScrape, scrape_async, scrape_many_async
    intelligent.py          # SiteAnalyzer, SmartRateLimiter
    auth.py                 # Authenticator, LoginCredentials
    forms.py                # FormSubmitter, Form, FormField
    pagination.py           # Paginator, PageInfo
    export.py               # DataExporter (JSON, CSV, Excel, SQLite, Text, Markdown)
    downloader.py           # Downloader for images/files
    cookies.py              # CookieManager — persistent cookie storage
    crawler.py              # crawl() function, CrawlResult
    interceptor.py          # RequestInterceptor, ResponseModifier
    parser.py               # HTML DOM builder
    cleaner.py              # Text cleaning utilities
    utils.py                # HTML analysis utilities
    exceptions.py           # IntelliScrapeError, DownloadError
    retry.py                # SmartRetry with engine fallback
    ip_manager.py           # IPManager, NaturalRotator
    link_checker.py         # Link validation

    engines/                # Scraping engines (4-tier)
        base.py             # BaseEngine ABC, ScrapeResult dataclass
        static.py           # StaticEngine (curl_cffi — Tier 1)
        playwright_stealth.py  # PlaywrightStealthEngine (Tier 2)
        camoufox.py         # CamoufoxEngine (Tier 3)
        stealth.py          # StealthEngine (nodriver — Tier 4)

    anti_detection/         # Anti-detection subsystem
        antibot.py          # AntiBotDetector — vendor fingerprinting
        behavior.py         # HumanBehavior — mouse paths, scroll patterns
        bypass.py           # CloudflareTurnstileBypass, DataDomeBypass, etc.
        consent.py          # CookieConsentHandler
        fingerprint.py      # FingerprintGenerator
        headers.py          # HeaderManager
        throttle.py         # SmartThrottle, RateLimiter
        tls.py              # TLSConfig — JA3/JA4 impersonation

    challenges/             # Challenge handling
        captcha.py          # CaptchaDetector, CaptchaSolver

    extractor/              # Content extraction
        structured.py       # StructuredExtractor, StructuredData

    proxy/                  # Proxy management
        __init__.py         # ProxyConfig, ProxyManager
        free_finder.py      # FreeProxyFinder
        manager.py          # IntelligentProxyManager
        providers.py        # BrightDataProvider, ScraperAPIProvider, etc.

    session/                # Session persistence
        __init__.py         # SessionManager
```

---

## Contributing

We welcome contributions! Whether it's:

- New anti-bot bypass patterns
- CAPTCHA solving techniques
- Proxy provider integrations
- Bug fixes
- Documentation

See [CONTRIBUTING.md](CONTRIBUTING.md) to get started.

```bash
# Clone the repo
git clone https://github.com/GuixJoy/IntelliScrape.git
cd IntelliScrape/IntelliScrape_library

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check intelliscrape/
```

---

## Community

- **GitHub Issues:** [Report bugs](https://github.com/GuixJoy/IntelliScrape/issues)
- **Discussions:** [Ask questions](https://github.com/GuixJoy/IntelliScrape/discussions)
- **PyPI:** [pypi.org/project/intelliscrape](https://pypi.org/project/intelliscrape/)

---

## License

MIT License — see [LICENSE](LICENSE) for details.

---

**Built with for the data community.**
