Metadata-Version: 2.4
Name: curlx
Version: 3.0.0
Summary: Powerful curl-based HTTP client with free proxy rotation, Instagram API, browser impersonation, smart rate-limit handling
License: MIT
Project-URL: Homepage, https://github.com/pooraddyy/curlx
Project-URL: Repository, https://github.com/pooraddyy/curlx
Project-URL: Bug Tracker, https://github.com/pooraddyy/curlx/issues
Keywords: http,curl,requests,tor,proxy,scraping,impersonation,rate-limit,instagram,browser,free-proxy,ip-rotation,fingerprint,cache
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Operating System :: OS Independent
Classifier: Environment :: Web Environment
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Provides-Extra: tor
Requires-Dist: stem>=1.8; extra == "tor"
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pyflakes; extra == "dev"

# curlx

| | |
|---|---|
| **Version** | 3.0.0 |
| **License** | MIT |
| **Python** | 3.8+ |
| **Platforms** | Linux, macOS, Windows |

A powerful, lightweight HTTP client built on top of curl with advanced features including free proxy rotation, Instagram API integration, browser fingerprint randomization, smart rate-limit handling, Tor support, response caching, and circuit breaker patterns.

---

## Table of Contents

1. [Installation](#installation)
2. [Quick Start](#quick-start)
3. [Core HTTP Client](#core-http-client)
4. [Free Proxy Rotation](#free-proxy-rotation)
5. [Instagram API](#instagram-api)
6. [Browser Impersonation](#browser-impersonation)
7. [Fingerprint Randomization](#fingerprint-randomization)
8. [Tor IP Rotation](#tor-ip-rotation)
9. [Smart Rate Limiting](#smart-rate-limiting)
10. [Response Caching](#response-caching)
11. [Retry Policies](#retry-policies)
12. [Advanced Configuration](#advanced-configuration)
13. [API Reference](#api-reference)

---

## Installation

```bash
pip install curlx
```

With Tor support:
```bash
pip install "curlx[tor]"
```

Requirements:
- curl installed on your system
- Python 3.8 or higher

---

## Quick Start

```python
import curlx

# Simple GET request
resp = curlx.get("https://httpbin.org/get", timeout=15)
print(resp.status_code, resp.json())

# POST with JSON
resp = curlx.post(
    "https://httpbin.org/post",
    json={"key": "value"},
    timeout=15
)
print(resp.json()["json"])

# Using Client with context manager
with curlx.Client(timeout=30, max_retries=3) as client:
    resp = client.get("https://api.github.com/user", auth=("user", "token"))
    print(resp.status_code)
```

---

## Core HTTP Client

### Client Options

| Parameter | Type | Default | Description |
|---|---|---|---|
| timeout | float / tuple | 30 | Request timeout in seconds |
| follow_redirects | bool | False | Follow HTTP redirects |
| verify | bool / str | True | SSL certificate verification |
| tls_version | str | None | Minimum TLS version (1.0, 1.1, 1.2, 1.3) |
| headers | dict | None | Default request headers |
| user_agent | str | None | Default User-Agent string |
| proxies | dict | None | Proxy configuration |
| max_retries | int | 0 | Number of retries on failure |
| retry_statuses | set | {429,500,502,503,504} | Status codes to retry |
| retry_backoff_factor | float | 0.5 | Exponential backoff multiplier |
| retry_jitter | bool | True | Add random jitter to delays |
| debug | bool | False | Enable curl verbose output |
| random_delay | float | 0.0 | Random delay before each request |
| smart_rate_limit | bool | True | Enable adaptive rate limiting |

### HTTP Methods

```python
client = curlx.Client(timeout=30)

client.get(url, params={...})
client.post(url, data={...}, json={...}, files={...})
client.put(url, data={...})
client.delete(url)
client.patch(url, data={...})
client.head(url)
client.options(url)
```

### Response Object

| Attribute | Type | Description |
|---|---|---|
| status_code | int | HTTP status code |
| url | str | Final URL after redirects |
| headers | dict | Response headers |
| cookies | CookieJar | Response cookies |
| content | bytes | Raw response body |
| text | str | Decoded response text |
| json() | Any | Parsed JSON response |
| ok | bool | True for 2xx/3xx status |
| elapsed | float | Request duration in seconds |
| history | list | Redirect history |

---

## Free Proxy Rotation

Automatically fetch and rotate free proxies from multiple sources.

### Supported Proxy Sources

- ProxyScrape
- Proxy-List.download
- Geonode
- GitHub (TheSpeedX proxy list)
- Free-Proxy-List.net

### Basic Usage

```python
from curlx.proxies import FreeProxyRotator

rotator = FreeProxyRotator(rotate_every=3)
proxies = rotator.get_proxy_dict()

client = curlx.Client(proxies=proxies, timeout=30)
resp = client.get("https://api.ipify.org?format=json")
print(resp.json())
```

### Advanced Proxy Configuration

```python
from curlx.proxies import FreeProxyRotator, ProxyPool

# Proxy pool with auto-refresh
pool = ProxyPool(auto_refresh=True, refresh_interval=300, max_size=200)
print(f"Available proxies: {pool.size}")

# Manual rotation
rotator = FreeProxyRotator(
    rotate_every=5,
    auto_fetch=True,
    max_retries_per_proxy=3,
    fallback_direct=True,
)

# Get current IP through proxy
ip = rotator.current_ip()
print(f"Current IP: {ip}")
```

### Instagram with Free Proxies

```python
from curlx.instagram import InstagramAPI

api = InstagramAPI(
    use_free_proxies=True,
    random_delay=3.0,
    max_retries=5,
    timeout=30.0,
)

user = api.get_user_info("nasa")
print(f"Followers: {user.followers}")
print(f"Posts: {user.posts_count}")
```

---

## Instagram API

Full-featured Instagram API wrapper with automatic proxy rotation, rate limiting, and anti-detection measures.

### Features

- User profile information
- User posts with pagination
- Follower/following lists
- User search
- Post media downloads
- Profile picture downloads
- Automatic IP rotation
- Smart rate limiting

### Authentication

```python
from curlx.instagram import InstagramAPI

# Without authentication (limited)
api = InstagramAPI()

# With session (full access)
api = InstagramAPI(session_id="YOUR_SESSION_ID")
```

### User Information

```python
from curlx.instagram import InstagramAPI

with InstagramAPI(use_free_proxies=True) as api:
    user = api.get_user_info("nasa")
    print(f"Username: {user.username}")
    print(f"Full Name: {user.full_name}")
    print(f"Followers: {user.followers}")
    print(f"Following: {user.following}")
    print(f"Posts: {user.posts_count}")
    print(f"Verified: {user.is_verified}")
    print(f"Private: {user.is_private}")
    print(f"Biography: {user.biography}")
```

### Get User Posts

```python
posts, next_cursor = api.get_user_posts("nasa", count=12)
for post in posts:
    print(f"Likes: {post.likes}, Comments: {post.comments}")
    print(f"Caption: {post.caption[:100]}")
    print(f"Is Video: {post.is_video}")
```

### Download Media

```python
from curlx.instagram import InstagramDownloader, InstagramAPI

api = InstagramAPI(session_id="YOUR_SESSION_ID")
downloader = InstagramDownloader(api=api, output_dir="./downloads")

# Download profile picture
path = downloader.download_profile_picture("nasa", hd=True)

# Download post media
paths = downloader.download_post_media("POST_SHORTCODE")
```

---

## Browser Impersonation

Mimic real browser TLS fingerprints and HTTP headers.

### Available Profiles

| Profile | Description |
|---|---|
| chrome124 | Google Chrome 124 (Windows) |
| chrome120 | Google Chrome 120 (Windows) |
| chrome124_mac | Google Chrome 124 (macOS) |
| firefox125 | Mozilla Firefox 125 |
| safari17 | Safari 17 (macOS) |
| edge124 | Microsoft Edge 124 |
| instagram_web | Instagram Web App |
| chrome_android | Chrome Mobile (Android) |

### Usage

```python
from curlx.curl_offi import build_impersonated_client, impersonate

# Quick impersonation
client = build_impersonated_client("chrome124", timeout=30)
resp = client.get("https://www.instagram.com/")

# API mode (XHR headers)
headers, tls = impersonate("instagram_web", api_mode=True)
client = build_impersonated_client("instagram_web", api_mode=True, timeout=30)

# Random profile
from curlx.curl_offi import random_profile
profile = random_profile()
```

---

## Fingerprint Randomization

Generate realistic browser fingerprints for anti-detection.

```python
from curlx.fingerprint import (
    FingerprintRandomizer,
    random_user_agent,
    generate_browser_fingerprint,
)

# Random user agent
ua = random_user_agent("chrome")

# Full fingerprint
fp = generate_browser_fingerprint()
print(fp["user_agent"])
print(fp["viewport"])
print(fp["timezone"])

# Randomizer with rotation
randomizer = FingerprintRandomizer()
headers = randomizer.headers
api_headers = randomizer.api_headers()

# Regenerate for next request
randomizer.regenerate()
```

---

## Tor IP Rotation

```python
from curlx.rotate import TorRotator

rotator = TorRotator(rotate_every=60)

# Get proxy configuration
proxies = rotator.proxy_dict()

# Manual rotation
rotator.rotate()

# Check current IP
print(rotator.current_ip())

# Pre-configured client
with rotator.build_client(timeout=30) as client:
    resp = client.get("https://api.ipify.org?format=json")
```

---

## Smart Rate Limiting

Adaptive per-domain rate limit tracking.

```python
from curlx.rate_limit import RateLimitHandler

handler = RateLimitHandler(
    min_delay=0.5,
    max_delay=120.0,
    per_domain=True,
    jitter_factor=0.15,
    backoff_factor=2.0,
    decay_factor=0.9,
)

handler.record_rate_limit("instagram.com", retry_after=30)
wait = handler.wait_time("instagram.com")
print(f"Wait {wait}s before next request")
```

---

## Response Caching

```python
from curlx.cache import ResponseCache, MemoryCache, cache_result

# Memory cache
cache = MemoryCache(default_ttl=300, max_size=1000)
cache.set("key", value, ttl=60)
result = cache.get("key")

# Response cache (memory + file)
resp_cache = ResponseCache(memory_ttl=300, file_ttl=3600)
key = resp_cache.cache_key("GET", "https://api.example.com/data")
resp_cache.set(key, response_data)

# Decorator
@cache_result(ttl=60)
def fetch_data(url):
    return curlx.get(url).json()
```

---

## Retry Policies

```python
from curlx.retry import RetryPolicy, RetryConfig, CircuitBreaker

# Circuit breaker
cb = CircuitBreaker(failure_threshold=5, recovery_timeout=60)

# Retry policy with circuit breaker
config = RetryConfig(
    max_retries=5,
    base_delay=1.0,
    max_delay=60.0,
    circuit_breaker=cb,
)
policy = RetryPolicy(config)

# Decorator
from curlx.retry import retry_with_policy

@retry_with_policy(max_retries=3, base_delay=1.0)
def fetch_url(url):
    return curlx.get(url)
```

---

## Advanced Configuration

### Custom SSL and TLS

```python
client = curlx.Client(
    verify="/path/to/ca-bundle.crt",
    tls_version="1.3",
    tls_insecure=False,
)
```

### Proxy Configuration

```python
client = curlx.Client(
    proxies={
        "http": "http://proxy:8080",
        "https": "http://proxy:8080",
        "all": "socks5://127.0.0.1:9050",
    }
)
```

### Timeout Configuration

```python
client = curlx.Client(timeout=30)
client = curlx.Client(timeout=(5, 30))
client = curlx.Client(connect_timeout=5, timeout=30)
```

---

## API Reference

### curlx Package

| Function | Description |
|---|---|
| curlx.get(url, ...) | Send GET request |
| curlx.post(url, ...) | Send POST request |
| curlx.put(url, ...) | Send PUT request |
| curlx.delete(url, ...) | Send DELETE request |
| curlx.patch(url, ...) | Send PATCH request |
| curlx.head(url, ...) | Send HEAD request |
| curlx.options(url, ...) | Send OPTIONS request |
| curlx.build_client(...) | Create new Client instance |

### Submodules

| Module | Purpose |
|---|---|
| curlx.client | Client and Session classes |
| curlx.response | Response and CookieJar classes |
| curlx.exceptions | All exception classes |
| curlx.rate_limit | RateLimitHandler |
| curlx.rotate | TorRotator for Tor proxy rotation |
| curlx.curl_offi | Browser impersonation profiles |
| curlx.proxies | Free proxy rotation |
| curlx.instagram | Instagram API wrapper |
| curlx.fingerprint | Browser fingerprint randomization |
| curlx.cache | Response caching |
| curlx.retry | Retry policies and circuit breaker |

---

## License

MIT License - see LICENSE file for details.
