Metadata-Version: 2.4
Name: scrapingpros
Version: 0.5.1
Summary: Python SDK for the Scraping Pros API — web scraping with browser rendering, proxy rotation, and structured data extraction.
Project-URL: Homepage, https://gitlab.com/7Puentes/scrapingpros-python-sdk
Project-URL: Documentation, https://api.scrapingpros.com/llms-full.txt
Project-URL: Repository, https://gitlab.com/7Puentes/scrapingpros-python-sdk
Project-URL: Issues, https://gitlab.com/7Puentes/scrapingpros-python-sdk/-/issues
Author-email: 7Puentes <dev@7puentes.com>
License-Expression: MIT
License-File: LICENSE
Keywords: api,browser-rendering,data-extraction,proxy,scraping,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-httpx>=0.30.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# scrapingpros

**Production web scraping at scale, without the plumbing.** Submit a list of URLs, stream results as the workers finish them, and let the server handle the hard parts — browsers, proxies, retries, anti-bot, soft-block detection. Submit 50,000 URLs in one call, walk away, come back to handled results.

[![PyPI](https://img.shields.io/pypi/v/scrapingpros.svg)](https://pypi.org/project/scrapingpros/) [![Python](https://img.shields.io/pypi/pyversions/scrapingpros.svg)](https://pypi.org/project/scrapingpros/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)

📚 **[Full docs](https://docs.scrapingpros.com/docs/category/python-sdk)** · 📦 **[PyPI](https://pypi.org/project/scrapingpros/)** · 📝 **[Release notes](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/release-notes)** · 🔧 **[API status](https://api.scrapingpros.com)**

---

## Install

```bash
pip install scrapingpros
```

Requires Python 3.10+. No signup needed — the demo token below gives you 5,000 credits/month, 30 req/min, all features enabled.

## The headline use case — `submit_batch()` + `iter_results()`

Streaming async batches are the centerpiece of the SDK. They scale from a handful of URLs up to tens of thousands, with progress, automatic resume, soft-block detection, and per-job traceability via `custom_id`:

```python
from scrapingpros import SyncClient

client = SyncClient("demo_6x595maoA6GdOdVb")

# Submit 50,000 URLs in one call — the API queues them across the worker pool.
batch = client.submit_batch("daily-products", [
    {"url": product_url, "custom_id": product_id}
    for product_id, product_url in catalog.items()
])

# Stream results as workers finish them. Progress, ETA, and counters are live.
for result in batch.iter_results():
    if result.guidance.success:
        save(product_id=result.custom_id, content=result.content)
    else:
        log_failure(result.custom_id, result.guidance.error_type, result.guidance.next_steps)

    # Live progress — no extra API calls
    print(f"{batch.pct:.1%}  ({batch.success_count}/{batch.total})  ETA {batch.eta_seconds}s")
```

What you get out of the box:

- **Streaming**, not "submit and poll". Results come back the moment each worker finishes — your loop body runs in real time.
- **`custom_id`** round-trips through every layer (request → job → result → webhook). Map results back to your domain objects without depending on order.
- **`result.guidance.success`** is the *server's* verdict on whether the page produced usable content. It catches soft-blocks (Google CAPTCHA pages with 200 + large body, Amazon "Robot Check") that a naive `status_code` check would miss.
- **Automatic refunds.** Credits are refunded on the spot for any response that doesn't deliver usable content (4xx/5xx, captchas, worker failures, timeouts). You pay for *successful* content only.
- **Per-job retries** with proxy rotation and IP/fingerprint changes. `retry_on_block=True` handles anti-bot sites without you writing retry logic.
- **Worker-restart resilient.** The SDK re-attaches transparently to in-flight runs across transient `ConnectionError` and worker churn.

## When to use sync (`client.scrape`)

Sync is for **one-off requests, exploration, and debugging**. It blocks on a single URL and returns the result inline. For anything that involves more than a handful of pages or runs on a schedule, use `submit_batch()`.

```python
result = client.scrape("https://example.com", format="markdown")
print(result.content)            # markdown if available, else html
print(result.guidance.success)   # canonical verdict
```

## Common patterns

### Browser-rendered + anti-bot protected sites

```python
result = client.scrape(
    "https://spa-site.com",
    browser=True,            # 5 credits, full JS rendering
    retry_on_block=True,     # auto-retry up to 3x with different IP/fingerprint
)
```

The API picks the right engine internally per target — you don't configure browser engines.

### Structured data extraction (CSS / XPath, no parsing on your side)

```python
result = client.scrape("https://quotes.toscrape.com/", extract={
    "quotes":  {"selector": "css:.text",   "multiple": True},
    "authors": {"selector": "css:.author", "multiple": True},
})
print(result.extracted_data["quotes"])
```

### Form-encoded POST (OAuth2, legacy APIs)

```python
from scrapingpros import MethodPOST

resp = client.scrape(token_url, http_method=MethodPOST(
    payload={"grant_type": "client_credentials"},
    content_type="form",   # since v0.5.0; default is "json"
))
```

### Wait for hidden DOM nodes (`<script>` tags with embedded JSON)

```python
from scrapingpros import WaitForSelectorAction

result = client.scrape(url, browser=True, actions=[
    WaitForSelectorAction(selector="css:script#__NEXT_DATA__", time=8000, state="attached"),
])
```

### Capture response bodies (auth tokens, GraphQL payloads)

```python
from scrapingpros import NetworkCaptureConfig

result = client.scrape(url, browser=True, network_capture=NetworkCaptureConfig(
    resource_types=["xhr", "fetch"],
    url_pattern="*identitytoolkit.googleapis.com*",   # captures matching response bodies (≤64 KB)
))
for entry in result.network_requests or []:
    if "body" in entry:
        token = parse_token(entry["body"])
```

### Async/await client

```python
from scrapingpros import AsyncClient

async with AsyncClient("demo_6x595maoA6GdOdVb") as client:
    result = await client.scrape("https://example.com", format="markdown")
    # Same surface as SyncClient, including submit_batch() / iter_results()
```

> **Note on naming.** Switching from `SyncClient` to `AsyncClient` only changes the local I/O loop — both classes can call every endpoint (sync and Collections) and neither is faster on its own. To scale to many URLs, the lever is `submit_batch()` / `batch_scrape()`, not which client class you instantiate. See [Choosing the right method](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/choosing-the-right-method).

### List return instead of streaming — `batch_scrape()`

Same server-side scaling as `submit_batch()`, simpler signature. Drop-in replacement for the deprecated `scrape_many()`:

```python
results = client.batch_scrape([
    {"url": u, "custom_id": product_id, "browser": True}
    for product_id, u in catalog.items()
])
for r in results:
    if r.guidance.success:
        save(r.custom_id, r.content)
```

## Pricing — pay only for usable content

| Plan | Price | Credits/mo | Rate | Concurrent |
|---|---|---|---|---|
| Demo (public) | Free | 5,000 | 30/min | 5 |
| Free | $0 | 1,000 | 30/min | 5 |
| Starter | $29 | 25,000 | 30/min | 10 |
| Growth | $69 | 100,000 | 60/min | 20 |
| Pro | $199 | 500,000 | 120/min | 50 |
| Scale | $499 | 2,500,000 | 200/min | 100 |
| Enterprise | Custom | Unlimited | 2,000/min | Custom |

1 simple request = 1 credit, 1 browser request = 5 credits. Credits are refunded automatically on failures (`is_success=false`). Anti-bot, proxy rotation, and per-country routing are included on every plan.

Check your usage at any time:

```python
client.scrape(url)
print(client.quota_remaining, "credits left")
print(client.billing())
```

## Error handling

```python
from scrapingpros import SyncClient, AuthenticationError, RateLimitError, QuotaExceededError

try:
    result = client.scrape(url)
except AuthenticationError:
    print("Invalid token — use demo_6x595maoA6GdOdVb for testing")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except QuotaExceededError:
    print("Monthly quota exceeded — upgrade your plan for more requests")
```

All SDK exceptions inherit from `ScrapingProsError`.

## Configuration

```python
client = SyncClient(
    "demo_6x595maoA6GdOdVb",            # or your dedicated token / SP_TOKEN env var
    base_url="https://api.scrapingpros.com",   # default
    timeout=120.0,                       # request timeout in seconds
    max_retries=3,                       # auto-retry on 429
)
```

## More

- **[Full documentation](https://docs.scrapingpros.com/docs/category/python-sdk)** with end-to-end recipes for batches, JS execution, viability tests, and anti-bot strategies.
- **[Release notes](https://docs.scrapingpros.com/docs/documentation/Python%20SDK/release-notes)** for every version.
- **[API reference for AI agents](https://api.scrapingpros.com/llms-full.txt)** — single-file `llms-full.txt` for Claude / GPT / Cursor.
- **MCP server** for AI agents: [Model Context Protocol](https://modelcontextprotocol.io) integration with 6 tools and anti-injection protection.

## License

MIT
