Metadata-Version: 2.4
Name: scraping-ai
Version: 0.4.1
Summary: Python SDK for the Scraping AI data extraction pipeline service
Author: SDT Bizdev Team
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Requires-Dist: respx>=0.21.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"

# Scraping AI Python SDK (`scraping-ai`)

A modern, type-safe Python SDK for interacting with Scraping AI service API endpoints. It supports both synchronous and asynchronous operations using `httpx` and `pydantic`.

This SDK focuses strictly on **core service execution endpoints** (pipeline orchestration, web extraction, crawling, finder, keywords, etc.), excluding account management, billing, payments, and administrative routes.

## Installation

```bash
pip install scraping-ai
```

## Quick Start (High-Level 1-Line Helpers)

### 1. Web Extraction in 1 Line

```python
from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Extract web data directly in one step (polls until finished)
data = client.extract(
    url="https://example.com",
    schema={"title": "string", "price": "number"}
)
print(data.results)
```

### 2. Page Crawling in 1 Line

```python
# Crawl a web page and get extracted content & markdown
crawl_result = client.crawl(url="https://example.com", use_browser=True)
print(crawl_result.results)
```

### 3. Discover URLs in 1 Line

```python
# Discover URLs on a target site up to a specific depth
found_urls = client.find_urls(base_url="https://example.com", max_depth=2)
print(found_urls.results)
```

### 4. Keyword Generation in 1 Line

```python
# Generate relevant search keywords for a topic
keywords_res = client.generate_keywords(context="dummy_context", num_keywords=15)
print(keywords_res.keywords)
```

### 5. Schema Generation in 1 Line

```python
# Generate JSON Schema for an extraction prompt
schema_res = client.generate_schema(schema_instruction="Extract title, author, date")
print(schema_res.schema_)
```

---

## Asynchronous Usage

```python
import asyncio
from scraping_ai import AsyncScrapingAIClient

async def main():
    async with AsyncScrapingAIClient(api_key="your_api_key_here") as client:
        # Async one-liner extraction
        data = await client.extract(url="https://example.com")
        print(data.results)

asyncio.run(main())
```

---

## Service Modules Overview

The SDK exposes low-level module controllers under clean, intuitive client attributes:

- **1-Line Helpers**: `client.extract(...)`, `client.crawl(...)`, `client.find_urls(...)`, `client.generate_keywords(...)`, `client.generate_schema(...)`.
- `client.extractor`: Data extraction module (`run`, `status`, `result`).
- `client.crawler`: Page crawling module (`run`, `status`, `result`).
- `client.finder`: URL discovery module (`run`, `status`, `result`).
- `client.keywords`: Keyword generation module (`run`, `status`, `result`).
- `client.schema`: JSON Schema generation module (`run`, `status`, `result`).
- `client.ranker`: Relevance ranking module (`run`, `status`, `result`).
- `client.search_url`: Search URL generator module (`run`, `status`, `result`).
- `client.search_results`: Search results discovery module (`run`, `status`, `result`).
- `client.exports`: S3 exports management (`create`, `list`, `get`).
- `client.pipeline`: Multi-step pipeline orchestration (`create`, `list`, `get`, `patch`, `run_flow`, `wait_for_task`).
- `client.data`: Query extracted records (`get_by_state`, `export_by_state`).
