Metadata-Version: 2.4
Name: scraping-ai
Version: 0.4.0
Summary: Python SDK for the Scraping AI data extraction pipeline service
Author: SDT Bizdev Team
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Requires-Dist: respx>=0.21.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"

# Scraping AI Python SDK (`scraping-ai`)

A modern, type-safe Python SDK for interacting with Scraping AI service API endpoints. It supports both synchronous and asynchronous operations using `httpx` and `pydantic`.

This SDK focuses strictly on **core service execution endpoints** (pipeline orchestration, web extraction, crawling, finder, keywords, etc.), excluding account management, billing, payments, and administrative routes.

## Installation

```bash
pip install scraping-ai
```

## Quick Start

### 1. One-Line Data Extraction

```python
from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Extract web data directly in one step
data = client.extract(
    url="https://example.com",
    schema={"title": "string", "price": "number"}
)
print(data)
```

### 2. Standalone Service Modules (Keywords, Crawler, Finder, Extractor, etc.)

```python
import asyncio
from scraping_ai import AsyncScrapingAIClient

async def main():
    async with AsyncScrapingAIClient(api_key="your_api_key_here") as client:
        # Start keyword generation
        run_info = await client.keywords.run(context="dummy_context", num_keywords=10)
        print(f"Accepted run for State ID {run_info.state_id}")

        # Check status
        status_info = await client.keywords.status(run_info.state_id)
        print(f"Current status: {status_info.status}")

        # Retrieve result when finished
        res = await client.keywords.result(run_info.state_id)
        print(f"Keywords generated: {res.keywords}")

asyncio.run(main())
```

### 3. Pipeline Orchestration

```python
from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Create a multi-step pipeline job
state = client.pipeline.create(
    base_url="https://example.com",
    user_instruction="dummy_user_instruction",
    schema_instruction="dummy_schema_instruction"
)
flow = client.pipeline.run_flow(state.id)
print(f"Pipeline started: {flow.first_step}")
```

## Service Modules Overview

The SDK exposes all service operations under clean client modules:

- `client.extract(url, ...)`: One-step extraction helper.
- `client.generate_keywords(context, ...)`: One-step keyword generation helper.
- `client.pipeline`: Pipeline creation, orchestration, auto-flow, and status polling.
- `client.keywords`: Standalone keyword generation module (`run`, `status`, `result`).
- `client.finder`: URL discovery module (`run`, `status`, `result`).
- `client.crawler`: Page crawling module (`run`, `status`, `result`).
- `client.extractor`: Data extraction module (`run`, `status`, `result`).
- `client.ranker`: Relevance ranking module (`run`, `status`, `result`).
- `client.schema`: JSON Schema generation module (`run`, `status`, `result`).
- `client.search_url`: Search URL generator module (`run`, `status`, `result`).
- `client.search_results`: Search results discovery module (`run`, `status`, `result`).
- `client.exports`: S3 exports management (`create`, `list`, `get`).
- `client.data`: Query extracted records (`get_by_state`, `export_by_state`).
