Metadata-Version: 2.4
Name: nstdata-ai-crawl
Version: 0.1.0
Summary: Python SDK for nstdata
Home-page: https://github.com/nstdata-ai/crawl-py
Author: nstdata-ai
Author-email: github@nstproxy.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.0.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# nstdata crawl SDK (Python)

[中文版](README_zh.md)

Python SDK for nstdata crawl, an advanced web scraping and crawling service.

## Installation

```bash
pip install nstdata-ai-crawl
```

## Features

- **Basic Scraping:** Get HTML and Markdown synchronously.
- **Screenshots & PDFs:** Take full-page screenshots and export pages as PDFs.
- **Browser Actions:** Orchestrate browser actions like wait for elements, fill forms, click buttons, scroll, and execute JS.
- **Custom Configuration:** Bring your own cookies, headers, and simulate mobile devices.
- **Content Filtering:** Automatically extract high-quality main content, filter out ads and navigational elements.
- **Proxy Support:** Specify proxy type (residential or datacenter) and exit country.
- **Asynchronous Execution & Polling:** Submit time-consuming tasks and poll for results.
- **Batch Crawling:** Traverse entire sites with depth and page limits.

## Quick Start

See [main.py](./main.py) for a complete list of usage examples.

```python
import os
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto, Format

# Initialize the client with your API token
TOKEN = os.getenv("NSTDATA_API_TOKEN", "YOUR_API_TOKEN")

def main():
    with NstDataClient(TOKEN) as client:
        # Submit a basic scraping task
        res = client.submit_scrape_task_sync(ScrapeRequestDto(
            url="https://example.com/",
            formats=[Format.MARKDOWN],
            timeout=60000,
        ))
        
        # Output the resulting markdown
        print(res.data.get_markdown())

if __name__ == "__main__":
    main()
```
