Metadata-Version: 2.4
Name: hej-ai-crawler
Version: 0.1.0
Summary: Scrape any webpage into clean markdown
Project-URL: Homepage, https://github.com/hej-ai/crawler-python
License-Expression: MIT
Keywords: crawler,markdown,scraper,web-scraping
Requires-Python: >=3.9
Requires-Dist: httpx>=0.24
Description-Content-Type: text/markdown

# hej-ai-crawler

Scrape any webpage into clean markdown.

## Install

```bash
pip install hej-ai-crawler
```

## Usage

```python
import asyncio
from hej_ai_crawler import crawler

scrape = crawler("your-api-key")

async def main():
    page = await scrape("https://example.com")
    print(page.title)    # "Example Domain"
    print(page.markdown) # clean markdown content
    print(page.links)    # internal links found on the page

asyncio.run(main())
```

### Scraping modes

```python
# Fast HTTP fetch + Readability extraction
await scrape("https://example.com", "static")

# Full headless browser render (default)
await scrape("https://example.com", "playwright")

# Tries static first, uses browser if content differs significantly
await scrape("https://example.com", "scout")
```

### Response

`ScrapedPage` dataclass with fields:

- `url` — final URL after redirects
- `title` — page title
- `markdown` — page content as markdown
- `links` — internal links found
- `mode` — mode used to produce the result

## License

MIT
