# WebSkrap

WebSkrap is an async-first Python scraping framework built on Playwright.

Repository: https://github.com/kacigaya/webskrap
Documentation: https://kacigaya.github.io/webskrap/
Package: https://pypi.org/project/webskrap/

## Install

```bash
pip install webskrap
python -m playwright install chromium
```

## Core API

- `WebSkrapClient`: async context manager for Playwright lifecycle.
- `WebSkrapSession`: persistent browser context wrapper.
- `SessionConfig`: browser launch and context settings.
- `BrowserProfile`: coherent browser-visible profile settings.
- `ResourcePolicy`: request blocking presets (`ALL`, `LITE`, `DOCUMENTS`).
- `FetchResult`: structured result returned from fetches.

## One-shot fetch

```python
import asyncio

from webskrap import ResourcePolicy, SessionConfig, WebSkrapClient


async def main() -> None:
    config = SessionConfig(resource_policy=ResourcePolicy.LITE)
    async with WebSkrapClient() as client:
        result = await client.fetch(
            "https://example.com",
            config=config,
            wait_until="load",
            screenshot="example.png",
        )
    print(result.status, result.final_url, result.title)
    print(result.screenshot_path)


asyncio.run(main())
```

## Persistent session

```python
import asyncio
from pathlib import Path

from webskrap import SessionConfig, WebSkrapClient


async def main() -> None:
    config = SessionConfig(user_data_dir=Path(".webskrap/session"), headless=True)

    async with WebSkrapClient() as client:
        session = await client.session("default", config=config)
        first = await session.fetch("https://example.com")
        second = await session.fetch("https://example.com/account")
    print(first.status, second.final_url)


asyncio.run(main())
```

## Patchright headless

```python
from pathlib import Path

from webskrap import SessionConfig, Viewport

config = SessionConfig(
    driver="patchright",
    channel="chrome",
    headless=True,
    user_data_dir=Path(".webskrap/headless-profile"),
    headless_screen=Viewport(width=1366, height=768),
    mask_headless_user_agent=True,
)
```

Use headed Patchright with `channel="chrome"` for the strictest stealth path.
Use `patchright_context_profile=True`, `reduce_fingerprint_surface=True`, and
`webrtc_ip_handling_policy="disable_non_proxied_udp"` only when those native
browser controls match the task.

## CLI and MCP

```bash
webskrap doctor
webskrap profiles
webskrap fetch https://example.com --output page.html --resource-policy lite
webskrap fetch https://example.com --driver patchright --channel chrome --headed --user-data-dir .webskrap/patchright-profile
webskrap-mcp
```

The MCP server exposes `fetch`, `stealth_fetch`, and `doctor`.

## Boundaries

WebSkrap does not include CAPTCHA solving, login-wall bypassing, credential bypassing, or access-control circumvention.

## Important Links

- Home: /
- Installation: /getting-started/installation/
- Quickstart: /getting-started/quickstart/
- Client: /user-guide/client/
- Sessions: /user-guide/sessions/
- Profiles: /user-guide/profiles/
- Stealth: /user-guide/stealth/
- Resource Policy: /user-guide/resource-policy/
- CLI: /user-guide/cli/
- API Reference: /api-reference/
