Metadata-Version: 2.4
Name: scrapely-python
Version: 0.1.38
Summary: Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform
Author-email: Scrapely <hello@scrape.ly>
License-Expression: MIT AND Apache-2.0
Project-URL: Homepage, https://scrape.ly
Project-URL: Documentation, https://scrape.ly/docs/sdk-python
Project-URL: Repository, https://github.com/scrapely-app/scrapely-python
Project-URL: Issues, https://github.com/scrapely-app/scrapely-python/issues
Project-URL: Changelog, https://github.com/scrapely-app/scrapely-python/blob/main/CHANGELOG.md
Keywords: scraping,actor,serverless,scrapely
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-APACHE
License-File: NOTICE
Requires-Dist: httpx>=0.25.0
Requires-Dist: crawlee<2.0.0,>=1.8.0
Requires-Dist: websockets>=12.0
Requires-Dist: scrapely-client-python<2,>=1.0.2
Requires-Dist: pydantic>=2.0.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: wreq<0.14,>=0.13; python_version >= "3.11"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=1.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: playwright>=1.40.0; extra == "dev"
Requires-Dist: browserforge>=1.2.0; extra == "dev"
Dynamic: license-file

# Scrapely Python SDK

Build Python Actors for the [Scrapely](https://scrape.ly) platform with lifecycle helpers, Crawlee-compatible storage, platform events, remote browser CDP, run control, webhooks, and pay-per-event accounting.

## Install

Scrapely requires Python 3.11 or newer.

```bash
python -m pip install scrapely-python
```

Install the appropriate Crawlee/Playwright browser dependencies separately when your Actor uses a browser.

## Quick start

```python
import asyncio

from scrapely import Actor


async def main() -> None:
    actor_input = await Actor.get_input_or_throw()
    await Actor.push_data({
        "url": actor_input["url"],
        "status": "processed",
    })


if __name__ == "__main__":
    asyncio.run(Actor.main(main))
```

`Actor.main()` initializes the Actor, awaits your callback, closes events and storage, and exits the process. For embedded applications and tests, use `async with Actor:` instead.

## HTTP client

```python
from scrapely import Actor
from scrapely.http import AsyncClient

async with Actor:
    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
    async with AsyncClient(proxy_configuration=proxy) as http:
        r = await http.get("https://www.example.com/")
        data = (await http.get("https://www.example.com/api/items", kind="json")).json()
```

Every request uses a current browser profile: TLS, HTTP/2, User-Agent, client hints and header order all come from it. Each request runs on an identity, which is a profile, a proxy session and a cookie jar. When a site blocks an identity (429, block page, captcha), the client retires it and retries the request at once on a fresh one, with a new IP, a new profile and an empty jar. It doesn't wait; `polite=True` turns on Retry-After and backoff. Moving from curl_cffi takes one line: `from scrapely.http import AsyncSession`.

## Highlights

- Local file-system storage and Scrapely API-backed datasets, key-value stores, and request queues
- Crawlee request loaders and storage objects
- Platform event WebSocket integration and state persistence
- Automatic remote Chromium CDP wiring through `CDP_URL`
- Actor start, call, abort, reboot, and metamorph operations
- Ad-hoc webhooks, proxy configuration, and charging/pricing types
- `scrapely.http`: an HTTP client with current browser fingerprints (TLS, HTTP/2, headers), identity rotation, block detection and retries; drop-in `AsyncSession` / `Session` for curl_cffi code
- Scrapy integration in `scrapely.scrapy` (`run_scrapy_actor`, a request-queue scheduler, a dataset pipeline); install Scrapy separately
- Legacy environment variable aliases and asynchronous client access

See the [complete Python SDK guide](https://scrape.ly/docs/sdk-python) for lifecycle details, option objects, storage selection, events, CDP behavior, run control, environment variables, errors, and the full public export list.

## License

MIT (see [LICENSE](./LICENSE)), except for three files that stay under the Apache License 2.0
(see [LICENSE-APACHE](./LICENSE-APACHE) and [NOTICE](./NOTICE)).
