Metadata-Version: 2.5
Name: render-lab-tasks-scrape
Version: 0.1.0
Summary: scrape tasks for Render Workflows
Project-URL: Repository, https://github.com/render-lab/render-tasks-python
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: httpx<0.29,>=0.28
Requires-Dist: render-lab-tasks-core<0.2,>=0.1.1
Requires-Dist: render==1.0.1
Description-Content-Type: text/markdown

# render-lab-tasks-scrape

Python port of all 7 registered tasks in the pinned TypeScript pack at
`45f9c2d44bd28e01ae9813e0d2ff56ee6c533816`. Python 3.12+.

Install this distribution and its dependencies:

```sh
pip install render-lab-tasks-scrape==0.1.0
```

Import definitions from `render_lab_tasks_scrape.tasks`; combine `app` objects
with `Workflows.from_workflows(...)`. JSON fields retain TS spelling; functions
use snake_case. Package root imports are inert.

## Tasks

| Registered name | Python export (also exposes `_impl`) |
| --- | --- |
| `scrape.crawl` | `crawl` |
| `scrape.extract` | `extract` |
| `scrape.extractLinks` | `extract_links` |
| `scrape.extractMetadata` | `extract_metadata` |
| `scrape.sitemap` | `sitemap` |
| `scrape.toMarkdown` | `to_markdown` |
| `scrape.fetch` | `fetch` |

## Dependencies and retries

Raw implementations accept context first and keyword-only `deps`. The `.client`
module exports typed `Port`, `Deps`, and `Client` classes. Inject a fake port, or
pass a caller-owned `httpx.AsyncClient` to `Client`. Default dependencies close
their client after one operation. Credentials resolve at first use. Transport
retries are disabled; `retry.py` owns durable policies. DTO types are in `.types`.
Requires core >=0.1.1.

## Environment

No environment variables. Caller-provided headers carry authentication.

## Verification and limitations

The extractor is a minimal regex parser. It does not execute JavaScript, honor
robots.txt, or provide DOM selectors. It follows redirects; 429/5xx raise and other
4xx return a result. Crawl uses a breadth-first queue, defaults to ten successful
pages on the same host, and does not cap the number of failed-page requests.
Use trusted URLs and caller-chosen bounds. No Firecrawl/Browserbase adapter is included.

Shared fixtures execute the actual pinned TS implementations and compare Python
outputs and requests. Sanity's SDK calls are translated into separately asserted
REST requests. These are hermetic checks, not real account or hosted verification.
See [the testing backlog](https://github.com/render-lab/render-tasks-python/issues/1)
and the root verification tracker. Licensed under MIT.
