Metadata-Version: 2.5
Name: dataapp-crawler-octoparse-browser
Version: 0.1.0
Summary: Octoparse browser providers for DataApp Crawler
Requires-Python: >=3.11
Requires-Dist: dataapp-crawler<0.2,>=0.1.0
Requires-Dist: octoparse-relay<0.2,>=0.1.10
Requires-Dist: playwright<2,>=1.64
Provides-Extra: managed
Requires-Dist: dataapp-resource-proxy<0.2,>=0.1.0; extra == 'managed'
Description-Content-Type: text/markdown

# Octoparse Crawler SDK

Octoparse browser providers for DataApp Crawler.

The accompanying browser-image source is maintained in
[`browserenv/`](browserenv/README.md). Python provider packages and the BrowserEnv
image share this repository but are built and released separately; BrowserEnv is
excluded from the Python wheel and sdist.

Install the browser package from the DataOS development index. It installs
Crawler and the Relay Python transport as dependencies. The Python import path
is `dataapp_crawler_octoparse`.

## Install

```bash
pip install --extra-index-url https://nexus.bazhuayu.com/repository/pypi-dataos-dev/simple/ \
  dataapp-crawler-octoparse-browser==0.1.0.dev4
```

## Providers

- `BrowserEnvRelay` connects to the Relay browser supplied by the DataApp Relay image.
- `RelayBrowser` starts and owns a local browser, Relay extension, and Relay server.

```python
from dataapp_crawler_octoparse import BrowserEnvRelay

browser = BrowserEnvRelay()
```

The SDK's BrowserEnv image bundles CloakBrowser and runs it in headed mode,
using Xvfb when needed. Applications connect through `BrowserEnvRelay`.

`RelayBrowser` is an advanced self-managed path, not the Relay Data App template
default. A fresh post-fix local candidate batch passed full Relay pairing,
HTTP/HTTPS DOM work, owned-process cleanup, and container cleanup with Chrome
151 in headless and Xvfb modes, CloakBrowser 146 headless, and Chromium 156
headless. Earlier Chrome pairing and CloakBrowser headless failures remain useful
pre-fix history, not the current result. These are versioned local candidate
checks, not universal browser or target-site guarantees; proxy combinations were
not repeated across every browser. The published clean `rc.3` template image is
a separate release artifact from the locally tested
`dataos-octoparse-integration:20261108-managed` candidate.

BrowserEnv proxy control uses `DATAAPP_BROWSER_PROXY_API_URL`,
`DATAAPP_BROWSER_PROXY_API_USERNAME`, and `DATAAPP_BROWSER_PROXY_API_PASSWORD`.
It also accepts the legacy `OCTOPUS_PROXY_SOFTWARE_API_*` settings when their
canonical equivalents are absent. The control endpoint must be HTTP on
loopback. An upstream `https://` proxy uses TLS with certificate and hostname
verification.

In a DataHub Run, choose `DataOSProxy(DataApp.resources())` as the
`DataAppCrawler` proxy provider to attribute browser traffic to the Run.
Select a lease duration that covers the browser work. Renewal depends on the
configured supplier.

## CAPTCHA and images

AWS WAF image selection and controlled extension grants are internal C10/C07
experiments. The current platform Runtime does not expose their routes.
Direct local Resource Proxy experiments require matching Relay, extension and
server candidates. Local protocol checks do not establish real-site submission,
clearance, data collection or availability through a platform Run.

CAPTCHA solving is opt-in. Install the `managed` extra with the local candidate
Resource Proxy client. In local development, open `ResourceProxy.from_env()`;
platform applications use `DataApp.resources()` when the corresponding
Runtime/DataHub routes are enabled. Model routes still require that platform
integration; local support does not imply availability on pre.

For local managed-resource testing, save the current environment's DataOS API
Key in a regular text file containing exactly one non-empty line, then run:

```bash
python -m dataapp_crawler run --api-key-file /path/to/api-key.txt -- python app.py
```

The source file cannot be a symlink or directory. The local launcher reads it locally and injects it only into the child process. Resource Proxy
receives the normal Authorization header, never the source file. Do not paste the
Key into chat or package the source file with the App. Platform Runs use the
injected Runtime identity.

The typed API accepts embedded Turnstile, reCAPTCHA v2, Enterprise v2, and
image-to-text tasks. Widget tasks return `.token`; image tasks return `.text`.
Module selection chooses a recognizer for a known image technology, rather
than detecting what a website uses:

```python
from dataapp_crawler.images import capture_image

image = await context.run(capture_image(page.locator("#captcha-image")))
task = await context.run(resources.solve_captcha(
    type="image_to_text",
    image_body=image.image_body,
    capmonster_module="botdetect",  # Only for a known BotDetect challenge.
))
```

`capture_image()` defaults to a visible rendered-element capture and does not
request a mutable image URL again. Chromium uses a clipped CDP screenshot when
available and otherwise falls back to the native Locator screenshot; hidden or
zero-size elements fail safely. When the developer has established that a
loaded `IMG` is the current challenge, `source="loaded_image"` explicitly copies
its pixels through Canvas without another GET and remains subject to cross-origin
pixel restrictions. The same image's `data_url` can be passed to
`resources.chat_completion()` as an inline model image. These small helpers do
not detect, solve, submit, or verify a challenge, and they do not turn `process`
into a processing framework. The developer still checks the
challenge identity, applies the candidate, checks the site verdict, and extracts
business data. Use the matching version's capability acceptance matrix to check
which task and site flows have been validated.

In the final real Relay check, default capture rejected the hidden zero-size IMG,
while explicit `loaded_image` returned its current PNG bytes without another
request. A visible wrapper capture no longer timed out, but that page's wrapper
pixels were blank; a successful screenshot therefore does not prove it contains
the challenge image or that a solver or site will accept it.

The managed cloud API through `DataApp.resources()` and Resource Proxy is the
default public solving path. Application code must explicitly create a bounded
task after the developer has enabled metered solving; CAPTCHA detection alone
never creates a paid task. In a local Runtime/DataHub resource-bridge acceptance,
the managed cloud API
passed one public reCAPTCHA Enterprise v2 demonstration form through real token
creation, browser submission, server acceptance, and Dataset output. BotDetect
image and vision-model candidates were submitted but rejected by the
demonstration site. Turnstile and regular reCAPTCHA v2 did not receive the same
acceptance test. Detection only classifies evidence; it does not solve a
challenge. None of these results establishes success on a business site.

### Native CapMonster extension with a developer-owned key (advanced opt-in)

For trusted local development, a developer may explicitly configure
BrowserEnv's bundled native CapMonster extension before browser startup:

```bash
# CAPMONSTER_API_KEY is already loaded from the developer's local secret store.
export CAPMONSTER_API_KEY
export CAPMONSTER_CAPTCHA_TYPES='ReCaptcha2,ReCaptchaEnterprise,Turnstile'
xvfb-run -a octoparse-relay-task -- uv run python app.py
```

`CAPMONSTER_CAPTCHA_TYPES` is a comma-separated allowlist. Each selected type
must be supported by the extension bundle in the chosen image. The key belongs
to the developer and is not a shared platform key. Do not commit it, pass it as
a build argument, write it to logs or Dataset records, or expose it in errors.
The launcher prepares a private extension copy before browser startup and removes
the two variables from the child application environment; the key still enters
the third-party extension context.

This BYOK path is currently limited to trusted local sessions. The launcher
rejects raw CapMonster keys when DataHub platform identity is present, so this
configuration is not a supported production DataApp path. A Resource Proxy
grant-backed automatic extension integration is under development and is not a
public product API. Use the explicit managed cloud API where platform credential
isolation is required. Do not run native automatic solving and explicit API
solving against the same challenge. Native Enterprise end-to-end site acceptance
has not yet been established.

## Documentation and examples

The Relay project template ships the following together:

- [Developer guide](../../examples/crawler/relay_template/help/docs/DEVELOPER_GUIDE.zh-CN.md):
  browser lifecycles, extension input behavior, fingerprints, proxies, and limits.
- [API reference](../../examples/crawler/relay_template/help/docs/API.zh-CN.md): Provider,
  Crawler helper, and advanced low-level Relay signatures and return values.
- [Importable scenarios](../../examples/crawler/relay_template/help/examples/README.md):
  pointer observation, scoped navigation, readiness, and extraction.
- [Complete search/list/detail flow](../../examples/crawler/relay_template/site_adapter.py):
  the site's business flow used by the Runtime `app.py` entry.

`Page` and `Locator` remain native Playwright objects. Pointer trajectory handling
is implemented by the extension; it is not an additional Python `human_click()`
method. Browser fingerprint behavior belongs to the selected browser environment.
