Metadata-Version: 2.5
Name: octoparse-relay
Version: 0.1.10
Summary: Development relay between Playwright and a WXT Chrome extension
Requires-Python: >=3.11
Requires-Dist: httpx<0.29,>=0.28
Requires-Dist: playwright>=1.64
Requires-Dist: websockets<16,>=15
Description-Content-Type: text/markdown

# Octoparse Relay

Octoparse Relay connects an existing Playwright script to the Chromium-based
browser profile where the Octoparse Relay extension is installed. Start
`relay-cli` once to pair the browser and reuse its loopback Relay across Python
runs. Without a usable CLI Relay, the SDK creates a temporary Relay as before.

## Shared Relay for agents

Install the Python package into the tool or project environment to get the
`relay-cli` entrypoint. With a matching extension built and loaded, run:

```bash
relay-cli start
relay-cli status
```

`start` launches a detached server, requests an available port, opens the browser
activation page and waits for the extension handshake. It prints JSON containing
the port and authenticated CDP URL. Calling it again reuses the running server.

Browser selection follows `--browser` > `BROWSER` > platform default:

```bash
relay-cli start --browser chrome
```

`--browser` accepts a controller name or `%s` command template supported by
Python's `webbrowser` on the current platform. Explicit selection fails instead
of falling back to a different browser. `BROWSER` retains standard-library
candidate-list behavior. To change an already paired browser configuration,
stop the Relay before starting it with the new choice.

The CLI atomically records its endpoint in the current user's cache:

- macOS: `~/Library/Caches/octoparse-relay/daemon.json`
- Windows: `%LOCALAPPDATA%/octoparse-relay/daemon.json`
- Linux: `$XDG_CACHE_HOME/octoparse-relay/daemon.json` (or `~/.cache/...`)

Both `connect()` and `connect_async()` automatically discover and check this
endpoint, then connect through the CLI's port without starting another server
or opening another activation page. Existing scripts need no changes:

```python
browser = playwright_relay.connect(p.chromium)
```

Set `OCTOPARSE_RELAY_URL=<cdp_url>` to override file-based discovery with a
local or remote Relay CLI server, for example:

```bash
export OCTOPARSE_RELAY_URL='wss://relay.example.com/relay?role=client&token=YOUR_TOKEN'
```

Relay URLs accept `ws://` or `wss://`, require `/relay`, `role=client`, and a
nonempty token. The SDK checks `/status` on the same host and port using HTTP
for WS or HTTPS for WSS. A reverse proxy must expose both `/relay` (WebSocket)
and `/status` (HTTP) at the domain root. The remote browser must already be
paired with that Relay. The CLI's loopback listener is unchanged; exposing the
remote service through a proxy or tunnel is managed separately.

The SDK still checks Relay status and falls back to a temporary local Relay if
unavailable or unpaired; a busy Relay raises an error. CDP connections use
Playwright defaults unless overridden through `**kwargs`, which both `connect()`
and `connect_async()` forward to `connect_over_cdp()`. `no_defaults` defaults to
`True` and can be overridden. `timeout` is in seconds, controls temporary Relay
pairing, and is multiplied by 1000 before being passed to CDP in milliseconds.
For example, `timeout=60` means 60 seconds for each stage.
Pairing defaults to 15 seconds when `timeout` is omitted or `None`; CDP keeps
its Playwright default. `timeout=0` disables both limits. Each stage has its own
wait limit, not a shared total deadline. Existing scripts keep their timeout
values in seconds; explicit timeouts now also apply to the CDP connection. Loopback status requests bypass proxies;
remote requests use the normal proxy settings.
The former `endpoint` argument and `OCTOPARSE_RELAY_ENDPOINT` variable have been
replaced by `OCTOPARSE_RELAY_URL`.

To connect directly to a standard Chromium CDP server instead:

```bash
export OCTOPARSE_CDP_URL='https://browser.example.com:9222'
```

Both `connect()` and `connect_async()` accept `endpoint_url` through `**kwargs`
(`connect(chromium, **kwargs)` / `connect_async(chromium, **kwargs)`):

```python
browser = playwright_relay.connect(p.chromium, endpoint_url="http://localhost:9222", timeout=30)
# Async: await playwright_relay.connect_async(p.chromium, endpoint_url="http://localhost:9222")
```

A nonempty `endpoint_url` takes priority over `OCTOPARSE_CDP_URL`. When omitted, `None`,
empty, or whitespace-only, address selection uses `OCTOPARSE_CDP_URL`, then
Relay discovery, then a temporary Relay. The selected URL is passed as the
`endpoint_url` argument to `connect_over_cdp()`; other kwargs are forwarded.
Both explicit `endpoint_url` and `OCTOPARSE_CDP_URL` bypass Relay discovery.
HTTP(S) and WS(S) CDP endpoints are supported. This mode does not check local
Relay ports, start a server, or open an extension activation page. Connection
errors propagate without falling back to a local Relay. Closing the connection
does not invoke any remote service stop/delete API.

Both URL environment variables are trimmed; empty or whitespace-only values
are treated as unset. Priority is `OCTOPARSE_CDP_URL` > `OCTOPARSE_RELAY_URL` >
the local discovery record > a temporary Relay.
`OCTOPARSE_RELAY_STATE_FILE` overrides the discovery file location.
Endpoint URLs contain credentials; treat them like passwords.

Close your own pages and Python connection after each run. Closing the client
keeps the background server and extension connection alive. One Python client
can use it at a time; a busy Relay reports an error instead of being replaced.
Separate Python executions do not share variables or event listeners. An open
Relay-owned page can be discovered again by the next client.

```bash
relay-cli stop
```

`stop` closes the server and extension pairing, preserving Chrome and its tabs.
If the server or extension connection is unavailable, the SDK falls back to its
temporary connection flow. Run `start` again to pair the background server.
The persistent client capability requires rebuilding and reloading the extension.

For source-checkout use before reinstalling the package entrypoint:

```bash
uv run python -m octoparse.playwright_relay.cli start
```

`start --no-browser` starts only the listener, for transport diagnostics. The
normal `start` command completes browser pairing. The CLI executes no Python
code; agents explore and validate scripts through the native SDK.

## Automatic AWS WAF CAPTCHA handling

Relay can resolve AWS WAF main-document image CAPTCHAs on Relay-owned tabs.
The extension handles the page SDK, challenge state, answer validation and
cookies; the Relay server calls an OpenAI-compatible vision model. There is no
popup or model credential storage in the extension. Other CAPTCHA providers,
third-party iframe CAPTCHAs and standalone Challenge solving are not supported.
Detection observes responses after tab ownership is established; it does not
reconstruct a CAPTCHA response that already occurred before an existing tab
was adopted.

Configure a persistent server with:

```bash
relay-cli start --base-url https://provider.example/v1 --api-key "" --model vision-model
```

Both persistent and temporary servers read these environment variables:

```bash
export OCTOPARSE_RELAY_LLM_BASE_URL=https://provider.example/v1
export OCTOPARSE_RELAY_LLM_API_KEY=""
export OCTOPARSE_RELAY_LLM_MODEL=vision-model
```

For each field, a nonempty environment variable overrides the corresponding
CLI argument; an empty variable falls back to the argument. The API base URL
includes `/v1` or the provider's equivalent prefix; Relay appends
`/chat/completions`. The API key is optional. An empty key omits the
Authorization header. The model must support multimodal image input and SSE
chat-completion responses.

Configuration is captured when the server starts. Restart the persistent
server to change it; a Python client borrowing an existing server uses that
server's configuration, not the client's environment. Temporary servers read
the environment of the Python process that creates them.

Missing or invalid model configuration does not prevent either server from
starting. The model channel reports `llm.error` with `Model request failed`; the extension
maps automatic CAPTCHA failures to `Failed to solve CAPTCHA automatically`. CDP and subsequent requests continue
working. A returned model answer is not proof that WAF accepted it; the
extension validates and submits it. Scripts should wait for the expected
business page; automatic solving does not extend Playwright timeouts.

Only image data and model prompts are sent to the model service. Site cookies,
tokens, vouchers and WAF verification state stay in the extension. Closing or
navigating a tab, losing Relay ownership, disconnecting the client or stopping
the server cancels its pending solving work.
An already-started page SDK `getToken()` call cannot be forcibly cancelled and
may still update the site's cookie; cancelled Relay work cannot submit a late
model answer or install its final token.

The extension requires HTTP(S) host access and the `webRequest`,
`webNavigation`, `scripting`, `cookies` and `storage` permissions in addition
to its existing debugger and tab access. Runtime ownership checks restrict
automatic solving to Relay-owned tabs; these checks do not narrow the install
permissions. Reload the rebuilt extension after updating.

## Agent skill

The Python package includes `skills/octoparse-relay/SKILL.md` starting with
version 0.1.3. After installation, locate the bundled instructions with:

```bash
relay-cli skill
```

Ask your agent to run `relay-cli skill` and read the returned `SKILL.md` before
using Relay. In a project managed by uv, use `uv run relay-cli skill`.
The command also works in a source checkout. It does not start the server or
open a browser. Agents that only discover skills in their own skill directories
still need the bundled skill directory copied or linked into that location.

## Build and install the extension

Install dependencies and build the unpacked extension:

```bash
pnpm install
pnpm build
```

In the daily Chromium-based browser profile:

1. Open `chrome://extensions` (or the browser's equivalent extensions page).
2. Enable Developer mode.
3. Choose **Load unpacked**.
4. Select this repository's `.output/chrome-mv3` directory.

`wxt.config.ts` contains one stable public `manifest.key` for the local
Developer Mode build. It keeps the extension ID stable across builds. The
private key is not part of this repository; changing or removing the checked-in
public key changes the ID and requires reinstalling the unpacked extension.

The extension only accepts activation messages from numeric loopback pages:
`http://127.0.0.1/*`. It does not connect to a Relay while idle.

## Publish the Python package

The published distribution is `octoparse-relay`. Its release version is the
`[project].version` value in `pyproject.toml`. Nexus release versions are
immutable: increase that version before publishing any changed package.

Publish only to `https://nexus.bazhuayu.com/repository/pypi-hosted/`. The
`nexus` alias used below must target this endpoint. Do not publish to
`pypi-group` or to any `/simple/` endpoint.

Build, validate, and upload both distributions:

```bash
release_dir="$(mktemp -d)"
uv build --no-sources --out-dir "$release_dir"
uvx twine check "$release_dir"/*.whl "$release_dir"/*.tar.gz

uvx twine upload --non-interactive -r nexus \
  "$release_dir"/*.whl \
  "$release_dir"/*.tar.gz
```

## Run the Playwright example

Install Python and dependencies with uv:

```bash
uv python install 3.11
uv sync --all-groups
```

Run the example from the repository root:

```bash
uv run python examples/relay_playwright.py
```

When no paired CLI Relay is available, each run performs this sequence:

```text
playwright_relay.connect()
  → bind a fresh 127.0.0.1 port
  → issue a fresh session token
  → open the local /activate URL in the system default browser
  → wake the installed extension through externally_connectable
  → wait for the extension WebSocket handshake
  → connect Playwright through the Octoparse Relay CDP endpoint
```

The example then opens `https://example.com` and prints `Example Domain`.
In this temporary mode, closing the returned Playwright browser closes the Octoparse Relay and sends the
extension back to its idle state. A later `playwright_relay.connect()` creates a new port
and token; no previous Octoparse Relay URL or Profile path is saved.

Relay always uses the native browser-window viewport. It ignores Playwright
viewport overrides, including later calls that try to resize a page.

Relay intentionally supports top-level page targets and iframe DOM frames,
including out-of-process iframes. It does not expose worker, shared-worker, or
service-worker targets through its CDP bridge.

The default browser must be a Chromium-based browser with this unpacked
extension installed in the Profile that receives externally opened URLs. The
SDK does not scan or persist browser Profiles. If the extension is missing,
disabled, or the URL opens in another Profile, `playwright_relay.connect()` times out
waiting for the authoritative extension handshake.

## Development browser

`pnpm dev` remains useful for WXT extension development and opens a separate
development browser Profile. The Developer Mode distribution test above is the
production-shaped path: build first, load `.output/chrome-mv3` into the daily
Profile, then run the Python example.

### Relay connection protocol

Both peers use `/relay` on the same loopback port. The extension connects with
`role=extension`; Playwright connects with `role=client`. Both URLs carry the
same fresh `session_token` in the `token` query parameter. Temporary Relays
accept one connection per role. CLI Relays keep one extension connection and
accept sequential CDP clients, resetting each client's debugger state while
retaining its physical pages. The token authorizes the session; roles
select the peer protocol and do not provide separate credential permissions.
Update the SDK and extension together.

The client sends native CDP requests directly. Extension control messages use
`type` as their message discriminator; the extension handshake uses `protocol`
for its version.
