Metadata-Version: 2.4
Name: openadapt-capture
Version: 1.0.0
Summary: GUI interaction capture - platform-agnostic event streams with time-aligned media
Project-URL: Homepage, https://github.com/OpenAdaptAI/openadapt-capture
Project-URL: Repository, https://github.com/OpenAdaptAI/openadapt-capture
Author-email: "MLDSAI Inc." <richard@openadapt.ai>
License-Expression: MIT
License-File: LICENSE
Keywords: automation,capture,events,gui,rpa
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: alembic>=1.0.0
Requires-Dist: av>=12.0.0
Requires-Dist: fire>=0.7.1
Requires-Dist: loguru>=0.7.0
Requires-Dist: mss>=6.0.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: oa-atomacos>=3.2.0; sys_platform == 'darwin'
Requires-Dist: openai>=2.11.0
Requires-Dist: pillow>=10.1.0
Requires-Dist: psutil>=5.0.0
Requires-Dist: pydantic-settings>=2.12.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pympler>=1.0.0
Requires-Dist: pynput>=1.7.6
Requires-Dist: sounddevice>=0.5.3
Requires-Dist: soundfile>=0.13.1
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: tqdm>=4.0.0
Requires-Dist: websockets>=12.0
Provides-Extra: all
Requires-Dist: faster-whisper>=1.1.0; extra == 'all'
Requires-Dist: magic-wormhole>=0.17.0; extra == 'all'
Requires-Dist: openadapt-privacy>=0.1.0; extra == 'all'
Requires-Dist: openai-whisper>=20230314; extra == 'all'
Provides-Extra: dev
Requires-Dist: matplotlib>=3.5.0; extra == 'dev'
Requires-Dist: numpy>=1.21.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-timeout>=2.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: privacy
Requires-Dist: openadapt-privacy>=0.1.0; extra == 'privacy'
Provides-Extra: share
Requires-Dist: magic-wormhole>=0.17.0; extra == 'share'
Provides-Extra: transcribe
Requires-Dist: openai-whisper>=20230314; extra == 'transcribe'
Provides-Extra: transcribe-fast
Requires-Dist: faster-whisper>=1.1.0; extra == 'transcribe-fast'
Description-Content-Type: text/markdown

# OpenAdapt Capture

[![Build Status](https://github.com/OpenAdaptAI/openadapt-capture/actions/workflows/test.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-capture/actions/workflows/test.yml)
[![PyPI version](https://img.shields.io/pypi/v/openadapt-capture.svg)](https://pypi.org/project/openadapt-capture/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Lifecycle: Experimental.** OpenAdapt Capture records native mouse, keyboard,
and screen activity into a time-aligned local capture session. Its current
product role is the optional desktop recorder used by
[`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow), OpenAdapt's
workflow compiler and governed runtime.

Start with the [OpenAdapt documentation](https://docs.openadapt.ai/) if you want
to record, compile, verify, and replay a workflow.

## Where it fits

| Recording path | Current implementation |
| --- | --- |
| Windows and RDP demonstrations | `openadapt-capture` records native input and screen video; `openadapt-flow` converts the session into compiler input. |
| Browser demonstrations | `openadapt-flow` records its Playwright browser directly. It does not require this package. |
| Chrome extension in this repository | Experimental DOM-capture code for development; it is not the supported web recorder or governed replay path. |

The browser path stays inside `openadapt-flow` because the compiler needs
ordered before/after frames, page state, secret-field redaction, and events in
its own recording format. The extension captures useful DOM context, but it
does not provide that end-to-end contract.

## Use it with OpenAdapt

Install the compiler with the optional native recorder:

```bash
pip install "openadapt-flow[capture]"
```

Record a desktop demonstration, then compile it:

```bash
openadapt-flow record --backend windows --out recording --task "Describe the workflow"
# Perform the workflow, then press Ctrl-C.

openadapt-flow compile recording --out bundle --name my-workflow
```

Use `--backend rdp` when recording inside the RDP client pixel space. Replay
setup and substrate-specific requirements are documented in the
[`openadapt-flow` desktop recording guide](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/desktop/RECORDING.md).

## Use it as a library

Install the capture package directly:

```bash
pip install openadapt-capture
```

Record from the command line:

```bash
capture record ./my-capture --description "Describe the workflow"
# Press Ctrl-C to stop.

capture info ./my-capture
```

Or inspect processed actions in Python:

```python
from openadapt_capture import CaptureSession

with CaptureSession.load("./my-capture") as capture:
    for action in capture.actions():
        print(action.timestamp, action.type, action.x, action.y)
        frame = action.screenshot
```

A capture normally contains:

```text
my-capture/
├── recording.db
├── oa_recording-*.mp4
└── profiling.json
```

Audio and individual images are optional.

## Window-scoped recording

**Status: implemented and unit-proven on all CI platforms; live-validated
end to end on macOS (frames, translated coordinates, bounds timeline, and
video verified against a real window on a real display). Windows uses a
Win32 + `mss` region grab and is exercised by the same unit suite; its live
smoke test awaits an interactive Windows desktop. Not yet validated against
a Parallels/Citrix client window specifically.**

By default the recorder captures the full screen. Window-scoped mode records
ONE window in that window's own pixel space — the mode built for
remote-display demonstrations (Parallels, Citrix Workspace, Microsoft Remote
Desktop), where `openadapt-flow`'s `rdp_window` replay drives the client
window's pixels directly. Recording scoped to the same window removes the
full-screen-vs-window coordinate mismatch at the source:

```python
from openadapt_capture import Recorder

with Recorder(
    "./my-capture",
    task_description="Demonstrate the workflow",
    window={"owner": "Parallels", "title": None},  # substring match
) as recorder:
    input("Perform the task, then press Enter...")
```

`owner` matches the application (macOS: window owner name; Windows: process
executable name) and `title` optionally disambiguates among its windows; both
are case-insensitive substrings, mirroring how `openadapt-flow`'s
remote-display backend identifies the same window at replay time. The
selectors can also be set via config/environment
(`RECORD_WINDOW_OWNER` / `RECORD_WINDOW_TITLE`).

In this mode:

- **Frames are the target window's pixels.** macOS captures the window's own
  buffer (`CGWindowListCreateImage`, the identical call flow's replay uses);
  Windows grabs the window's screen region, so keep the window unoccluded.
- **Input coordinates are translated at capture time** into the captured
  frame's pixel space (`pixel = (global_point - window_origin) * scale`, the
  exact inverse of the replay mapping). Input outside the window records
  out-of-range coordinates rather than being silently clamped.
- **The window scoping is persisted**: the recording's config JSON carries the
  target, resolved window, initial bounds, scale, and viewport
  (`CaptureSession.window_capture`), and the window is re-resolved every
  frame with bounds changes recorded as window events — a bounds timeline
  converters can use to be exact even when the window moves.
- **Fail-loud guarantees:** recording refuses to start if the window cannot be
  resolved and captured; input arriving before the first frame is discarded
  with a warning instead of being recorded in the wrong coordinate space; a
  mid-recording window *resize* skips unencodable video frames loudly
  (screenshots and the bounds timeline stay exact) — avoid resizing the
  target during a demonstration.

Note for converters: window-mode coordinates are already in captured-frame
pixels (`coordinate_space == "window_pixels"`); do not rescale them by
`pixel_ratio`.

## Data and privacy boundary

A raw capture can contain everything visible on screen and everything typed,
including credentials, personal data, or protected health information. Keep the
entire capture directory inside its approved local boundary and apply an
appropriate retention policy.

Capture does not upload a session by default. The sharing command, remote
transcription, and profiling transfer are explicit opt-in operations. Installing
the `privacy` extra alone does not automatically scrub a recording.

The current desktop-to-Flow conversion has no field geometry for reliable
secret redaction and no live UIA locator. `openadapt-flow` therefore refuses its
desktop `--secret` option, and converted desktop workflows rely on retained
visual evidence unless a separate structural recording path arms them. Review
the desktop guide before recording sensitive workflows.

The experimental Chrome extension can observe pages across its configured host
permissions and can emit DOM text and keyboard events to a local WebSocket.
Treat it as development code; do not deploy it in a sensitive browser profile.

## Current limitations

- Native recording requires a visible user session plus the operating system's
  screen-recording and input-monitoring permissions.
- Desktop capture records pixels and coordinates, not a structural accessibility
  locator for each demonstrated target.
- The Flow adapter rejects unsupported input such as drag, non-left-click, and
  modifier-chord actions instead of silently compiling an incomplete workflow.
- Browser-extension installation, security hardening, and compiler integration
  are not part of the current product path.

See the organization-wide
[repository lifecycle registry](https://github.com/OpenAdaptAI/.github/blob/main/REPOSITORY_LIFECYCLE.md)
and [`openadapt-flow` product status](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/PRODUCT_STATUS.md)
for the evidence behind current maturity labels.

## Optional extras

| Extra | Adds |
| --- | --- |
| `transcribe-fast` | Local faster-whisper transcription |
| `transcribe` | Local openai-whisper transcription |
| `privacy` | `openadapt-privacy` dependency for explicit integrations; no automatic scrubbing |
| `share` | Explicit Magic Wormhole transfer |
| `all` | All optional dependencies |

## Development

```bash
uv sync --dev
uv run pytest -m "not slow"
```

Slow native-capture tests require a visible session and operating-system
permissions:

```bash
uv run pytest -m slow
```

## License

[MIT](LICENSE)
