Metadata-Version: 2.5
Name: openadapt-capture
Version: 1.3.0
Summary: Record desktop mouse, keyboard, and screen activity, then load it from Python
Project-URL: Homepage, https://github.com/OpenAdaptAI/openadapt-capture
Project-URL: Repository, https://github.com/OpenAdaptAI/openadapt-capture
Author-email: "MLDSAI Inc." <richard@openadapt.ai>
License-Expression: MIT
License-File: LICENSE
Keywords: automation,capture,events,gui,rpa
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: alembic>=1.20.0
Requires-Dist: fire>=0.7.1
Requires-Dist: loguru>=0.7.0
Requires-Dist: matplotlib>=3.10.9
Requires-Dist: mss>=6.0.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: pillow>=10.1.0
Requires-Dist: psutil>=5.0.0
Requires-Dist: pydantic-settings>=2.12.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pympler>=1.0.0
Requires-Dist: pyobjc-framework-applicationservices>=12.2.1; sys_platform == 'darwin'
Requires-Dist: pyobjc-framework-screencapturekit>=12.2.1; sys_platform == 'darwin'
Requires-Dist: pywinauto>=0.6.9; sys_platform == 'win32'
Requires-Dist: sounddevice>=0.5.3
Requires-Dist: soundfile>=0.13.1
Requires-Dist: sqlalchemy>=2.0.54
Requires-Dist: tqdm>=4.0.0
Requires-Dist: xcffib>=1.8.0; sys_platform == 'linux'
Provides-Extra: all
Requires-Dist: faster-whisper>=1.1.0; extra == 'all'
Requires-Dist: magic-wormhole>=0.17.0; extra == 'all'
Requires-Dist: openadapt-privacy>=0.1.0; extra == 'all'
Requires-Dist: openai-whisper>=20230314; extra == 'all'
Requires-Dist: pygobject<3.51,>=3.46; (sys_platform == 'linux') and extra == 'all'
Provides-Extra: dev
Requires-Dist: cryptography>=42.0.0; extra == 'dev'
Requires-Dist: keyring>=24.0.0; extra == 'dev'
Requires-Dist: numpy>=1.21.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-timeout>=2.4.0; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: linux
Requires-Dist: pygobject<3.51,>=3.46; (sys_platform == 'linux') and extra == 'linux'
Provides-Extra: privacy
Requires-Dist: openadapt-privacy>=0.1.0; extra == 'privacy'
Provides-Extra: shadow
Requires-Dist: cryptography>=42.0.0; extra == 'shadow'
Requires-Dist: keyring>=24.0.0; extra == 'shadow'
Provides-Extra: share
Requires-Dist: magic-wormhole>=0.17.0; extra == 'share'
Provides-Extra: transcribe
Requires-Dist: openai-whisper>=20230314; extra == 'transcribe'
Provides-Extra: transcribe-fast
Requires-Dist: faster-whisper>=1.1.0; extra == 'transcribe-fast'
Description-Content-Type: text/markdown

# openadapt-capture

[![Build Status](https://github.com/OpenAdaptAI/openadapt-capture/actions/workflows/test.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-capture/actions/workflows/test.yml)
[![PyPI version](https://img.shields.io/pypi/v/openadapt-capture.svg)](https://pypi.org/project/openadapt-capture/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/LICENSE)

Record what a person does on a Windows or macOS desktop, then read it back in
Python. Capture saves mouse clicks, drags, scrolls, keystrokes, and screen
video to one local folder, and pairs every action with the screen frame from
the moment it happened.

Use it when you need real demonstrations of GUI work: to build an automation
or replay tool, to collect training or test data for a computer-use model, or
to study how a task gets done. [OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt)
uses it to record the demonstrations that it compiles into replayable
programs.

**Status:** maintained. OpenAdapt uses it to record the demonstrations it
compiles. Everything stays in a local folder.

| Platform | Records | Notes |
|---|---|---|
| macOS | Mouse, keyboard, screen video, accessibility | Needs Screen Recording, Input Monitoring, and Accessibility permissions |
| Windows | Mouse, keyboard, screen video, accessibility | Needs a signed-in desktop session |
| Linux | Mouse, keyboard, screen video; accessibility with the `linux` extra | X11 only, not Wayland |
| Remote desktop | Pixels only | No accessibility data across the connection |

## Quick start

You need Python 3.10 or later, and before the first recording:

- **FFmpeg.** Capture writes video with the `ffmpeg` executable on your
  `PATH`. If you don't have one, install FFmpeg with your package manager. On
  Capture 1.3.0 or later, `capture install-ffmpeg` can instead install a
  pinned, checksum-verified build for your user account.
- **macOS.** In System Settings > Privacy & Security, allow your terminal app
  under Screen Recording, Input Monitoring, and Accessibility. Then quit and
  reopen the terminal.
- **Windows.** Run from a signed-in desktop session.

```bash
pip install openadapt-capture
capture record ./my-capture --description "Rename a file"
```

If FFmpeg or a permission is missing, `capture record` stops before it records
and says what to fix. Otherwise, do the task. To stop, press Ctrl+C in the
terminal, or press Ctrl three times in any app. Then look at what you
recorded:

```bash
capture info ./my-capture
```

```python
from openadapt_capture import CaptureSession

with CaptureSession.load("./my-capture") as capture:
    for action in capture.actions():
        print(action.timestamp, action.type, action.x, action.y, action.text)
        frame = action.screenshot  # PIL image of the screen at that moment
```

## What a recording contains

```text
my-capture/
├── recording.db                     # actions, raw input events, window metadata (SQLite)
├── oa_recording-*.mp4               # screen video
├── capture-state.json               # whether the session finished and passed its integrity check
├── capture-artifact-manifest.json   # size and SHA-256 of every other file
├── capture-terminal.json            # completion record that covers the manifest
└── *.json                           # profiling data, and sign-in markers if you used them
```

Each action has a type (click, double-click, drag, scroll, typed text, or
keyboard shortcut), screen coordinates, any typed text, and a timestamp.
`action.screenshot` and `action.after_screenshot` give the frames before and
after it.

Where the operating system exposes it, Capture also stores the accessibility
element under each action: its role, name, bounds, available actions, and
window, from Windows UI Automation or macOS Accessibility. Read it with
`action.structural_observation`. Capture never guesses these fields from
pixels. Turn this off with `Recorder(..., capture_structural_observations=False)`.

## Record from Python

```python
from openadapt_capture import Recorder

with Recorder("./my-capture", task_description="Rename a file") as recorder:
    recorder.wait_for_ready()
    input("Do the task, then press Enter...")
```

To record one window instead of the whole desktop, pass
`window={"owner": "Parallels", "title": None}` (substring match), or use
`capture record --window-owner Parallels`. Coordinates are then in that
window's own pixels, which is what remote-desktop clients such as RDP and
Citrix need. Details: [docs/WINDOW_CAPTURE.md](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/docs/WINDOW_CAPTURE.md).

To control a running recording from another process:

```bash
capture status --session-id SESSION_ID
capture stop --session-id SESSION_ID
```

`capture record` prints the session ID when it starts. From Python, use
`status_recording(session_id)` and `stop_recording(session_id, timeout=60)`.
`stop` waits until the video and database are written and checked, and
returns an error if anything is incomplete.

If the task includes signing in, wrap the sign-in in an authentication
handoff. Capture then stops recording the screen, input, and accessibility
data until the sign-in ends, and stores only a marker:

```python
handoff = recorder.begin_authentication(
    methods="password_manager",
    requires_user_presence=True,
)
# Sign in.
recorder.end_authentication(handoff, outcome="completed")
```

See [docs/AUTHENTICATION_HANDOFF.md](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/docs/AUTHENTICATION_HANDOFF.md).

## Use it with OpenAdapt

To compile a recording into a program that replays the task, install
OpenAdapt with this package as an extra. OpenAdapt needs Python 3.10, 3.11,
or 3.12.

```bash
pip install "openadapt[capture]"
openadapt flow record --backend windows --out recording --task "Rename a file"
openadapt flow compile recording --out bundle --name rename-file
```

Other platforms, replay, and remote desktops are covered in the
[desktop recording guide](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/desktop/RECORDING.md).
The compiler accepts left and right clicks, left-button drags, typed text,
named keys, key combinations, and scrolling. It rejects a recording that
contains anything else, such as a middle click.

## FFmpeg

`capture install-ffmpeg` downloads one pinned FFmpeg build for your platform,
checks its SHA-256 against a digest stored in this package, and installs it
under your user data folder. Builds exist for macOS (Apple silicon and Intel),
Linux x86-64, and Windows x86-64. `--dry-run` prints the URL, digest, and
destination without downloading anything, and `capture uninstall-ffmpeg`
removes the build. The package itself contains no FFmpeg, and Capture
downloads nothing unless you run that command.

An FFmpeg you already set up takes priority. Capture looks in this order:
`Recorder(ffmpeg_path=...)` or `OPENADAPT_FFMPEG_PATH`, then
`OPENADAPT_DESKTOP_FFMPEG_PATH`, then OpenAdapt Desktop's runtime, then your
`PATH`, and the installed build last. Before recording, Capture runs a short
encode-and-decode test and refuses to start if it fails.

## Microphone narration

`capture record --audio` also records your voice and transcribes it on this
computer. It's off by default. It needs a local transcription backend:

```bash
pip install "openadapt-capture[transcribe-fast]"
```

Capture refuses `--audio` before it opens the microphone if no local backend
is installed, and it deletes the audio after transcription unless you set
`RECORD_AUDIO_RETAIN_WAVEFORM`.

## Privacy

A recording can contain anything that was on screen or typed, including
passwords and personal data. It stays on your computer. Capture uploads
nothing unless you run `capture share` or set `send_profile=True`. The
`privacy` extra installs `openadapt-privacy` but doesn't remove anything from
a recording by itself.

Shadow-mode pattern mining, which notices repeated tasks and offers a draft,
is off by default, needs the person's acknowledgement to turn on, and keeps
no typed values. Its encrypted buffer lives in the unsynced app-data folder
and is cleared, with its key destroyed, at least every 24 hours and when
shadow mode stops. OpenAdapt never sends shadow mode data off the computer,
but OS backup, sync, and endpoint security tools can still copy local files.
See
[docs/SHADOW_MINING.md](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/docs/SHADOW_MINING.md).

How Capture handles sensitive data, its licensing boundaries, and how releases
are qualified are covered in [docs/COMPLIANCE.md](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/docs/COMPLIANCE.md).

## Limits

- Recording needs a signed-in desktop session and the permissions listed in
  the quick start.
- On Linux, recording needs an X11 session. Capture doesn't record under
  Wayland. It saves the text of a key only when an application receives the
  key. After a click or key that no application receives (for example,
  typing with no window focused), the rest of that recording saves each
  key's code but not its text. `scripts/linux_xvfb_quickstart.sh` runs the
  quick start on Linux in a container.
- Accessibility data appears only where the application exposes it. Across a
  remote-desktop connection, Capture sees pixels only.
- Changing a display (connecting a monitor, rotating, or changing resolution
  or scale) ends the recording. Start a new one afterward.
- These need Capture 1.3.0 or later: `capture install-ffmpeg`,
  `capture status` and `capture stop` (and `status_recording` and
  `stop_recording`), authentication handoffs, `action.after_screenshot`, and
  the `capture-*.json` files. If `pip install --upgrade openadapt-capture`
  gives you an earlier version, install from GitHub:
  `pip install "git+https://github.com/OpenAdaptAI/openadapt-capture"`.
- For web apps, use OpenAdapt's browser recorder. The Chrome extension in
  [`experimental/`](https://github.com/OpenAdaptAI/openadapt-capture/tree/main/experimental) is an unfinished prototype and isn't
  part of the package.

## Extras

| Extra | Adds |
| --- | --- |
| `transcribe-fast` | Local transcription with faster-whisper |
| `transcribe` | Local transcription with openai-whisper |
| `privacy` | The `openadapt-privacy` library, for your own integration |
| `share` | `capture share`, using Magic Wormhole |
| `linux` | Accessibility observation on Linux (AT-SPI) |
| `all` | All of the above |

## Development

```bash
uv sync --extra dev
uv run pytest -m "not slow"
```

The slow tests record for real, so they need a desktop session and the
operating system permissions above:

```bash
uv run pytest -m slow
```

[docs/DESIGN.md](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/docs/DESIGN.md) describes the recording pipeline.

## License

[MIT](https://github.com/OpenAdaptAI/openadapt-capture/blob/main/LICENSE)
