Metadata-Version: 2.4
Name: vdisplay
Version: 0.1.8
Summary: Cross-platform virtual display orchestration with virtual and mirror sessions
Author: Tom
Author-email: Tom Sapletta <tom@sapletta.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/wronai/vdisplay
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: linux
Provides-Extra: windows
Provides-Extra: macos
Provides-Extra: pillow
Requires-Dist: Pillow>=10.0; extra == "pillow"
Provides-Extra: sampler
Requires-Dist: Pillow>=10.0; extra == "sampler"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: Pillow>=10.0; extra == "dev"
Requires-Dist: fastapi>=0.110; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: goal>=2.1.0; extra == "dev"
Requires-Dist: costs>=0.1.20; extra == "dev"
Requires-Dist: pfix>=0.1.60; extra == "dev"
Requires-Dist: dsl2vdisplay; extra == "dev"
Requires-Dist: vdisplay-agent; extra == "dev"
Requires-Dist: uvicorn>=0.27; extra == "dev"
Requires-Dist: pydantic>=2; extra == "dev"
Requires-Dist: sqlmodel>=0.0.22; extra == "dev"
Provides-Extra: core
Requires-Dist: pydantic>=2; extra == "core"
Requires-Dist: tenacity>=8.0; extra == "core"
Requires-Dist: structlog>=24.0; extra == "core"
Provides-Extra: control
Requires-Dist: dsl2vdisplay; extra == "control"
Requires-Dist: nlp2vdisplay; extra == "control"
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == "browser"
Provides-Extra: terminal
Requires-Dist: pyte>=0.8.1; extra == "terminal"
Requires-Dist: pexpect>=4.9; extra == "terminal"
Requires-Dist: wcwidth>=0.2.13; extra == "terminal"
Provides-Extra: agent
Requires-Dist: vdisplay-agent; extra == "agent"
Requires-Dist: fastapi>=0.110; extra == "agent"
Requires-Dist: uvicorn>=0.27; extra == "agent"
Requires-Dist: sqlmodel>=0.0.22; extra == "agent"
Provides-Extra: img2nl
Requires-Dist: img2nl[analyze]; extra == "img2nl"
Dynamic: license-file

# vdisplay


## AI Cost Tracking

![PyPI](https://img.shields.io/badge/pypi-costs-blue) ![Version](https://img.shields.io/badge/version-0.1.8-blue) ![Python](https://img.shields.io/badge/python-3.9+-blue) ![License](https://img.shields.io/badge/license-Apache--2.0-green)
![AI Cost](https://img.shields.io/badge/AI%20Cost-$4.57-orange) ![Human Time](https://img.shields.io/badge/Human%20Time-4.7h-blue) ![Model](https://img.shields.io/badge/Model-openrouter%2Fqwen%2Fqwen3--coder--next-lightgrey)

- 🤖 **LLM usage:** $4.5681 (7 commits)
- 👤 **Human dev:** ~$475 (4.7h @ $100/h, 30min dedup)

Generated on 2026-06-09 using [openrouter/qwen/qwen3-coder-next](https://openrouter.ai/qwen/qwen3-coder-next)

---

Cross-platform **virtual display orchestration API** for Python.

One unified API, multiple OS backends with different capabilities. Monitors and windows include an **`nl`** field — a natural-language description of what they contain.

CLI, DSL, REST, MCP, and the local **vdisplay-agent** broker all route through **`application.executor`** and shared **application services** (`src/vdisplay/application/`).

## Quick start

```bash
pip install "vdisplay[pillow]"
# or from source (recommended for development):
pip install -e ".[pillow,dev]"
pip install -e "packages/vdisplay-agent[serve]"
pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay
```

```bash
unset DISPLAY   # optional: auto-resolves host display to :0
vdisplay all
vdisplay monitors
vdisplay windows --apps-only
```

## Documentation

| Doc | Description |
|-----|-------------|
| [docs/index.md](docs/index.md) | Documentation hub — start here |
| [docs/installation.md](docs/installation.md) | System and Python setup |
| [docs/docker-guide.md](docs/docker-guide.md) | Running in Docker |
| [docs/examples.md](docs/examples.md) | All usage examples by environment |
| [docs/troubleshooting.md](docs/troubleshooting.md) | Common errors and fixes |
| [docs/agent-broker.md](docs/agent-broker.md) | **vdisplay-agent** — install-once broker, env vars, REST/MCP |
| [docs/architecture.md](docs/architecture.md) | CommandRequest + executor routing (local vs agent) |
| [examples/README.md](examples/README.md) | Runnable example projects |
| [packages/README.md](packages/README.md) | Control layer — DSL, MCP, REST, NL, agent |

## Examples

| Example | Mode | Host X11 | Run |
|---------|------|----------|-----|
| [headless-virtual](examples/headless-virtual/) | virtual | No | `cd examples/headless-virtual && docker compose up --build` |
| [agent-broker](examples/agent-broker/) | broker | No | `cd examples/agent-broker && ./run.sh` |
| [ci-agent](examples/ci-agent/) | virtual | No | `cd examples/ci-agent && docker compose run --rm ci-agent` |
| [dev-workspace](examples/dev-workspace/) | dev | No | `cd examples/dev-workspace && docker compose run --rm dev` |
| [host-mirror](examples/host-mirror/) | mirror | Yes | `cd examples/host-mirror && ./run.sh` |
| [host-relay](examples/host-relay/) | relay | Yes | `cd examples/host-relay && ./run.sh` |
| [run_all_examples.sh](examples/run_all_examples.sh) | mixed | varies | `./examples/run_all_examples.sh` |

Details: [docs/examples.md](docs/examples.md) · per-example READMEs in each folder above.

## List monitors, windows, and display state

Each monitor and window includes **`nl`** — a human-readable summary of what it contains.

```bash
# everything — monitors + application windows + adopted (off-screen)
vdisplay all
vdisplay all | jq '{monitors: .monitor_count, windows: .window_count, adopted: .adopted_count}'

# monitors only (xrandr) — name, geometry, rotation, primary, nl
vdisplay monitors
vdisplay monitors | jq '.monitors[] | {name, primary, nl}'

# application windows only — title, app, pid, class, nl
vdisplay windows --apps-only
vdisplay windows --app "Firefox"
vdisplay windows --class jetbrains-toolbox --pid 32977
vdisplay windows --min-width 400 --min-height 300

# platform capabilities + monitors
vdisplay info

# diagnose DISPLAY, socket, monitor count
vdisplay diagnose

# adopted (off-screen) windows — persisted in ~/.cache/vdisplay/
vdisplay relay list
```

| Command | Shows |
|---------|--------|
| `vdisplay all` | monitors + windows + adopted |
| `vdisplay monitors` | connected displays only |
| `vdisplay windows` | visible application windows only |
| `vdisplay relay list` | windows moved off-screen by relay |
| `vdisplay info` | platform capabilities |
| `vdisplay diagnose` | DISPLAY diagnostics |

`vdisplay outputs` still works but is **deprecated** — use `vdisplay all` or `vdisplay monitors`.
`vdisplay relay list-windows` is deprecated — use `vdisplay windows`.

Filter windows with `--app`, `--class`, `--pid`, `--min-width`, `--min-height`. Use `--all` (default) to include internal/helper windows; `--apps-only` excludes them.

## Desktop workflows (GNOME, multi-monitor)

Examples for a typical Linux desktop session (`DISPLAY=:0`, Wayland + XWayland). Replace monitor names (`DP-2`, `DP-1`, `HDMI-1`) with output from `vdisplay monitors`.

> **Wayland note:** vdisplay lists **XWayland** windows via `xdotool`. Native Wayland apps (Firefox Wayland, Cursor, GNOME Terminal) may not appear in `vdisplay windows` — use **portal screencast** for screenshots of the full desktop or a chosen monitor.

### JetBrains Toolbox / IDE

Toolbox usually runs on XWayland and shows up in window lists:

```bash
# find Toolbox on the primary monitor
vdisplay windows --app "JetBrains"
vdisplay windows --class jetbrains-toolbox | jq '.windows[] | {window_id, app_label, monitor_name, nl}'

# hide Toolbox off-screen (automation / clean screenshot)
vdisplay relay adopt-window --app "JetBrains"
vdisplay relay list

# move Toolbox to another physical monitor
vdisplay relay adopt-window --app "JetBrains" --target HDMI-1
vdisplay relay release-window --app "JetBrains"

# before/after screenshots (Wayland: agent + screencast first — see below)
vdisplay agent serve                              # terminal 1
vdisplay agent screencast start                   # terminal 2 — pick monitor in portal
vdisplay relay screenshot -o before.png --source DP-2
vdisplay relay adopt-window --app "JetBrains"
vdisplay relay screenshot -o toolbox-hidden.png --source DP-2
vdisplay relay release-window --app "JetBrains"
vdisplay relay screenshot -o after-restore.png --source DP-2
```

IntelliJ / PyCharm (when launched via Toolbox, often XWayland):

```bash
vdisplay windows --app "IntelliJ"
vdisplay windows --class idea | jq '.windows[] | {title, pid, nl}'
vdisplay relay adopt-window --title "vdisplay"   # partial title match
```

### Firefox (XWayland)

If Firefox runs on XWayland (`about:support` → Window Protocol: X11):

```bash
vdisplay windows --app "Firefox"
vdisplay windows --class firefox | jq '.windows[] | {window_id, title, nl}'
vdisplay relay adopt-window --app "Firefox"
vdisplay relay adopt-window --title "Mozilla Firefox"
vdisplay relay release-window --app "Firefox"
```

Native Wayland Firefox is **not** listed by `vdisplay windows` — capture the monitor instead:

```bash
vdisplay agent screencast start
vdisplay screenshot -o firefox-monitor.png --source DP-2
```

### Chromium / Chrome / Edge

Same as Firefox — only visible when running on XWayland:

```bash
vdisplay windows --class google-chrome
vdisplay windows --class chromium
vdisplay windows --app "Chrome" | jq '.windows[].nl'
```

### GNOME Terminal, Cursor, native Wayland apps

These often **do not** appear in `vdisplay windows`. Workflow:

```bash
# 1) see which monitors exist and what XWayland apps are visible
vdisplay all | jq '{monitors: [.monitors[].name], x11_windows: [.windows[].app_label]}'

# 2) screenshot the monitor where the app is shown
vdisplay agent screencast start
vdisplay screenshot -o terminal.png --source DP-1
vdisplay screenshot -o primary.png --source primary
```

### LibreOffice, GIMP, Steam (XWayland)

```bash
vdisplay windows --app "LibreOffice"
vdisplay windows --class soffice
vdisplay windows --app "GIMP"
vdisplay relay adopt-window --class steam
vdisplay relay release-window --class steam
```

### VS Code / Cursor / Slack / Discord (Electron)

Electron apps often expose a partial AT-SPI tree when launched with accessibility enabled. Window listing still depends on XWayland:

```bash
# list visible Electron windows (when on XWayland)
vdisplay windows --app "Cursor"
vdisplay windows --app "code"
vdisplay windows --class slack | jq '.windows[] | {title, app_label, nl}'

# semantic control — AT-SPI backend (GTK_A11Y / Java ATK wrapper may be required for some builds)
export GTK_A11Y=1
vdisplay diagnose control
vdisplay control list --app "Cursor" --backend atspi --max-depth 4
vdisplay control click --app "Cursor" --role button --name "Run" --verify

# if native Wayland build — window not in list; capture monitor instead
vdisplay agent screencast start
vdisplay screenshot -o cursor.png --source DP-2
```

### Thunderbird, Nautilus, GNOME Calculator

```bash
vdisplay windows --app "Thunderbird"
vdisplay windows --class org.gnome.Nautilus
vdisplay windows --class org.gnome.Calculator

# hide file manager off-screen before a clean desktop shot
vdisplay relay adopt-window --class org.gnome.Nautilus
vdisplay relay screenshot -o desk-clean.png --source DP-2
vdisplay relay release-window --class org.gnome.Nautilus
```

### Spotify, OBS, Blender (mixed X11 / Wayland)

```bash
vdisplay windows --app "Spotify"
vdisplay windows --class obs
vdisplay windows --class blender | jq '.windows[] | {title, monitor_name, nl}'

# move Blender to second monitor (XWayland only)
vdisplay relay adopt-window --class blender --target HDMI-1
```

### Messaging & video calls (Telegram, Signal, Zoom, Teams)

Electron and Qt wrappers — window listing depends on XWayland; capture the monitor for native Wayland builds:

```bash
vdisplay windows --app "Telegram"
vdisplay windows --class signal
vdisplay windows --app "zoom"
vdisplay windows --class teams | jq '.windows[] | {title, app_label, nl}'

# hide chat window before desktop recording
vdisplay relay adopt-window --app "Telegram"
vdisplay agent screencast start
vdisplay screenshot -o desk-no-telegram.png --source DP-2
vdisplay relay release-window --app "Telegram"

# Zoom/Teams on Wayland — usually not in window list; full-monitor capture
vdisplay screenshot -o meeting.png --source primary
```

### Developer tools (DBeaver, Android Studio, Docker Desktop, Wireshark)

Java/Electron apps — often XWayland; Java ATK helps AT-SPI:

```bash
vdisplay windows --app "DBeaver"
vdisplay windows --class android-studio
vdisplay windows --class Docker
vdisplay windows --class wireshark | jq '.windows[] | {title, pid, nl}'

# semantic click in DBeaver SQL editor toolbar (AT-SPI)
export GTK_A11Y=1
vdisplay control list --app DBeaver --backend atspi --max-depth 5
vdisplay control click --app DBeaver --role button --name Execute --verify

# hide Android Studio emulator window off-screen during screenshot
vdisplay relay adopt-window --title "Android Emulator"
vdisplay relay screenshot -o ci-screen.png --source DP-2
vdisplay relay release-window --title "Android Emulator"
```

### Media players & creative (VLC, Audacity, Krita, Inkscape)

GTK/Qt apps — GTK tools benefit from `GDK_BACKEND=x11` on Wayland for AT-SPI:

```bash
vdisplay windows --class vlc
vdisplay windows --app "Audacity"
vdisplay windows --class krita
vdisplay windows --class inkscape

# pause VLC via AT-SPI (when tree is exposed)
GDK_BACKEND=x11 GTK_A11Y=1 vdisplay control click --app vlc --role button --name Pause --verify

# move Inkscape to second monitor before export
vdisplay relay adopt-window --class inkscape --target HDMI-1
vdisplay relay release-window --class inkscape
```

### Terminals beyond GNOME Terminal (Tilix, Alacritty, kitty)

Native Wayland terminals rarely appear in `vdisplay windows`. Prefer a **terminal session** or monitor capture:

```bash
# if running on XWayland
vdisplay windows --class Tilix
vdisplay windows --class Alacritty
vdisplay windows --class kitty

# reliable TUI control — open PTY instead of chasing desktop window
dsl2vdisplay -c 'terminal open --session-id dev-1 --command bash'
vdisplay control list --backend terminal --session-id dev-1
vdisplay control click --backend terminal --session-id dev-1 --terminal-line 1 --terminal-col 0

# screenshot where the terminal is visible (Wayland)
vdisplay agent screencast start
vdisplay screenshot -o alacritty.png --source DP-1
```

### System apps (Settings, Software, Files, Calculator, virt-manager)

GNOME shell apps are often native Wayland — use monitor capture or AT-SPI with X11 backend:

```bash
vdisplay windows --class gnome-control-center    # Settings (XWayland only)
vdisplay windows --class org.gnome.Software
vdisplay windows --class virt-manager
vdisplay windows --class VirtualBox

# virt-manager VM window on second monitor
vdisplay relay adopt-window --class virt-manager --target DP-1

# Settings on Wayland — not listed; capture primary
vdisplay screenshot -o settings.png --source primary
```

### Password managers & notes (Bitwarden, Obsidian, 1Password)

Electron apps — partial AT-SPI when accessibility is enabled:

```bash
vdisplay windows --app "Bitwarden"
vdisplay windows --class obsidian
vdisplay windows --app "1Password"

export GTK_A11Y=1
vdisplay control list --app obsidian --backend atspi --format tree
vdisplay control click --app Bitwarden --role button --name Unlock --verify
```

### AT-SPI control on native GTK apps (GNOME Wayland)

On Wayland, force GTK apps onto X11 for a reliable accessibility tree:

```bash
export DISPLAY=:0
export GDK_BACKEND=x11
export GTK_A11Y=1
export NO_AT_BRIDGE=0

# demo fixture used in tests — or any GTK3 app you start the same way
./tests/fixtures/run_gtk_demo.sh &
sleep 2

vdisplay control list --app vdisplay-gtk-demo --backend atspi
vdisplay control click --app vdisplay-gtk-demo --role button --name Increment --verify
vdisplay control set-value --app vdisplay-gtk-demo --role input --index 0 --value hello --verify
```

### Browser sessions (Playwright DOM — any site)

Open a **browser session** first, then control via DOM selectors (not the desktop Firefox window):

```bash
# DSL
dsl2vdisplay -c 'browser open --url https://example.com --session web-1'
dsl2vdisplay -c 'browser open --url https://example.com --session web-ff --vendor firefox --headed'

# CLI equivalent via agent (terminal 1: vdisplay agent serve)
vdisplay agent browser-open --url https://example.com --session-id web-1

# control the open session
vdisplay control list --backend browser --session-id web-1
vdisplay control click --backend browser --session-id web-1 --selector "#submit" --verify
vdisplay diagnose control --selector "#go" --session-id web-1   # routing + profile inference
```

### Terminal sessions (TUI / PTY)

```bash
dsl2vdisplay -c 'terminal open --session-id term-1 --command bash'
vdisplay control list --backend terminal --session-id term-1
vdisplay control click --backend terminal --session-id term-1 --terminal-line 3 --terminal-col 10
```

### Desktop app cheat sheet

| App | `vdisplay windows` | Control backend | Notes |
|-----|-------------------|-----------------|-------|
| JetBrains Toolbox / IDE | `--app JetBrains` | `atspi` (X11) | Often XWayland; Java ATK wrapper helps |
| Firefox (native Wayland) | usually hidden | monitor screenshot | Use screencast + `--source` |
| Firefox (X11) | `--class firefox` | `atspi` or relay | Check `about:support` → Window Protocol |
| Chromium / Chrome | `--class google-chrome` | `atspi` / relay | XWayland only in window list |
| Cursor / VS Code / Slack | `--app Cursor` / `code` | `atspi` (partial) | Wayland build → screenshot fallback |
| GNOME Terminal / Tilix | often hidden | `terminal` session | Open PTY session instead of desktop window |
| Alacritty / kitty | `--class` (X11 only) | `terminal` or screenshot | Native Wayland → monitor capture |
| Telegram / Signal / Discord | `--app` / `--class` | relay / screenshot | Electron; hide with `relay adopt-window` |
| Zoom / Teams / Meet | often hidden | monitor screenshot | Video UI rarely in XWayland list |
| DBeaver / Android Studio | `--app` / `--class` | `atspi` (Java) | `GTK_A11Y=1`; emulator via `--title` |
| Docker Desktop / Wireshark | `--class Docker` | relay / `atspi` | Mixed Electron/Qt |
| VLC / Audacity / Krita | `--class vlc` etc. | `atspi` (GTK/Qt) | `GDK_BACKEND=x11` on Wayland host |
| GIMP / Blender / OBS | `--class` / `--app` | `atspi` / relay | Move with `--target HDMI-1` |
| Thunderbird / Evolution | `--app Thunderbird` | `atspi` / relay | Mail clients on XWayland |
| Nautilus / Calculator | `--class org.gnome.*` | relay / `atspi` | Hide file manager before clean shot |
| virt-manager / VirtualBox | `--class virt-manager` | relay | VM window on second monitor |
| Bitwarden / Obsidian | `--app` | `atspi` (partial) | Electron; unlock via `--role button` |
| GTK demo / native GTK | `--app` / title | `atspi` | `GDK_BACKEND=x11` on Wayland host |
| Web app (any URL) | N/A | `browser` + session | `browser open` then DOM selectors |

### Typical desktop session (discover → hide → capture → control)

Copy-paste workflow when several apps are open on a GNOME desktop:

```bash
# 0) broker + portal (once per session)
vdisplay agent serve                              # terminal 1
vdisplay agent screencast start                   # terminal 2

# 1) inventory — what XWayland exposes vs monitors only
vdisplay all | jq '{
  monitors: [.monitors[] | {name, primary, nl}],
  windows: [.windows[] | {app_label, title, monitor_name}]
}'

# 2) hide noisy windows before a clean screenshot
vdisplay relay adopt-window --app "JetBrains"
vdisplay relay adopt-window --class slack
vdisplay relay adopt-window --class org.gnome.Nautilus
vdisplay relay screenshot -o desk-clean.png --source DP-2

# 3) semantic control on an app that exposes AT-SPI (GTK/Java on X11)
export GDK_BACKEND=x11 GTK_A11Y=1
vdisplay control list --app DBeaver --backend atspi --format tree
vdisplay control click --app DBeaver --role button --name Execute --verify

# 4) web workflow — separate from desktop Firefox
dsl2vdisplay -c 'browser open --url https://example.com --session web-1 --headed'
vdisplay control click --backend browser --session-id web-1 --selector "a" --verify

# 5) restore hidden windows
vdisplay relay release-window --app "JetBrains"
vdisplay relay release-window --class slack
vdisplay relay release-window --class org.gnome.Nautilus
```

### Multi-monitor layout (3 outputs)

Inspect geometry, then capture or move windows per output:

```bash
vdisplay monitors | jq '.monitors[] | {name, primary, x, y, width, height, nl}'

# one PNG per connected output (Wayland: screencast + Entire Screen in portal for all)
vdisplay agent screencast start
vdisplay screenshot --all-monitors --out-dir ./captures
ls ./captures/    # DP-2.png, DP-1.png, HDMI-1.png

# single monitor by name
vdisplay screenshot -o dp1.png --source DP-1
vdisplay screenshot -o hdmi.png --source HDMI-1

# mirror primary onto second output (X11 multi-head only; Wayland → use screencast screenshot)
vdisplay mirror screenshot -o mirrored.png --source primary --target DP-1
```

### Isolated virtual desktop (any app, no portal)

Run apps on owned Xvfb `:99` — works headless and in CI:

```bash
sudo apt install xterm   # or firefox, etc.

vdisplay virtual start --width 1920 --height 1080 --display :99
vdisplay virtual launch xterm -hold
vdisplay virtual launch --display :99 xterm -e 'sleep 5; echo hello'

# separate session: launch spins up its own Xvfb, runs command, stops
vdisplay virtual launch xterm -hold
vdisplay virtual screenshot -o vd-xterm.png --display :99

# Python API — long-running agent on :99
python3 - <<'PY'
from vdisplay import VirtualDisplaySession
vd = VirtualDisplaySession.create(width=1280, height=720, display=":99")
vd.start()
vd.launch(["xterm", "-hold"])
vd.save_screenshot("agent-screen.png")
input("Press Enter to stop…")
vd.stop()
PY
```

### Quick agent setup (copy-paste for desktop capture)

```bash
# terminal 1 — leave running
vdisplay agent serve

# terminal 2
vdisplay agent health
vdisplay agent screencast start          # once per session; pick monitor or Entire Screen
vdisplay screenshot -o desk.png --source DP-2
vdisplay relay screenshot -o desk.png --source DP-2
vdisplay mirror screenshot -o desk.png --source primary
vdisplay agent screencast status
vdisplay agent screencast stop
```

If capture is **blank**, the portal shared a different monitor than `--source` — use `vdisplay monitors` + match the name, or restart screencast and choose **Entire Screen**.

### Example output

Monitor:

```json
{
  "name": "DP-2",
  "primary": true,
  "width": 4320,
  "height": 7680,
  "rotation": "left",
  "rotation_degrees": 90,
  "monitor_id": 0,
  "nl": "Primary monitor DP-2 (4320×7680, rotated left (90°)). Visible apps: Toolbox."
}
```

Window:

```json
{
  "window_id": "8388615",
  "app_label": "Toolbox",
  "monitor_id": 0,
  "monitor_name": "DP-2",
  "monitor_index": 0,
  "pid": 32977,
  "nl": "Toolbox application window (880×1326 at (4470,298)) on monitor DP-2, process jetbrains-toolb, class jetbrains-toolbox."
}
```

## Screenshots

### Quick reference

| Goal | Command |
|------|---------|
| One monitor (host) | `vdisplay screenshot -o screen.png --source DP-2` |
| All monitors | `vdisplay screenshot --all-monitors --out-dir ./captures` |
| Virtual Xvfb (no portal) | `vdisplay virtual screenshot -o screen.png --display :99` |
| Mirror + capture | `vdisplay mirror screenshot -o mirror.png --source primary --target DP-1` |
| Describe saved PNG (`img2nl`) | `vdisplay screenshot -o screen.png` → JSON field `nl` |
| Loop every 1s (after screencast) | see [Continuous capture](#continuous-capture-wayland) below |

With **`VDISPLAY_AGENT_URL`** set, screenshots route through the broker (same providers, one runtime).

```bash
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765

# single monitor — use output name from: vdisplay monitors
vdisplay screenshot -o screen.png --monitor 1
vdisplay screenshot -o screen.png --source DP-2

# all connected monitors (DP-2.png, DP-1.png, HDMI-1.png, …)
vdisplay screenshot --all-monitors --out-dir ./captures

# isolated Xvfb session (works without portal / on headless CI)
vdisplay virtual screenshot -o screen.png --display :99
vdisplay screenshot -o vd.png --mode virtual --vd-display :99

# mirror: prefer mirror screenshot on Wayland (uses ScreenCast when active)
vdisplay mirror screenshot -o mirror.png --source primary --target DP-1
```

### Wayland host (GNOME) — start here

On **GNOME Wayland**, `DISPLAY=:0` is XWayland — direct X11/DRM capture often yields **black PNGs** (especially with NVIDIA).

**One-time setup** — needs **two terminals** (broker must stay running):

```bash
# terminal 1 — leave this running (blocks the shell)
vdisplay agent serve
# same as: vdisplay-agent serve
```

```bash
# terminal 2 — all capture commands go here
vdisplay agent health                    # should return {"status":"ok",...}
vdisplay agent screencast start          # pick monitor in portal dialog (once)
# GNOME: Settings → Privacy → Screen Recording → allow vdisplay-agent / terminal
```

> **Common mistake:** running `vdisplay agent screencast start` before `vdisplay agent serve`
> gives `Set VDISPLAY_AGENT_URL … or start: vdisplay-agent serve`. Start the broker first.

`VDISPLAY_AGENT_URL` is optional when the broker listens on `127.0.0.1:8765` (auto-detect).
Explicit export still works: `export VDISPLAY_AGENT_URL=http://127.0.0.1:8765`

**Then capture as often as you like** (no new prompts):

```bash
vdisplay screenshot -o host.png --source DP-2
vdisplay screenshot -o host.png --source primary
vdisplay mirror screenshot -o mirror.png --source primary --target DP-1
vdisplay relay screenshot -o relay.png --source DP-2
```

REST equivalent:

```bash
curl -X POST http://127.0.0.1:8765/session/screencast/start \
  -H 'content-type: application/json' -d '{"interactive": true}'
vdisplay screenshot -o host.png --source DP-1
```

> **Note:** `vdisplay mirror start -o mirror.png` runs xrandr mirror then tries driver-level capture directly.
> On Wayland use **`vdisplay mirror screenshot`** (or `vdisplay screenshot`) after screencast is active.

### Continuous capture (Wayland)

After `vdisplay agent screencast start`, grab frames every second without portal prompts:

```bash
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765
mkdir -p ./captures
i=0
while true; do
  vdisplay screenshot -o "./captures/frame-$(printf '%04d' "$i").png" --source DP-2
  i=$((i + 1))
  sleep 1
done
```

Python (virtual display — no portal needed):

```python
import time
from pathlib import Path
from vdisplay import VirtualDisplaySession

vd = VirtualDisplaySession.create(display=":99")
vd.start()
try:
    for i in range(60):
        vd.save_screenshot(f"frame-{i:04d}.png")
        time.sleep(1)
finally:
    vd.stop()
```

See also [examples/ci-agent/](examples/ci-agent/) (`VD_FRAMES=60` in Docker).

### Describe a screenshot (`img2nl`)

By default, `vdisplay screenshot` enriches JSON with an **`nl`** field — a natural-language description of the saved PNG (Polish locale by default):

```bash
vdisplay screenshot -o screen.png --source DP-2 | jq '{path, nl, img2nl}'
```

Requires optional package: `pip install img2nl[analyze]`. Disable with `VDISPLAY_IMG2NL=0` or `vdisplay screenshot --no-img2nl`.

To describe an **existing** image file:

```python
from vdisplay.application.services.img2nl_enrich import describe_screenshot_image
print(describe_screenshot_image("screen.png")["text"])
```

## Actions — virtual, mirror, relay

```bash
# virtual display (isolated Xvfb)
vdisplay virtual start --width 1920 --height 1080 --display :99
vdisplay virtual launch xterm -hold          # flags after the command name are OK
vdisplay virtual screenshot -o screen.png --display :99

# mirror — needs two monitors; list names first with: vdisplay monitors
vdisplay mirror start --source primary --target DP-1          # xrandr only (no PNG)
vdisplay mirror screenshot -o mirror.png --source primary --target DP-1   # Wayland: screencast first
VD_SOURCE=DP-2 VD_TARGET=HDMI-1 ./examples/host-mirror/run.sh   # Docker; black PNGs on Wayland

# relay — hide window off-screen, restore later (separate CLI calls OK)
# 1) find a window to move
vdisplay windows --apps-only
vdisplay windows --app "JetBrains" | jq '.windows[] | {window_id, app_label, nl}'

# 2) adopt (move off-screen) — match by app, title, class, pid, or window-id
vdisplay relay adopt-window --app "JetBrains"
vdisplay relay adopt-window --title "Firefox"
vdisplay relay adopt-window --class jetbrains-toolbox --pid 32977
vdisplay relay adopt-window --window-id 8388615

# 3) move to another monitor instead of off-screen
vdisplay relay adopt-window --app "Firefox" --target HDMI-1

# 4) list adopted windows (persisted in ~/.cache/vdisplay/)
vdisplay relay list
vdisplay relay list | jq '.adopted[] | {window_id, app_label, title, nl}'

# 5) restore — separate CLI call works (geometry saved on adopt)
vdisplay relay release-window --app "JetBrains"
vdisplay relay release-window --window-id 8388615

# 6) screenshot while relaying (Wayland: agent serve + screencast first)
# terminal 1: vdisplay agent serve
vdisplay agent screencast start
vdisplay relay screenshot -o before.png --source DP-2
vdisplay relay adopt-window --app "JetBrains"
vdisplay relay screenshot -o after-hide.png --source DP-2
vdisplay relay release-window --app "JetBrains"
vdisplay relay screenshot -o after-restore.png --source DP-2

# 7) full host demo (before / after adopt / after release PNGs)
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765
vdisplay agent screencast start
WINDOW_APP=JetBrains ./examples/host-relay/run-host.sh
# output: examples/host-relay/output/before_automation.png, after_adopt.png, after_release.png
```

**Relay tips**

- Adopted window positions **persist** across CLI calls (`~/.cache/vdisplay/__vdisplay_stash__-*.json`).
- Relay moves windows within the **same X11 session** — it does not move apps into Xvfb `:99`.
- On **Wayland**, window move uses XWayland; screenshots need **agent + screencast** (Docker `./run.sh` often yields black PNGs — use `./run-host.sh` instead).
- Filter windows first: `vdisplay windows --app "Toolbox"` → use `--app`, `--title`, or `--window-id` from JSON.

## Control plane (Semantic UI control)

The control plane provides a unified API for accessibility-first, semantic control of native applications, web browsers, and terminals, using computer vision (`img2nl` / screenshot diffs) to verify action execution.

Supported backends:
- `atspi` — Linux Desktop AT-SPI2 accessibility tree
- `browser` — Playwright headless/headed browser DOM
- `terminal` — Terminal session emulation and cursor positioning
- `x11` — `xdotool` keyboard/mouse pointer injection fallback

### CLI Examples

```bash
# 1) Diagnose control backends and readiness
vdisplay diagnose control

# 2) List all interactive controls (buttons, inputs, links)
vdisplay control list --backend auto

# 3) Filter controls by application (e.g. Firefox) in tree format
vdisplay control list --app firefox --backend auto --format tree

# 4) Click a button by name and verify state change semantically
vdisplay control click --role button --name "Save" --verify

# 5) Type text into an input field and verify value change
vdisplay control set-value --role input --name "Username" --value "admin" --verify

# 6) Click a control using CSS selectors in browser environment
vdisplay control click --backend browser --selector "#submit-btn" --session-id "web-1" --verify

# 7) Open browser session then route automatically (DSL)
dsl2vdisplay -c 'browser open --url https://example.com --session web-1 --vendor firefox'
dsl2vdisplay -c 'control click --backend browser --session-id web-1 --selector "#go" --verify'

# 8) Desktop GTK app on Wayland host (X11 backend for AT-SPI)
GDK_BACKEND=x11 GTK_A11Y=1 vdisplay control list --app vdisplay-gtk-demo --backend atspi

# 9) Java IDE toolbar (DBeaver / IntelliJ on XWayland)
vdisplay control click --app DBeaver --role button --name "Execute SQL" --verify

# 10) Electron app — unlock dialog (Bitwarden / Obsidian)
vdisplay control click --app Bitwarden --role button --name Unlock --verify

# 11) xdotool fallback when AT-SPI tree is empty
vdisplay control click --backend x11 --app firefox --role button --name Reload
```

### Python API Example

```python
from vdisplay.application.services import control as control_svc

# List controls from the active accessibility tree
controls = control_svc.controls_list(display=":0", app="Firefox", format="flat")
for c in controls:
    print(f"[{c['role']}] {c['name']} (at {c['x']},{c['y']})")

# Programmatically click a button and verify success
result = control_svc.control_click(
    display=":0",
    role="button",
    name="Log In",
    verify=True,
    screenshot_verify=True
)
print("Action OK:", result["ok"])
```

## Natural language and DSL

```bash
# NL → DSL → JSON (built into vdisplay)
vdisplay nlp "list monitors on display zero"
vdisplay nlp "show application windows" --dsl-only

# DSL bus (same semantics as CLI query verbs)
dsl2vdisplay -c 'MONITORS DISPLAY :0'
dsl2vdisplay -c 'WINDOWS DISPLAY :0 APPS_ONLY'
dsl2vdisplay -c 'ALL DISPLAY :0'

# NL package (thin wrapper; pip install -e packages/nlp2vdisplay)
nlp2vdisplay to-dsl "list monitors on display zero"
nlp2vdisplay apply "show application windows on display zero"
# shorthand — bare prompt defaults to apply:
nlp2vdisplay "show application windows on display zero"
```

Legacy DSL verbs still work: `OUTPUTS` maps to `MONITORS`, `WINDOWS` unchanged.

## vdisplay-agent broker

Install **once** on the host. CLI, DSL, REST, and MCP set `VDISPLAY_AGENT_URL` and route capture, sessions, and discovery through the broker — no per-app portal prompts or direct DRM access from adapters.

```bash
pip install -e "packages/vdisplay-agent[serve]"
pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay

# terminal 1 — broker (localhost)
vdisplay-agent serve
# or: vdisplay agent serve

# terminal 2 — clients
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765
vdisplay agent health
vdisplay monitors                    # 3 monitors in ~50 ms via agent /outputs
dsl2vdisplay -c 'MONITORS'
rest2vdisplay serve --port 8216 --agent-url $VDISPLAY_AGENT_URL
mcp2vdisplay serve                   # tool: vdisplay_agent_status
```

| Env var | Role |
|---------|------|
| `VDISPLAY_AGENT_URL` | Client → broker URL |
| `VDISPLAY_AGENT_TOKEN` | Optional Bearer auth |
| `VDISPLAY_CAPTURE_ALLOW_PORTAL` | Opt-in portal screenshot (agent only) |

**Wayland host capture:** start ScreenCast once in the agent (`POST /session/screencast/start`), then use normal screenshot commands. See [docs/agent-broker.md](docs/agent-broker.md) and [examples/agent-broker](examples/agent-broker/).

Virtual display screenshots work without portal. DRM/fbdev capture requires `video` group on some setups.

Full reference: [docs/agent-broker.md](docs/agent-broker.md) · [packages/vdisplay-agent](packages/vdisplay-agent/)

## Control layer equivalents

| Intent | CLI | DSL |
|--------|-----|-----|
| Full state | `vdisplay all` | `ALL DISPLAY :0` |
| Monitors | `vdisplay monitors` | `MONITORS DISPLAY :0` |
| Windows | `vdisplay windows --apps-only` | `WINDOWS DISPLAY :0` |
| Adopt window | `vdisplay relay adopt-window --app X` | `ADOPT APP X` |
| Screenshot | `vdisplay screenshot -o out.png` | `SCREENSHOT OUT out.png DISPLAY :99` |
| Validate tools | `vdisplay diagnose` | `VALIDATE DISPLAY :0` |

## Modes

| Mode | Purpose | Isolation | Screenshot | Window move |
|------|---------|-----------|------------|-------------|
| `virtual` | Private Xvfb session for agents | Yes | Yes | No (use `launch()`) |
| `mirror` | Duplicate existing display output | No | Yes | N/A |
| `relay` | Move window within same X11 session | Partial | Yes (`relay screenshot`) | Yes |
| `screencast` | Portal ScreenCast in agent (Wayland) | No | Yes (after consent) | N/A |

## Requirements (Linux v0.1)

| Component | Used by |
|-----------|---------|
| `Xvfb`, `xwd`, `scrot` | virtual / X11 capture |
| `xrandr` | mirror mode |
| `xdotool` | relay + input |
| `python3-dbus`, `python3-gi` | portal ScreenCast (Wayland host) |
| `ffmpeg` (PipeWire) or GStreamer `pipewiresrc` | ScreenCast frame grab |
| `Pillow` (optional) | faster PNG encoding |

```bash
sudo apt install xvfb x11-apps x11-utils xdotool scrot x11-xserver-utils
sudo apt install python3-dbus python3-gi   # Wayland ScreenCast in agent
pip install "vdisplay[pillow]"
```

Full setup: [docs/installation.md](docs/installation.md)

## Python API

### Sessions (backends)

```python
from vdisplay import VirtualDisplaySession, MirrorSession, WindowRelaySession
from vdisplay.discovery import list_monitors, list_windows

# Inspect monitors and windows with nl descriptions
for monitor in list_monitors():
    print(monitor["nl"])

for window in list_windows(apps_only=True):
    print(window["nl"])

# Virtual isolated display
vd = VirtualDisplaySession.create(width=1920, height=1080)
vd.start()
vd.launch(["xterm"])
vd.save_screenshot("screen.png")
vd.stop()

# Mirror existing desktop (same session, no isolation)
# On GNOME Wayland: start agent screencast before save_screenshot, or use capture_host_to_file()
m = MirrorSession.create(source="primary", target="DP-1")
m.start()
m.save_screenshot("mirror.png")   # needs ScreenCast on Wayland
m.stop()

# Relay window off-screen and restore (persists across CLI calls)
r = WindowRelaySession.create()
r.start()
r.adopt_window(match_app="JetBrains")
r.release_window(match_app="JetBrains")
r.stop()
```

### Application layer (shared by CLI / DSL / REST / MCP / agent)

```python
from vdisplay.application.commands import CommandRequest
from vdisplay.application.executor import execute
from vdisplay.application.services import discovery, capture, session, info

# Single execution entry (routes to agent when VDISPLAY_AGENT_URL is set)
result = execute(CommandRequest.from_dsl({"verb": "MONITORS"}, line="MONITORS"))
print(result.data)

# Direct service use-cases (always in-process)
monitors = discovery.list_monitors(display=":0")
meta = capture.capture_screenshot(output="screen.png", monitor=1)
session.virtual_start(width=1280, height=720, display=":99")
caps = info.platform_info()
```

Via broker SDK:

```python
from vdisplay.client import AgentClient

client = AgentClient("http://127.0.0.1:8765")
client.outputs()
client.start_screencast(interactive=True)
```

### Window heuristics (testable submodules)

```python
from vdisplay.windows import list_windows_enriched, find_windows, pick_best_window
from vdisplay.windows.rank import dedupe_app_windows
from vdisplay.windows.filter import is_internal_window
```

## Project layout

```
src/vdisplay/
├── application/
│   ├── commands.py         # CommandRequest / CommandResult / CommandVerb
│   ├── executor.py         # single entry: execute() → local or agent
│   ├── handlers/           # local + agent command handlers
│   └── services/           # discovery, capture, session, info
├── commands/               # CLI registry (set_defaults per subcommand)
├── windows/                # scan → normalize → filter → rank → query
├── capture/
│   ├── providers/          # drm, fbdev, mss, x11, portal (opt-in)
│   └── portal_screencast.py  # persistent ScreenCast (Wayland)
├── backends/               # virtual, mirror, relay
├── client.py               # AgentClient SDK
└── cli.py                  # thin entry: register_all + args.func(args)

packages/
├── dsl2vdisplay/           # grammar + CQRS bus → executor
├── vdisplay-agent/         # localhost broker (privileged runtime)
├── rest2vdisplay/          # HTTP → DSL
├── mcp2vdisplay/           # MCP tools
└── nlp2vdisplay/           # NL → DSL
```

## Control layer (DSL / MCP / REST / NL)

Programmatic interfaces on top of the same application services. All query results include `nl` on monitors and windows.

| Package | Role |
|---------|------|
| [dsl2vdisplay](packages/dsl2vdisplay/) | Grammar + CQRS bus (`MONITORS`, `WINDOWS`, `ADOPT`, …) |
| [nlp2vdisplay](packages/nlp2vdisplay/) | Natural language → DSL |
| [uri2vdisplay](packages/uri2vdisplay/) | `vdisplay://cmd/...` → DSL |
| [cli2vdisplay](packages/cli2vdisplay/) | REPL over DSL |
| [mcp2vdisplay](packages/mcp2vdisplay/) | MCP server tools |
| [rest2vdisplay](packages/rest2vdisplay/) | REST API on port 8216 |
| [vdisplay-agent](packages/vdisplay-agent/) | Local broker — sessions, capture, relay |

```bash
vdisplay-agent serve --port 8765
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765

pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay
rest2vdisplay serve --port 8216 --agent-url $VDISPLAY_AGENT_URL
mcp2vdisplay serve
curl -s http://127.0.0.1:8216/health | jq .
curl -s -X POST http://127.0.0.1:8216/v1/dsl -H 'content-type: text/plain' -d 'MONITORS' | jq .
```

Full reference: [packages/README.md](packages/README.md)

## Limitations

- Existing windows on `DISPLAY=:0` **cannot** move into Xvfb `:99` — different X servers.
- Use `VirtualDisplaySession.launch()` for apps on the virtual display.
- Use `WindowRelaySession` to hide/show windows on the current session.
- `mirror` controls the same desktop through a duplicated output, not an isolated copy.
- `nl` on monitors lists apps whose window center falls on that output geometry.
- On **GNOME Wayland**, Docker X11 forwarding often produces black screenshots — use [vdisplay-agent](docs/agent-broker.md) on the host instead.
- Windows/macOS backends are planned; Linux/X11 + Wayland (via agent) supported in v0.1.

Troubleshooting: [docs/troubleshooting.md](docs/troubleshooting.md)

## Development

```bash
pip install -e ".[pillow,dev]"
pip install -e "packages/vdisplay-agent[serve]"
pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay
pytest tests/ -q
./examples/agent-broker/run.sh
./examples/run_all_examples.sh   # where host X11 is available
```

Architecture: [docs/architecture.md](docs/architecture.md)

## License

Licensed under Apache-2.0.
