Metadata-Version: 2.5
Name: wayland-computer-use
Version: 0.1.0
Summary: Desktop control for AI agents on native Wayland — portal RemoteDesktop input with real Unicode, KWin window focus, screenshots
Project-URL: Homepage, https://github.com/wh0ami3/wayland-computer-use
Project-URL: Issues, https://github.com/wh0ami3/wayland-computer-use/issues
Author-email: jesse <j3ss3302@gmail.com>
License: MIT License
        
        Copyright (c) 2026 jesse
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: accessibility,agent,automation,computer-use,desktop-automation,kde,kwin,libei,linux,plasma,rpa,wayland,xdg-desktop-portal
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: X11 Applications :: KDE
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Desktop Environment
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Provides-Extra: linux
Requires-Dist: dbus-fast>=2.21; extra == 'linux'
Provides-Extra: windows
Requires-Dist: pyautogui>=0.9.54; extra == 'windows'
Description-Content-Type: text/markdown

# wayland-computer-use

Desktop control for AI agents on **native Wayland**, where `xdotool` and
`pyautogui` do not work and cannot be made to work.

One file, no dependencies beyond the portal client. Extracted from a working
Telegram agent that drives a KDE Plasma 6 desktop.

```bash
wayland-computer screenshot
wayland-computer click 640 480
wayland-computer type "привет, мир"      # this is the part that usually fails
wayland-computer key ctrl+alt+t
wayland-computer focus chromium
```

---

## Why this exists

Wayland is deliberately built so that a client cannot read another window's
pixels or deliver synthetic input to it. That is a security feature, and it
means the entire X11 automation stack is dead on a modern session:

- **Ubuntu 26.04 LTS** (23 April 2026) is the first Ubuntu LTS whose default
  desktop ships **without a GNOME-on-Xorg session** to fall back to.
- **RHEL 10** removed the X.Org server, keeping only XWayland — it is listed
  under *Removed features* in the release documentation.
- **GNOME 49** disabled the GNOME-on-X11 session (GDM still launches other
  X11 sessions); the X11 backend itself was removed in **GNOME 50**.

So anything built on `xdotool`, `pyautogui` or XTEST — which is most
computer-use agents, most desktop RPA, and most desktop QA harnesses —
silently stops working. This library takes the paths that still work.

## What is actually solved here

### Unicode input, including Cyrillic

The one that costs everyone else a day. On Wayland the working path is to send
**keysyms**, not scancodes, and there is a universal rule for any codepoint:

```python
def char_keysym(ch: str) -> int:
    cp = ord(ch)
    if 0x20 <= cp <= 0x7E:      # basic Latin maps to itself
        return cp
    return 0x01000000 | cp      # the portals' universal Unicode rule
```

Four lines, and Cyrillic, Greek, CJK and everything else types correctly.

For comparison, as of August 2026: `ydotool type` handles US-ASCII only against
a hardcoded layout (the UTF-8 patch has been sitting unmerged since 7 July);
`kwin-mcp` looks each character up in a table built from four rows of US QWERTY
and, on a miss, **skips the character and reports success**; Handy needs a
separate `kwtype` binary on KDE because `wtype` has no
`zwp_virtual_keyboard_manager_v1` there.

### Input through the portal, with the consent dialog shown once

`xdg-desktop-portal` `RemoteDesktop` for keyboard and pointer, `ScreenCast` for
absolute coordinates. The permission dialog appears on first use; after that a
`restore_token` is persisted and reused. No root, no `uinput` group, no udev
rules, no daemon.

### Window focus that works when nothing else does

The agent opens a browser, starts typing the query — and the focus is still on
Telegram, so the query goes into the chat. On Wayland you cannot fix this with
`wmctrl` or `xdotool`: the compositor does not hand other windows to anyone.

The only door is KWin scripting over D-Bus (`loadScript` / `run` /
`unloadScript`). One catch worth knowing before you try it yourself: `print()`
inside a KWin script goes to **journald**, not to `qdbus` stdout, and there is
no other channel out of the script. This library reads the answer back from the
journal.

### Screenshots that fail loudly instead of hanging

Spectacle with a hard timeout and retries — a hung compositor would otherwise
block the call forever. Screenshots land in a private `0700` directory with
unique names, and the last ~20 are kept.

### Output designed for a model, not a human

Every command prints **one line**. Errors print one line on stderr, never a
traceback. This is not tidiness: a 1400-character Python stack full of absolute
paths costs an agent an entire turn, and it learns nothing from it. Exit codes
are part of the contract — `0` done, `2` bad arguments, `3` not found,
`1` everything else — so a caller can branch without parsing prose.

### Human-paced pointer and typing

The cursor travels along an eased arc with slight jitter; typing has a live
rhythm with longer pauses after spaces and punctuation. Two reasons this is not
decoration: a teleporting cursor is unwatchable in a demo, and some toolkits
only fire hover state on real motion. `WCU_INPUT_STYLE=instant` turns it off.

---

## Install

Not on PyPI yet — install from the repository:

```bash
pip install "wayland-computer-use[linux] @ git+https://github.com/wh0ami3/wayland-computer-use"
```

Requirements on Linux: a Wayland session, `xdg-desktop-portal` with a backend
that implements `RemoteDesktop` and `ScreenCast` (KDE and GNOME both do), and
`spectacle` for screenshots on KDE.

Windows is supported through `pyautogui` (`pip install wayland-computer-use[windows]`).

## Use from Python

```python
import asyncio
from wayland_computer_use import Computer

async def main():
    c = Computer()
    path = await c.screenshot()
    await c.click(640, 480)
    await c.type_text("привет")
    await c.key("ctrl+s")
    ok, detail = await c.focus("chromium")

asyncio.run(main())
```

## Use from the shell

Built to be called by an agent as a tool.

| command | what it does |
|---|---|
| `screenshot [path.png]` | capture the screen, print the path |
| `click X Y [left\|right\|middle\|double]` | click at absolute coordinates |
| `move X Y` | move the cursor |
| `type "text"` | type text, Unicode included |
| `key "ctrl+alt+t"` | press a combination |
| `scroll DY` | scroll, negative scrolls up |
| `focus "substring"` | focus a window; on miss, prints the open windows |
| `find "description"` | locate an element (needs an aimer, see below) |
| `clickon "description"` | locate and click |

## Aimer plugin

`find` and `clickon` need a vision model to turn "the blue Save button" into
coordinates. Rather than bundling one, point `WCU_AIMER` at your own:

```bash
export WCU_AIMER="my_module:aim"     # aim(image_path, target) -> (x, y) | None
```

Anything works — a local Qwen-VL over an Ollama endpoint, a hosted model, or
template matching. Not set, or no hit, and the command exits `3`, so the caller
can fall back to aiming by itself.

## Configuration

| variable | default | meaning |
|---|---|---|
| `WCU_HOME` | `~/.wayland-computer-use` | state: restore token, screenshots, last pointer position |
| `WCU_INPUT_STYLE` | `human` | `instant` disables the glide and typing rhythm |
| `WCU_AIMER` | unset | `module:function` for `find` / `clickon` |

## Status and limits

Beta. Used daily on Arch Linux with KDE Plasma 6 on Wayland; that is the only
configuration tested end to end.

Known limits, stated plainly:

- **Screenshots go through Spectacle**, so on KDE only. `CaptureWindow` on
  `org.kde.KWin.ScreenShot2` returns `NoAuthorized` without a
  `X-KDE-DBUS-Restricted-Interfaces` entry in a `.desktop` file
  ([KDE bug 446628](https://bugs.kde.org/show_bug.cgi?id=446628)), so
  single-window capture is not available — capture full screen and crop by the
  window's `frameGeometry`.
- **Window focus is KWin-specific.** GNOME and wlroots compositors need their
  own backend; the interface is there, the implementations are not.
- **No accessibility tree yet.** AT-SPI2 would let an agent read window state
  as text instead of pixels, which is cheaper in tokens and far more accurate
  than aiming at a screenshot. That is the next thing worth building.
- The portal consent dialog cannot be avoided on first run by design. If you
  need it gone entirely, KWin exposes a private EIS interface that skips it.

### How fast does it type?

A portal-based input path in another project was measured at 3-4 characters per
second and its author called it unusable, so this is worth measuring rather
than claiming. `bench/typing_speed.py` reports the transport ceiling and the
real `type_text` rate, for ASCII and non-ASCII, on your machine:

```bash
python bench/typing_speed.py
```

Two things keep the rate up here: paced typing is a deliberate default that
`WCU_INPUT_STYLE=instant` removes, and any text over 120 characters skips
per-character typing entirely and goes through the clipboard in one paste.

Contributions welcome, especially a GNOME backend and an AT-SPI2 reader.

## License

MIT.
