Metadata-Version: 2.4
Name: easy-uiauto
Version: 0.5.0
Summary: UI automation toolkit based on pyautogui and uiautomation
Author-email: Poggi-Tang <2322806855@qq.com>
Project-URL: Homepage, https://github.com/Poggi-Tang/easyautomation
Project-URL: Repository, https://github.com/Poggi-Tang/easyautomation.git
Project-URL: Issues, https://github.com/Poggi-Tang/easyautomation/issues
Project-URL: Changelog, https://github.com/Poggi-Tang/easyautomation/blob/master/CHANGELOG.md
Project-URL: Documentation, https://github.com/Poggi-Tang/easyautomation/blob/master/README.md
Keywords: auto,ui,uiauto,automation,gui-testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyautogui>=0.9.54
Requires-Dist: pynput>=1.8.1
Requires-Dist: uiautomation>=2.0.29
Requires-Dist: pyperclip>=1.8.0
Requires-Dist: build>=1.4.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: mcp
Requires-Dist: mcp[cli]<2.0.0,>=1.0.0; extra == "mcp"
Provides-Extra: vision
Requires-Dist: opencv-python>=4.8; extra == "vision"
Requires-Dist: pytesseract>=0.3.10; extra == "vision"
Dynamic: license-file

# easy-uiauto

English | [简体中文](https://github.com/Poggi-Tang/easyautomation/blob/main/README.zh-CN.md)

[![PyPI](https://img.shields.io/pypi/v/easy_uiauto?cacheSeconds=300)](https://pypi.org/project/easy-uiauto/)
[![Python](https://img.shields.io/pypi/pyversions/easy-uiauto?cacheSeconds=300)](https://pypi.org/project/easy-uiauto/)
[![License](https://img.shields.io/github/license/Poggi-Tang/easyautomation)](https://github.com/Poggi-Tang/easyautomation/blob/main/LICENSE)
[![CI](https://github.com/Poggi-Tang/easyautomation/actions/workflows/ci.yml/badge.svg)](https://github.com/Poggi-Tang/easyautomation/actions/workflows/ci.yml)
[![Publish](https://github.com/Poggi-Tang/easyautomation/actions/workflows/publish.yml/badge.svg)](https://github.com/Poggi-Tang/easyautomation/actions/workflows/publish.yml)

`easy-uiauto` is a UI automation toolkit based on pyautogui and uiautomation.

It provides a comprehensive set of APIs for GUI automation, including mouse control, keyboard input, 
window management, and control location. It is suitable for automated testing, RPA (Robotic Process 
Automation), and other desktop automation scenarios.

![logo](https://github.com/Poggi-Tang/easyautomation/blob/main/src/image/easy-uiauto.png)

## Features

- Mouse control: click, double-click, right-click, drag and drop
- Keyboard input: text input, key press/release, combination keys
- Window management: activate, maximize, switch windows
- Control location: XPath-based positioning, image recognition
- Visual feedback: real-time control highlighting during recording
- Action recording: record user interactions and generate scripts
- Rich text field support: clipboard-based text input
- Cross-framework support: Win32, Qt, and other UI frameworks

## Installation

Install from PyPI:

```bash
pip install easy-uiauto
```

Or install from source:

```bash
git clone https://github.com/Poggi-Tang/easyautomation.git
cd easyautomation
pip install -e .
```

### MCP Server

Install the optional MCP dependencies when using easy-uiauto from an MCP client:

```bash
pip install "easy-uiauto[mcp]"
```

Install local OCR and image-template fallback support when needed:

```bash
pip install "easy-uiauto[mcp,vision]"
```

The `vision` extra provides OpenCV template matching and the Python Tesseract
adapter. OCR also requires the system Tesseract executable and relevant language
data (for example `eng` or `chi_sim`). The MCP tools are
`find_control_by_image`, `click_by_image`, `find_text_on_screen`, and
`click_text_on_screen`.

Remote multimodal location does not run a local model or require an AI SDK. Set
an OpenAI-compatible vision endpoint and credentials, then use
`find_control_by_vision` or `click_by_vision`:

```bash
EASY_UIAUTO_VISION_API_URL=https://your-api.example/v1/chat/completions
EASY_UIAUTO_VISION_API_KEY=your-api-key
EASY_UIAUTO_VISION_MODEL=your-vision-model
```

Those tools upload the current screenshot to the configured endpoint only for
that request. Use them as a final fallback after UIA, OCR, or image matching.

The MCP server is part of the library and reuses the same automation APIs:

```bash
easy_uiauto --help
easy_uiauto --version
easy_uiauto
```

The standard commands below register, inspect, or remove the global MCP
configuration through the client's own CLI. The standard install command does
not overwrite an existing entry with the same name.

For a complete Codex deployment, install the package and run full setup:

```bash
pip install --upgrade "easy-uiauto[mcp,vision]"
easy_uiauto --full-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model
```

Full setup securely prompts for a missing API key, installs missing Python
vision dependencies, installs Tesseract through winget when necessary, replaces
only the `easy_uiauto` Codex MCP entry, and validates UIA, local OCR, and remote
AI vision. The OCR and AI checks use generated test images; they do not upload
the current desktop. Each result includes its elapsed time, and any failed check
makes the command fail.

For a minimal Codex deployment with remote AI vision, use the quick setup
command. When the API key is not already present in the Windows user
environment, it prompts once through hidden terminal input or a password dialog
for non-interactive agents. It persists the three vision variables, replaces
only the `easy_uiauto` Codex MCP entry, and skips OCR installation and UI tests:

```bash
easy_uiauto --quick-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model
```

One-sentence instruction for a Codex agent:

> Install the latest `easy-uiauto[mcp,vision]` from PyPI, then run
> `easy_uiauto --full-setup-codex --vision-url URL --vision-model MODEL`; do not
> inspect unrelated projects or search the web, report every validation result
> and elapsed time, then ask me to restart Codex.

```bash
easy_uiauto --install-codex
easy_uiauto --show-codex-config
easy_uiauto --uninstall-codex

easy_uiauto --install-claude-code
easy_uiauto --show-claude-code-config
easy_uiauto --uninstall-claude-code
```

Codex registration uses its global `config.toml`. Claude Code registration uses
the `user` scope, so it is available to every local project. Restart the client
after installing or removing the server.

The long-running TCP service is also available:

```bash
easy_uiauto_service --help
python -m easy_uiauto.mcp.service --port 9876
```

### Application UI Knowledge and CLI

Scan any visible Windows application into an Obsidian-compatible knowledge vault:

```bash
easy_uiauto_ui scan "Window title"
easy_uiauto_ui apps
easy_uiauto_ui commands <app-id>
easy_uiauto_ui search <app-id> "search terms"
easy_uiauto_ui run <app-id> <page.region.control.action> --text "optional text"
easy_uiauto_ui batch <app-id> '["main.keypad.6.click", "main.keypad.plus.click"]'
easy_uiauto_ui learn-effect <app-id> <command> --recover
easy_uiauto_ui explore <app-id> --policy safe --max-actions 10 --max-depth 3
easy_uiauto_ui interactions <app-id>
easy_uiauto_ui teach <app-id> <control-id> "Meaning" intent "Description"
```

The default vault is `~/easy_uiauto_vault`. Set `EASY_UIAUTO_KNOWLEDGE_DIR`
to use another Obsidian vault. Each application stores Markdown/YAML records,
page screenshots, per-control visual-state PNGs, interaction before/after images,
a quarantine directory, and a generated `operations/UI-CLI.md` catalog.
`.easy_uiauto/index.json` is only a disposable search cache and can be rebuilt with
`easy_uiauto_ui reindex`.

The default `visual-first` scan makes one multimodal request to identify the current
page, functional regions, and only task-relevant controls. Pixel targets are mapped
back to local UIA controls and scored for stable names, automation IDs, supported
actions, and tight bounds. This avoids a complete UIA-tree walk and a second remote
semantic pass. Use `--strategy full-uia` only for diagnostic coverage when visual
targeting is unsuitable. Both strategies store user-facing meaning, stable intent,
description, aliases, evidence, risk, actions, ambiguity, and confidence. Commands are
published only after high-confidence semantics and LOCATION/image validation. The visual
request uses SSE streaming when supported and window capture uses virtual-desktop
coordinates, including secondary monitors. If vision returns no controls, visual-first
reports an empty scan instead of silently invoking a second remote request.

`learn-effect` captures the target window and local full-desktop state before and after
one verified command. It waits for delayed changes to stabilize, reduces pixel changes
to local regions, checks only UIA controls in those regions, inventories new or transient
top-level windows, captures action-control properties, and asks vision to generalize a
success condition. Full desktop images stay local; only target-window images and crops
of newly opened related windows are sent to the configured endpoint. `explore` repeats
this for known reversible commands, deduplicates page/command states, presses Escape to
recover, and stops after interference or failed recovery. `safe` permits only safe
commands; `supervised` also permits reversible state-changing commands. External,
destructive, or confirmation-required commands are never explored automatically.
Scrolling and dragging are intentionally not part of exploration.

At runtime, resolution follows `LOCATION`, stored multi-state image templates, local
OCR, and finally opt-in remote vision. Use `--allow-vision-fallback` or
`allow_vision_fallback=true` for the last step. Ambiguous, stale, or missing controls
are quarantined and excluded from execution until a successful rescan repairs them.

Locator/image verification, semantic verification, and functional execution are tracked
separately. Scanning does not click every control: doing so could send, publish, purchase,
delete, or otherwise change external state. AI meanings are marked as inferred until
directly taught or observed, and execution history never masquerades as proof of the
resulting application behavior. Use `easy_uiauto_ui teach` or the `teach_ui_control` MCP
tool to correct uncertain meanings; manual semantics survive rescans but cannot bypass a
failed locator. External or destructive commands require `--confirm` or `confirm=true`.

For consecutive operations on one stable page, use `easy_uiauto_ui batch` or the
`run_ui_commands` MCP tool. A batch loads the command catalog once, finds and captures
the window once, verifies every unique control before performing any action, executes
the ordered sequence, and rebuilds the knowledge index once. Steps may be command
strings or objects such as `{"command":"main.form.message.set-text","text":"hello"}`.
Split a workflow after any command that navigates to another page.

Knowledge created before contextual control semantics were introduced remains readable,
but its old commands are intentionally not executable. Rescan each page once to add the
required semantic evidence, confidence, risk, and verification fields.

The same workflow is available through MCP tools:

- `scan_window_knowledge`
- `list_ui_knowledge_apps`
- `search_ui_knowledge`
- `list_ui_commands`
- `run_ui_command`
- `run_ui_commands`
- `learn_ui_command_effect`
- `explore_ui_workflows`
- `list_ui_interactions`
- `teach_ui_control`
- `rebuild_ui_knowledge_index`

`--full-setup-codex` also installs or updates the bundled
`easy-uiauto-learning` and `easy-uiauto-operate` Codex skills. Use
`--install-codex-skills` to update only those skills.

For MCP client configuration, start the server with `python -m easy_uiauto.mcp.server`.
Control-vector persistence is optional. To enable it, set
`EASY_UIAUTO_CONTROL_VECTOR_DB_DIR` to a directory containing
`control_vector_store.py`; otherwise capture tools still return records but do not persist them.

Control lookup uses the library's canonical `LOCATION` object rather than a
flat selector. Obtain it from a recorded action or from
`get_control_at_position`, then pass the returned `LOCATION` object directly to
`find_control`:

```json
{
  "WindowName": "My Application",
  "Name": "Save",
  "ClassName": "ButtonClass",
  "ControlType": "ButtonControl",
  "foundIndex": 1,
  "AutomationId": "saveButton",
  "Xpath": [
    {"ControlType": "WindowControl", "Name": "My Application", "searchDepth": 1},
    {"ControlType": "ButtonControl", "Name": "Save", "foundIndex": 1, "searchDepth": 2}
  ],
  "Img": "",
  "PARAMETERS": {}
}
```

`find_control` also accepts a complete recorded action containing `LOCATION`
and the complete result from `get_control_at_position`. Legacy flat arguments
remain supported for compatibility, but full XPath data is more reliable for
duplicate or deeply nested controls.

## Quick Start

### Basic Control Operations

```python
from easy_uiauto.ctrl import Controller

# Left click on a control
Controller.left_click(
    ActionTitle="Click OK Button",
    WindowName="My Application",
    Name="OK",
    ClassName=None,
    ControlType="ButtonControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={}
)

# Input text into a field
Controller.input_text(
    ActionTitle="Enter Username",
    WindowName="Login Dialog",
    Name="Username",
    ClassName=None,
    ControlType="EditControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"输入文本": "test_user"}
)

# Keyboard shortcut
Controller.key_group(
    ActionTitle="Save File",
    WindowName="Notepad",
    Name="",
    ClassName=None,
    ControlType="",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"组合键": "ctrl+s"}
)
```

### Recording User Actions

```python
from easy_uiauto.record import run_record

# Start recording user actions
run_record(write_file=True)
# Press ESC to stop recording
# Generated script will be saved to Record{timestamp}.py
```

## Project Structure

```text
easyautomation
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── publish.yml
│       └── release.yml
├── src/
│   └── easy-uiauto/
│       ├── __init__.py
│       ├── ctrl.py          # Core controller (mouse/keyboard actions)
│       ├── draw.py          # Visual feedback (control highlighting)
│       ├── record.py        # Action recording
│       └── utils.py         # Utility functions (control location, caching)
├── tests/
├── CHANGELOG.md
├── LICENSE
├── README.md
├── README.zh-CN.md
└── pyproject.toml
```

## Release Automation

This repository is prepared for a professional Python package workflow:

- **CI** runs lint and tests on push and pull request.
- **Semantic Release** updates the version, changelog, tag, and GitHub Release.
- **Trusted Publishing** publishes to PyPI from GitHub Actions without a PyPI API token.
- **Build artifacts** include both source distribution and wheel.

## Development

```bash
pip install -e .[dev]
pytest
ruff check .
```

## Usage Examples

For more examples, please refer to the test files in the `demo/` directory or check the docstrings in the source code.

## License

MIT License. See [LICENSE](https://github.com/Poggi-Tang/easyautomation/blob/main/LICENSE).

## Contact

Scan the QR code to add me on WeChat:

![WeChat QR Code](https://github.com/Poggi-Tang/easyautomation/blob/main/src/image/or_code.bmp)
