Metadata-Version: 2.4
Name: uisurf-agent
Version: 0.1.0
Summary: A universal agent for reasoning over browser, desktop, and mobile UIs and performing actions on them.
Project-URL: Homepage, https://github.com/haruiz/uisurf-agent
Project-URL: Repository, https://github.com/haruiz/uisurf-agent
Project-URL: Issues, https://github.com/haruiz/uisurf-agent/issues
Author-email: haruiz <henryruiz22@gmail.com>
Requires-Python: >=3.11
Requires-Dist: a2a-sdk[http-server]>=0.3.25
Requires-Dist: adbutils[all]>=2.12.0
Requires-Dist: google-cloud-aiplatform>=1.141.0
Requires-Dist: mss>=10.1.0
Requires-Dist: pillow>=12.1.1
Requires-Dist: playwright>=1.58.0
Requires-Dist: pyautogui>=0.9.54
Requires-Dist: pydantic>=2.12.5
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: rich>=14.3.3
Requires-Dist: termcolor>=3.1.0
Requires-Dist: typer>=0.16.0
Requires-Dist: uvicorn>=0.41.0
Description-Content-Type: text/markdown

# uisurf-agent

`uisurf-agent` is a Python package for running UI automation agents against:

- a web browser via Playwright
- the local desktop via desktop control utilities
- an Android device or emulator via ADB and `adbutils`

The project includes:

- a package-level Typer CLI at [src/uisurf_agent/cli.py](src/uisurf_agent/cli.py)
- A2A server entrypoints for browser, desktop, and mobile agents
- interactive local execution for manual testing

## Requirements

- Python 3.11+
- `uv`
- Node.js only if you also use the bundled `a2a-inspector`
- Playwright browser binaries installed for browser automation
- a desktop environment if you run the desktop agent
- Android platform tools plus an authorized ADB device or emulator if you run the mobile agent

## Install

Create the environment and install dependencies:

```bash
uv sync
```

Install Playwright browsers if needed:

```bash
uv run playwright install
```

## Environment

The package reads configuration from environment variables and `.env` files via `python-dotenv`.

Common variables:

- `UISURF_PROVIDER`: registered provider name. Default is `gemini`.
- `UISURF_MODEL_ID` / `MODEL_ID`: provider-specific model identifier. Gemini defaults to `gemini-3-flash-preview`.
- `UISURF_PROVIDER_BASE_URL` / `UISURF_MODEL_BASE_URL` / `MODEL_BASE_URL`: optional provider-specific model base URL.
- `AGENT_HOST`: default bind host for A2A servers
- `BROWSER_AGENT_PORT`: default browser A2A port, default `8001`
- `DESKTOP_AGENT_PORT`: default desktop A2A port, default `8002`
- `MOBILE_AGENT_PORT`: default mobile A2A port, default `8003`
- `BROWSER_AGENT_PUBLIC_URL`: public URL advertised by the browser A2A server
- `DESKTOP_AGENT_PUBLIC_URL`: public URL advertised by the desktop A2A server
- `MOBILE_AGENT_PUBLIC_URL`: public URL advertised by the mobile A2A server
- `BROWSER_FAST_MODE`: speeds up browser settling by waiting less aggressively
- `AUTO_MODE`: automatically approve safety-gated A2A actions by default
- `BROWSER_AGENT_AUTO_MODE`: browser A2A override for automatic safety approval
- `DESKTOP_AGENT_AUTO_MODE`: desktop A2A override for automatic safety approval
- `MOBILE_AGENT_AUTO_MODE`: mobile A2A override for automatic safety approval
- `INCLUDE_THOUGHTS`: global default for model thought streaming when supported
- `BROWSER_INCLUDE_THOUGHTS`: browser-only override for thought streaming
- `DESKTOP_INCLUDE_THOUGHTS`: desktop-only override for thought streaming
- `MOBILE_INCLUDE_THOUGHTS`: mobile-only override for thought streaming
- `BROWSER_AGENT_MAX_STEPS`: browser A2A step limit, default `40`
- `DESKTOP_AGENT_MAX_STEPS`: desktop A2A step limit, default `40`
- `MOBILE_AGENT_MAX_STEPS`: mobile A2A step limit, default `40`
- `DESKTOP_OBSERVATION_DELAY_MS`: delay before each desktop screenshot capture
- `MOBILE_OBSERVATION_DELAY_MS`: delay before each mobile screenshot capture
- `MOBILE_DEVICE_SERIAL`: Android device serial used by mobile A2A when `--mobile-serial` is omitted
- `MAX_OBSERVATION_IMAGES`: number of screenshot observations that keep image payloads in history
- `OBSERVATION_SCALE`: default screenshot scale, from `0 < scale <= 1`
- `BROWSER_OBSERVATION_SCALE`: browser-only screenshot scale override
- `DESKTOP_OBSERVATION_SCALE`: desktop-only screenshot scale override
- `MOBILE_OBSERVATION_SCALE`: mobile-only screenshot scale override

Screenshot scaling only changes the image sent to the model. Action coordinates
still map to the full browser viewport, desktop resolution, or Android device
resolution.

When using the built-in `gemini` provider, you will also need the credentials
required by the Google client.

## CLI

The CLI is implemented with [Typer](https://github.com/fastapi/typer) and is exposed through both:

```bash
uv run uisurf-agent --help
```

Current top-level command:

```bash
uv run uisurf_agent run --help
```

Convenience `make` targets are also available for local A2A server runs:

```bash
make run-browser
make run-desktop
make run-mobile
make run-dev
make run-all
```

These targets start long-running local servers instead of one-off tasks. By default they bind to:

```bash
browser: http://127.0.0.1:8001/
desktop: http://127.0.0.1:8002/
mobile: http://127.0.0.1:8003/
```

You can override the bind host and ports from the shell:

```bash
make run-browser HOST=127.0.0.1 BROWSER_PORT=8001
make run-desktop HOST=127.0.0.1 DESKTOP_PORT=8002
make run-mobile HOST=127.0.0.1 MOBILE_PORT=8003 MOBILE_SERIAL=5C060DLCR002MM MOBILE_OBSERVATION_DELAY_MS=1500
make run-dev HOST=127.0.0.1 BROWSER_PORT=8001 DESKTOP_PORT=8002 MOBILE_PORT=8003
make run-all HOST=127.0.0.1 BROWSER_PORT=8001 DESKTOP_PORT=8002 MOBILE_PORT=8003
```

If one of those ports is already in use, the `make` target will exit early with a
clear message. Rerun with free ports if needed.

If you want the previous one-off local task mode, use the interactive targets:

```bash
make run-browser-interactive TASK="Open example.com and summarize the page"
make run-desktop-interactive TASK="Open Terminal and run pwd"
```

### Run the browser agent interactively

```bash
uv run uisurf_agent run browser_agent \
  --task "Open example.com and summarize the page" \
  --fast-mode \
  --no-include-thoughts \
  --max-observation-images 2 \
  --observation-scale 0.75 \
  --max-steps 20
```

Run headless:

```bash
uv run uisurf_agent run browser_agent \
  --headless \
  --task "Go to Hacker News and summarize the top 5 stories"
```

Select a registered provider:

```bash
uv run uisurf_agent run browser_agent \
  --provider gemini \
  --model-id gemini-3-flash-preview \
  --task "Open example.com and summarize the page"
```

### Extending providers

Provider implementations and custom model registration are documented in
[mkdocs/models/external-models.md](mkdocs/models/external-models.md).

### Extending browser, desktop, and mobile tools

Browser, desktop, and mobile custom tools use the same registry pattern. A
declaration function describes the tool schema sent to the model. An optional
handler executes the tool when the selected agent receives that function call:

```python
from typing import Any

from uisurf_agent import BrowserAgent, register_browser_tool


async def save_page_title(agent: BrowserAgent, args: dict[str, Any]) -> None:
    title = await agent._browser_controller.page.title()
    print(f"{args['label']}: {title}")


@register_browser_tool(handler=save_page_title)
def save_page_title(label: str) -> dict[str, str]:
    """Save the current page title with a label."""
    return {"label": label}
```

Use `register_desktop_tool(...)` for desktop-only tools,
`register_mobile_tool(...)` for Android-only tools, or
`register_tool("browser" | "desktop" | "mobile", ...)` when the environment is
selected dynamically. If a registered tool has no handler, the agent falls back
to a controller method with the same name. Installed packages can expose tool
setup through the `uisurf_agent.tools` entry-point group. Entry points may
expose a registration function, a `ToolRegistration`, or a declaration callable
named like `browser.my_tool`, `desktop.my_tool`, or `mobile.my_tool`.

### Run the desktop agent interactively

```bash
uv run uisurf_agent run desktop_agent \
  --task "Open Terminal and run pwd" \
  --desktop-observation-delay-ms 750 \
  --no-include-thoughts \
  --max-observation-images 2 \
  --observation-scale 0.75 \
  --max-steps 10
```

Automatically approve safety-gated actions:

```bash
uv run uisurf_agent run desktop_agent \
  --task "Open a text editor and type Hello World" \
  --auto-mode
```

### Run the mobile agent interactively

First verify that ADB can see your Android device or emulator:

```bash
adb devices -l
```

Then run `MobileAgent`:

```bash
uv run uisurf_agent run mobile_agent \
  --task "Open Android Settings and tell me what is visible" \
  --mobile-serial 5C060DLCR002MM \
  --auto-mode \
  --max-steps 10
```

If `--mobile-serial` is omitted, the controller uses the first authorized ADB
device.

For Python API usage, see
[mkdocs/agents/mobile.md](mkdocs/agents/mobile.md).

## A2A Server Mode

Browser, desktop, and mobile agents can be exposed as A2A servers through the CLI.

### Browser A2A server

```bash
uv run uisurf_agent run browser_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8080
```

If `--port` is omitted, the browser agent defaults to `8001`.

### Desktop A2A server

```bash
uv run uisurf_agent run desktop_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8081
```

If `--port` is omitted, the desktop agent defaults to `8002`.

### Mobile A2A server

```bash
uv run uisurf_agent run mobile_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8003 \
  --mobile-serial 5C060DLCR002MM
```

If `--port` is omitted, the mobile agent defaults to `8003`. If
`--mobile-serial` is omitted, the server uses `MOBILE_DEVICE_SERIAL` or the first
authorized ADB device.

## MCP Mode

The CLI accepts `--mode mcp`, but MCP server mode is not implemented yet. The command currently exits with a clear error instead of starting a server.

## Docker

The repository includes a containerized runtime that starts:

- noVNC on port `6080`
- a Chromium instance inside the container with remote debugging on port `9222`
- the browser agent A2A server on port `8001`
- the desktop agent A2A server on port `8002`

### Provide environment variables

The recommended approach is to place secrets and runtime settings in a local `.env` file in the repository root. The wrapper script [run.sh](run.sh) will automatically pass that file to Docker with `--env-file` if it exists.

Example `.env`:

```dotenv
GEMINI_API_KEY=your_key_here
UISURF_PROVIDER=gemini
UISURF_MODEL_ID=gemini-3-flash-preview
AGENT_HOST=0.0.0.0
BROWSER_AGENT_PORT=8001
DESKTOP_AGENT_PORT=8002
BROWSER_FAST_MODE=true
AUTO_MODE=false
INCLUDE_THOUGHTS=false
DESKTOP_OBSERVATION_DELAY_MS=750
MAX_OBSERVATION_IMAGES=2
OBSERVATION_SCALE=0.75
PASSWORD_REQUIRED=false
```

You can also point the wrapper at a different file:

```bash
ENV_FILE=.env.local sh ./run.sh
```

### Build and run the container

From the repository root:

```bash
sh ./run.sh
```

The script builds the image from [docker/Dockerfile](docker/Dockerfile), starts the container, and publishes the default ports to the host.

Default host endpoints:

- noVNC: `http://localhost:6080`
- browser A2A server: `http://localhost:6080/browser/`
- desktop A2A server: `http://localhost:6080/desktop/`

### Override published ports

The wrapper script supports environment variable overrides:

```bash
BROWSER_AGENT_PUBLIC_URL=http://localhost:6081/browser/ \
DESKTOP_AGENT_PUBLIC_URL=http://localhost:6081/desktop/ \
PORT=6081 \
sh ./run.sh
```

### View logs

```bash
docker logs -f uisurf-agent-test
```

### Stop the container

```bash
docker rm -f uisurf-agent-test
```

### Notes

- The container startup will fail early if neither `GEMINI_API_KEY` nor `GOOGLE_API_KEY` is provided.
- VNC password auth is disabled by default. Set `PASSWORD_REQUIRED=true` if you want the frontend to require a VNC password again.
- Inside the container, Chromium is started separately and the browser controller connects to it over CDP at `http://127.0.0.1:9222`.
- The desktop and browser agents are both started through the package CLI from [src/uisurf_agent/cli.py](src/uisurf_agent/cli.py).
- `MobileAgent` is available from the package CLI/Python API and A2A mode, but the default Docker runtime does not start a mobile A2A server or provision an Android emulator/device.

## Logging

The package configures a Rich-backed library logger in [src/uisurf_agent/__init__.py](src/uisurf_agent/__init__.py). The logger name is `uisurf_agent`.

## Package Layout

- [src/uisurf_agent/cli.py](src/uisurf_agent/cli.py): Typer CLI
- [src/uisurf_agent/a2a/browser_a2a.py](src/uisurf_agent/a2a/browser_a2a.py): browser A2A server
- [src/uisurf_agent/a2a/desktop_a2a.py](src/uisurf_agent/a2a/desktop_a2a.py): desktop A2A server
- [src/uisurf_agent/a2a/mobile_a2a.py](src/uisurf_agent/a2a/mobile_a2a.py): mobile A2A server
- [src/uisurf_agent/agents/browser_agent.py](src/uisurf_agent/agents/browser_agent.py): browser automation agent
- [src/uisurf_agent/agents/desktop_agent.py](src/uisurf_agent/agents/desktop_agent.py): desktop automation agent
- [src/uisurf_agent/agents/mobile_agent.py](src/uisurf_agent/agents/mobile_agent.py): Android automation agent
- [src/uisurf_agent/utils/mobile_controller.py](src/uisurf_agent/utils/mobile_controller.py): ADB-backed mobile controller

## Development

Validate the package modules compile:

```bash
uv run python -m compileall src/uisurf_agent
```

Show CLI help locally through the package module:

```bash
uv run python -m uisurf_agent --help
```
