Skip to content

Getting Started

uisurf-agent is the runtime package for UISurf UI automation agents. It provides three environment-specific agents behind the same observe/reason/act interface:

  • BrowserAgent for websites and web applications through Playwright
  • DesktopAgent for local operating-system UI through desktop automation utilities
  • MobileAgent for Android devices or emulators through ADB and adbutils

The package can be used directly from Python, through the Typer CLI, or as an A2A server where supported.

Project Shape

The runtime is organized around a few stable layers:

  • agents capture UI observations, call a model, and dispatch requested actions
  • controllers perform environment-specific side effects such as browser clicks, desktop input, or ADB commands
  • models translate provider-specific APIs into the neutral Observation, ModelTurn, ToolCall, and ToolResult contracts
  • tools declare model-visible actions and optionally provide custom handlers
  • A2A adapters expose agents to external orchestration systems where supported

Architecture At A Glance

flowchart TD User["User task"] --> Entry["Runtime entrypoint<br/>Python API / CLI / A2A"] Entry --> Agent["UI Agent<br/>BrowserAgent / DesktopAgent / MobileAgent"] Agent --> Observe["Observation<br/>screenshot + environment metadata"] Observe --> Model["Model adapter<br/>provider-specific API"] Model --> Decision["ModelTurn<br/>final answer or tool calls"] Decision --> Tools["Tool dispatch<br/>built-in or registered actions"] Tools --> Controller["Environment controller<br/>Browser / Desktop / Mobile controller"] Controller --> Surface["UI surface<br/>web browser / desktop / Android device"] Surface --> Observe

The concrete agent can be BrowserAgent, DesktopAgent, or MobileAgent. The concrete controller follows the same choice: browser, desktop, or mobile. The rest of the loop stays the same across environments.

The same high-level loop applies across environments:

observe UI -> encode observation -> model reasons -> execute tool calls
  -> capture post-action state -> repeat until done

Each agent page explains how that loop maps to its environment.

Install

Create the Python environment and install dependencies:

uv sync

Install Playwright browser binaries when using BrowserAgent:

uv run playwright install

For MobileAgent, install Android platform tools on the host and verify that ADB can see the target device:

adb devices -l

Choose An Agent

Start with the agent page that matches the UI surface you need:

  • Browser Agent: web pages, web apps, search, forms, dashboards
  • Desktop Agent: native apps, terminal workflows, file dialogs, OS windows
  • Mobile Agent: Android apps, mobile settings, push/SMS flows, device-specific UI

Each page shows the supported run modes:

  • Python API
  • interactive CLI
  • A2A server mode when available

For complete copyable snippets, see Code Examples. That page includes Python scripts, CLI commands, a runtime agent selector, and A2A server commands.

Configure Models

Model and provider setup is documented in the Models section:

Common environment variables and CLI flags are listed in Configuration.

Extend Actions

Use Tool Extensions when you need to expose additional browser, desktop, or mobile actions to a model.

Use Human In The Loop when actions need explicit approval before execution.

Server And Container Modes

Browser, desktop, and mobile agents can run as A2A servers. See A2A Servers.

The default Docker runtime starts the browser and desktop A2A servers with noVNC for observation. Mobile A2A requires an ADB-reachable Android device or emulator. See Docker.