Skip to content

uisurf-agent

uisurf-agent is a Python library for running UI automation agents against:

  • a web browser through Playwright
  • a local desktop through desktop control utilities
  • an Android device or emulator through ADB

The package provides:

  • BrowserAgent for browser automation
  • DesktopAgent for desktop automation
  • MobileAgent for Android automation
  • a Typer CLI for interactive and server-based runs
  • A2A server entrypoints for browser, desktop, and mobile agents
  • a model extension system for Gemini, OpenAI-compatible servers, local models, and custom model integrations
  • a tool extension system for adding browser, desktop, and mobile actions

Core Idea

The agents own UI observation and action execution. Models own provider-specific message formats, model calls, tool-call parsing, tool-result formatting, and context compaction. Tool declarations are shared through a registry so custom actions can be exposed to models and executed by browser, desktop, or mobile agents.

BrowserAgent / DesktopAgent / MobileAgent
  -> capture UI observation
  -> encode observation as a provider message
  -> ask configured Model to generate a ModelTurn
  -> add the provider response to history
  -> execute ToolCalls
  -> encode ToolResults as provider messages

The built-in model is gemini, but external providers can be added with the Model protocol and @register_provider(...). Browser, desktop, and mobile actions can be added with register_browser_tool(...), register_desktop_tool(...), register_mobile_tool(...), or the uisurf_agent.tools entry-point group.