Desktop Agent

DesktopAgent drives a local desktop through desktop control utilities.

Use it for tasks that require operating-system UI control, local applications, terminal commands, file dialogs, or workflows that cross browser and native app boundaries.

Requirements

• A graphical desktop session available to the process
• Desktop automation permissions granted by the host operating system when required
• Provider credentials configured for the selected model provider

What It Owns

• desktop setup and cleanup
• screenshots of the current screen
• actions such as opening apps, clicking, typing, scrolling, moving the cursor, running terminal commands, and window management
• conversion from normalized 0-1000 coordinates to screen pixels

Reasoning Loop

DesktopAgent follows the shared observe/reason/act loop:

1. Observe: capture a screenshot of the active desktop display and track the last high-level desktop target.
2. Encode: send the task, screenshot, and desktop state metadata through the configured model adapter.
3. Reason: ask the model for either a final answer or desktop tool calls.
4. Act: execute desktop actions through DesktopController, such as opening apps, clicking, typing, scrolling, running terminal commands, or managing windows.
5. Report: capture a fresh screenshot and desktop state after each action and return them to the model as tool results.
6. Repeat: continue until the model returns a final answer or the step limit is reached.

The desktop loop is best when the task depends on native applications, operating system windows, terminal workflows, or UI that is not available through a web page.

Run With Python

import asyncio

from uisurf_agent import DesktopAgent


async def main() -> None:
    async with DesktopAgent(observation_delay_ms=750) as agent:
        async for event in agent.run(
            "Open Terminal and run pwd",
            max_steps=10,
        ):
            print(event.eventType, event.payload)


asyncio.run(main())

Run With The CLI

uisurf_agent run desktop_agent \
  --task "Open Terminal and run pwd" \
  --desktop-observation-delay-ms 750 \
  --max-steps 10

Run As An A2A Server

uisurf_agent run desktop_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8002

If --port is omitted, the desktop A2A server defaults to 8002.

Desktop A2A mode also reads DESKTOP_AGENT_MAX_STEPS, DESKTOP_AGENT_AUTO_MODE, DESKTOP_INCLUDE_THOUGHTS, DESKTOP_OBSERVATION_DELAY_MS, DESKTOP_OBSERVATION_SCALE, and DESKTOP_AGENT_PUBLIC_URL from the environment.

Common Runtime Options

Option	Description
--desktop-observation-delay-ms	Delay before each desktop screenshot capture.
--auto-mode	Automatically approve safety-gated actions.
--max-steps	Maximum observe/reason/act iterations.
--max-observation-images	Number of recent image-bearing items that keep image payloads.
--observation-scale	Scale screenshots before sending them to the model.

Model and provider configuration is documented in the Models section.

Notes

Desktop behavior depends on the host operating system and available permissions. On macOS, for example, the terminal or Python process may need Accessibility and screen-recording permissions before desktop screenshots and input actions work.