Mobile Agent

MobileAgent drives an Android device or emulator through ADB using adbutils.

Use it for tasks that depend on Android-only surfaces such as native apps, SMS, push approvals, app deep links, device settings, or mobile UI testing.

Requirements

• Android platform tools available on the host
• an authorized Android device or emulator visible to ADB
• adbutils[all], installed by the project dependency set
• Provider credentials configured for the selected model provider

Check connected devices:

adb devices -l

For a real phone, enable Developer Options and USB debugging, then accept the authorization prompt on the device.

What It Owns

• ADB device setup and cleanup
• screenshots of the current Android display
• optional UIAutomator XML hierarchy observations
• Android actions such as tapping, typing, swiping, scrolling, Back, Home, and Recents
• app launching by package name
• conversion from normalized 0-1000 coordinates to device pixels

Reasoning Loop

MobileAgent follows the shared observe/reason/act loop:

1. Observe: capture an Android screenshot and optionally dump the UIAutomator XML hierarchy.
2. Encode: send the task, screenshot, device metadata, and compact UI hierarchy context through the configured model adapter.
3. Reason: ask the model for either a final answer or mobile tool calls.
4. Act: execute Android actions through MobileController, such as tapping, typing, swiping, pressing Back/Home, opening Recents, launching packages, or opening deep links.
5. Report: capture a fresh screenshot, device state, and UI hierarchy after each action and return them to the model as tool results.
6. Repeat: continue until the model returns a final answer or the step limit is reached.

The mobile loop is best when the task depends on Android-native apps, device state, mobile-only authentication flows, notifications, or real-device behavior.

Run With Python

import asyncio

from uisurf_agent import MobileAgent


async def main() -> None:
    async with MobileAgent(
        auto_mode=True,
        serial="5C060DLCR002MM",  # or None to use the first authorized device
        observation_delay_ms=500,
    ) as agent:
        async for event in agent.run(
            "Open Android Settings and tell me what is visible.",
            max_steps=10,
        ):
            print(event.eventType, event.payload, "final=", event.isFinal)


asyncio.run(main())

Run With The CLI

uv run uisurf_agent run mobile_agent \
  --task "Open Android Settings and tell me what is visible" \
  --mobile-serial 5C060DLCR002MM \
  --auto-mode \
  --max-steps 10

If --mobile-serial is omitted, the controller selects the first authorized ADB device.

Run As An A2A Server

uv run uisurf_agent run mobile_agent \
  --mode a2a \
  --host 0.0.0.0 \
  --port 8003 \
  --mobile-serial 5C060DLCR002MM

If --port is omitted, the mobile A2A server defaults to 8003. If --mobile-serial is omitted, the server uses MOBILE_DEVICE_SERIAL or the first authorized ADB device.

Mobile A2A mode also reads MOBILE_AGENT_MAX_STEPS, MOBILE_AGENT_AUTO_MODE, MOBILE_INCLUDE_THOUGHTS, MOBILE_OBSERVATION_DELAY_MS, MOBILE_OBSERVATION_SCALE, and MOBILE_AGENT_PUBLIC_URL from the environment.

Common Runtime Options

Option	Description
--mobile-serial	Android device serial. Defaults to the first authorized ADB device.
--mobile-observation-delay-ms	Delay before each mobile screenshot capture.
--auto-mode	Automatically approve safety-gated actions.
--max-steps	Maximum observe/reason/act iterations.
--max-observation-images	Number of recent image-bearing items that keep image payloads.
--observation-scale	Scale screenshots before sending them to the model.

Model and provider configuration is documented in the Models section.

Notes

The mobile controller uses ADB, so the selected device must remain connected and authorized for the duration of the run. Real-device behavior can differ from emulator behavior, especially for permissions, lock screens, push notifications, and OEM-customized settings screens.