Provider Model System

The model system decouples UI control from model APIs.

Agents call a provider-neutral Model interface:

class Model(Protocol):
    async def encode_observation(self, observation: Observation) -> Any: ...
    async def generate_response(self, history: list[Any]) -> ModelTurn: ...
    async def add_response_to_history(self, response: ModelTurn, history: list[Any]) -> None: ...
    async def encode_action_results(self, results: list[ToolResult]) -> Any: ...
    def prepare_history(self, history: list[Any], max_observation_images: int) -> list[Any]: ...

Neutral Types

Observation

Current UI state captured by an agent.

Field	Description
task	User instruction, usually present only in the first observation.
screenshot	Raw screenshot bytes.
mime_type	Image MIME type. Defaults to image/png.
metadata	Extra state such as current URL, desktop state, mobile state, screen geometry, or UI hierarchy text.

ToolCall

Provider-neutral request from a model to execute a UI action.

Field	Description
id	Optional provider tool-call id.
name	Canonical action name understood by the agent.
args	JSON-compatible action arguments.

ToolResult

Result of executing a tool call.

Field	Description
id	Optional id copied from the originating tool call.
name	Tool/action name.
result	Status such as success, unknown_function, user_denied, or error: ....
screenshot	Optional post-action screenshot bytes.
metadata	Post-action state such as URL, desktop state, mobile state, or screen geometry.
safety_acknowledged	Whether required human confirmation was granted.

ModelTurn

Normalized model response.

Field	Description
text	Final answer when the model is done.
tool_calls	Ordered tool calls requested by the model.
thoughts	Optional provider-exposed reasoning/debug text.
refused	Whether the provider refused the request.
failure_message	Terminal message for the agent to emit.
raw	Original provider response or candidate.

Built-In Provider

The built-in provider is registered as:

@register_provider("gemini")
class GeminiModel:
    ...

It contains all Gemini-specific message construction, Gemini computer-use config, candidate parsing, function-response formatting, and image compaction.

Tool Extensions

Browser, desktop, and mobile tools are extensible through a provider-neutral tool registry. Tool declarations are exposed to models, and optional handlers are used by agents when the model calls those tools.

See Tool Extensions for register_browser_tool(...), register_desktop_tool(...), register_mobile_tool(...), controller fallback behavior, and package entry points.