Metadata-Version: 2.4
Name: tapyr-cli
Version: 0.1.0
Summary: Playwright for Android, built for AI agents. CLI + MCP server for LLM-driven phone control.
Author-email: Rafie Amandio <88525718+RafieAmandio@users.noreply.github.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/RafieAmandio/tapyr
Project-URL: Documentation, https://github.com/RafieAmandio/tapyr
Project-URL: Repository, https://github.com/RafieAmandio/tapyr
Project-URL: Bug Tracker, https://github.com/RafieAmandio/tapyr/issues
Keywords: android,adb,automation,mcp,ai-agents,playwright
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Pillow>=10.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

![Tests](https://github.com/RafieAmandio/tapyr/actions/workflows/test.yml/badge.svg)

# Tapyr

**Playwright for Android, built for AI agents.**

A CLI and stdio MCP server that gives an LLM (Claude, Cursor, any MCP-capable
agent) structured perception and action primitives for a connected Android
device, plus a file-based knowledge layer so learned workflows and app notes
persist across sessions.

Tapyr is **not** an agent. It is the hands, eyes, and memory that an agent
uses. Bring your own brain.

## Why

LLMs can already read text and reason about intent. They cannot, by default,
open your Gojek app, tap "GoFood", type "sate", and hand off to checkout.
Tapyr closes that gap with a small, opinionated set of primitives designed
for tool-use from the ground up:

- **JSON-first output.** Every command returns `{"ok": bool, ...}` — no
  log-scraping.
- **Semantic targeting.** `tap --text "Order"` or `tap --id 7` (from the
  last perception), not `tap 540 1820`.
- **Set-of-Mark perception.** Screenshot + UI dump + an annotated PNG with
  numbered boxes so vision models can pick an element by number.
- **Cross-device coordinates.** `tap_xy` and `swipe` accept percent strings
  (`"50%"`) that resolve against the live `wm size` at replay time. Workflows
  written on 720x1604 replay on 1080x2400 unchanged.
- **Workflow executor.** YAML workflows with template variables, weighted
  `choose_one` branches, `repeat`, `sequence`, `expect_screen`, `verify`,
  `if` / `skip_if` / `run_if` conditional gates, per-step `retries` with exponential
  backoff, and a debug mode that snapshots fingerprints and annotated PNGs
  per step. `kb run tiktok_engage --arg count=10` — done.
- **Safety gates.** `checkpoint: name, requires_confirmation: true` halts a
  replay cleanly at payment / destructive steps; resume past the gate with
  `tapyr kb run NAME --resume-from checkpoint_name` without re-executing the
  expensive setup.
- **Observability.** `tapyr kb stats` surfaces per-workflow success rate,
  halted runs, and drift-marker counts from the JSONL audit log — flaky
  workflows bubble to the top. Soft `expect_screen` mismatches emit a
  structured `needs_llm_fallback` marker so an orchestrating agent can
  machine-detect drift without parsing prose warnings.
- **Calibration for canvas apps.** Apps that render via custom canvas
  (TikTok, Instagram Reels, YouTube Shorts) are invisible to uiautomator.
  `calibrate-rail` scans for a vertical icon rail (right-side action column),
  `calibrate-nav` scans for a horizontal one (bottom nav), and the `tap_rail`
  workflow action consumes saved calibrations by semantic name (`icon:
  comment`). See below.
- **Lock screen + device status.** `tapyr unlock [--pin XXXX]` dismisses the
  keyguard idempotently (no-ops if already unlocked). `tapyr status` returns
  a one-shot snapshot — locked? foreground? screen size? wakefulness?
  battery? — in a single call instead of five sequential adb round trips.
- **File-backed memory.** Workflows are YAML. App notes are Markdown. The
  agent can read, write, and share them.
- **One binary, many front doors.** The same core runs as a CLI, an MCP
  stdio server, or a self-installing plugin for Claude Desktop / Claude
  Code / Cursor.

## Architecture

```
┌─────────────────────────────────────────────┐
│  executor     replay workflows, verify      │
├─────────────────────────────────────────────┤
│  knowledge    workflows + app notes + FTS   │
├─────────────────────────────────────────────┤
│  calibration  whiteness scan for canvas UIs │
├─────────────────────────────────────────────┤
│  actions      semantic tap, swipe, type     │
├─────────────────────────────────────────────┤
│  perception   screenshot + ui_dump + SoM    │
├─────────────────────────────────────────────┤
│  driver       adb subprocess wrapper        │
└─────────────────────────────────────────────┘
```

Each layer is usable independently. Agents operate at whatever level the
task needs: replay a saved workflow (fast path), drop to semantic taps
(normal path), coordinate-based taps on a calibrated rail (canvas-app
escape hatch), or raw pixels (last resort).

## Install

```bash
pipx install tapyr-cli        # or: pip install tapyr-cli
tapyr doctor                  # tells you exactly what's missing
tapyr doctor --fix            # auto-install adb if missing, re-run checks
tapyr install-adb             # one-shot Playwright-style fetch of platform-tools
tapyr demo --dry-run          # preview the 7-step scripted walkthrough
tapyr demo                    # with a phone plugged in: the hero shot
```

`tapyr install-adb` fetches the official Android platform-tools bundle from
`dl.google.com` into `~/.tapyr/bin/`, falling back to a `find_adb()` chain
extension so PATH-installed adb still wins if present. Cached after first
run; `--force` re-downloads. Optional `TAPYR_ADB_SHA256` env var for
enterprise users who need a pinned checksum.

`tapyr demo` runs a zero-side-effect, 7-step walkthrough (doctor → wake →
perceive → HOME → deep-link Settings → BACK → save a marker workflow) that
gives the first-run experience a single command. No typing into fields, no
taps inside third-party apps, no installs. `--dry-run` works without a
phone for curious users and CI.

You will also need Android `platform-tools` on your `PATH` (for `adb`) and a
phone with USB debugging enabled. `doctor` walks you through both.

The `TAPYR_HOME` environment variable overrides the default `~/.tapyr`
state directory. Useful for multi-device test rigs or CI.

### Wire it into your agent

```bash
tapyr install claude-desktop   # writes ~/Library/.../claude_desktop_config.json
tapyr install claude-code      # writes ~/.claude/mcp.json
tapyr install cursor           # writes ~/.cursor/mcp.json
tapyr install print            # just prints the JSON snippet
```

Then restart the client. The agent now has a `tapyr` tool group with:

- **Intent resolver:** `do` (natural-language → action, no API knowledge needed)
- **Perception:** `perceive`, `screenshot`
- **Actions:** `tap`, `swipe`, `type_text`, `key`, `scroll_until`, `wait_for`,
  `unlock`, `exec_steps`
- **Device:** `device_status`, `list_devices`
- **Apps:** `open_app`, `deep_link`, `foreground`, `list_packages`,
  `read_notifications`
- **Calibration:** `calibrate_rail`, `calibrate_nav`, `calibrations_list`,
  `calibration_show`, `calibration_delete`
- **Credential vault:** `fill_credential`, `vault_set`, `vault_list`,
  `vault_delete` (macOS / Linux / Windows; no `vault_get` by design)
- **Safety policy:** `safety_list`, `safety_set_rule`, `safety_delete_rule`
- **Knowledge base:** `kb_search_workflows`, `kb_suggest_workflows`,
  `kb_read_workflow`, `kb_run_workflow`, `kb_save_workflow`,
  `kb_delete_workflow`, `kb_rename_workflow`, `kb_clone_workflow`,
  `kb_update_workflow_metadata`, `kb_lint_workflow`, `kb_diff_workflows`,
  `kb_find_references`, `kb_check_calibration`, `kb_record_workflow`,
  `kb_add_tags`, `kb_remove_tags`, `kb_read_history`, `kb_stats`,
  `kb_read_app_note`, `kb_append_app_note`, `kb_list_app_notes`,
  `kb_list_workflows`, `kb_export_bundle`, `kb_import_bundle`,
  `kb_install_bundle`
- **Environment:** `doctor`

**56 MCP tools** across 11 families — the full surface an agent needs to
perceive, act, learn, and manage workflows on any connected Android device.

## Usage (CLI)

```bash
# Sanity check
tapyr doctor --pretty
tapyr devices
tapyr status --pretty            # one-shot: locked, foreground, battery, ...

# See the screen the way an agent sees it
tapyr perceive --pretty
#   -> ~/.tapyr/cache/last_screenshot.png
#   -> ~/.tapyr/cache/last_annotated.png  (numbered overlays)
#   -> JSON element list with ids
tapyr perceive --clickable-only    # trim layout noise, keep only tap targets
tapyr perceive --compact           # minimal payload: id + text + clickable only
tapyr watch --interval 2 --max-polls 10  # poll for screen changes, JSON Lines output

# AI CLI mode — just say what you want
tapyr do "like this post"                  # resolves intent → action automatically
tapyr do "go home"                         # builtin: HOME key (no perceive needed)
tapyr do "type hello world"                # detects type intent
tapyr do "open com.whatsapp"               # opens app by package
tapyr do "scroll down"                     # swipe gesture
tapyr do "like this" --dry-run             # preview the resolved action without executing
tapyr perceive --since-last        # delta mode: only what changed since the last call
tapyr --timeout 60 perceive        # global adb-shell timeout override (slow devices)

# Drive the phone
tapyr unlock --pin 1234          # dismisses keyguard, idempotent
tapyr tap --text "GoFood"
tapyr type "sate"
tapyr key enter
tapyr tap --id 3
tapyr tap --xy 50% 90%           # percent coords work everywhere
tapyr swipe 50% 75% 50% 25%
tapyr scroll-until --text "Add to cart"
tapyr wait-for --text "Order confirmed" --timeout 15

# Skip the UI entirely when you know the intent
tapyr deep-link "gojek://gofood/home"
tapyr open com.whatsapp

# Read push notifications / SMS OTPs (needs --noredact on device)
tapyr read-notifications --package com.android.messaging --pretty

# Knowledge base
tapyr kb list
tapyr kb list --query "food order"
tapyr kb search "checkout" --tag food                  # FTS + tag filter
tapyr kb read gojek_order_food
tapyr kb run tiktok_natural_engagement --arg count=10
tapyr kb run unlock_flow gofood_order verify_order    # pipeline: run 3 in sequence
tapyr kb run my_workflow --debug                      # per-step PNGs
tapyr kb run my_workflow --dry-run                    # preview only
tapyr kb run pay_flow --confirm-checkpoints           # walk past gates
tapyr kb run pay_flow --confirm-safety                # walk past require_confirmation safety rules
tapyr kb run pay_flow --resume-from confirm_payment   # pick up from halt
tapyr kb run my_workflow --dry-run --dry-run-state '{"screen_texts":["OK"]}' # evaluate if/skip_if gates against a fixture
tapyr kb lint my_workflow                             # static validation
tapyr kb stats --days 7                               # flaky workflows
tapyr kb history --limit 20 --action run_workflow
tapyr kb delete old_workflow
tapyr kb rename old_name new_name                      # atomic rename + FTS reindex
tapyr kb tag add gojek_order food delivery             # add tags to a saved workflow
tapyr kb tag remove gojek_order wip                    # remove tags
tapyr kb clone gojek_order --as gojek_order_v2         # duplicate for safe editing
tapyr kb diff workflow_v1 workflow_v2 --pretty         # colored unified-diff
tapyr kb history --export csv > history.csv             # spreadsheet-friendly audit export
tapyr kb find-refs --calibration tiktok_rail           # who uses this?
tapyr kb check-calibration tiktok_rail                 # did re-cal break anything?
tapyr kb app com.gojek.app
tapyr kb app com.gojek.app --append "Payment confirm button is bottom-right"
tapyr kb apps                                          # list all documented packages
tapyr kb export ~/gojek-pack.tapyr.zip --all            # pack workflows+cals+notes
tapyr kb import ~/gojek-pack.tapyr.zip --merge rename   # skip/overwrite/rename
tapyr kb install https://example.com/gojek-pack.tapyr.zip --sha256 ...  # fetch + verify + import

# Credential vault (macOS / Linux / Windows)
tapyr vault set com.whatsapp password --stdin           # reads from stdin, never argv
tapyr vault list                                         # {package, field} only — no secrets
tapyr fill-credential com.whatsapp password              # paste into focused field

# Action safety policy
tapyr safety list                                        # show current deny/require_confirmation/allow rules
tapyr safety set deny key --equals POWER                 # never POWER key on this device
tapyr safety set deny shell --contains "pm uninstall"    # forbid uninstalls via shell action
tapyr safety set require_confirmation deep_link --package com.android.vending
tapyr safety delete deny key --equals POWER              # exact-match delete

# Parallel test fleets
TAPYR_MULTI_DEVICE=1 tapyr --serial emulator-5554 perceive  # scoped cache + history
```

## Calibration for canvas-rendered apps

Some apps render their entire UI through a custom canvas engine — TikTok,
Instagram Reels, YouTube Shorts. Their windows have focus but `shown=false`
in `dumpsys`, so `uiautomator dump` falls through to whatever's drawn
underneath (usually the launcher). `perceive` returns zero meaningful
elements. Semantic `tap --text` and `tap --content_desc` do not work.

Tapyr's workaround: scan the screen for white icon clusters along a vertical
rail and use the cluster centers as blind-tap targets.

```bash
# 1. One-time setup per app+device: scan the rail, label the clusters,
#    save the result. The x-column auto-detects in the right 15% of the
#    screen; pass --x-col to override.
tapyr calibrate-rail \
  --y-min 500 --y-max 1400 \
  --min-cluster-pixels 30 \
  --labels heart,comment,bookmark,share \
  --save tiktok_rail.json

# 2. In a workflow yaml, reference the saved calibration by name:
#
#    - action: tap_rail
#      calibration: tiktok_rail
#      icon: comment                       # v1: bare label
#      icon: right_rail.comment            # v2: namespaced (region.label)
#
#    Unified v2 calibrations merge multiple regions (right_rail, bottom_nav)
#    into one file per app. Bare labels auto-resolve when unambiguous;
#    namespaced labels (right_rail.comment) disambiguate across regions.
#    v1 files (single rail) keep working unchanged.

# 3. Discover what's saved
tapyr calibrations --pretty
```

The `examples/tiktok_natural_engagement.yaml` workflow is a fully worked
reference: a weighted `choose_one` loop that randomly likes, favorites,
shares, or comments (with Indonesian gen-Z phrases) 5-45% of the time,
passively scrolls the rest, and posts real comments via TikTok's native
input field.

See `examples/README.md` for four more showcase workflows:
**whatsapp_send_message** (the hello-world tutorial),
**gofood_order** (Indonesian use case with a payment checkpoint),
**spotify_play_playlist** (morning-routine weighted rotation), and
**instagram_reels_engagement** (canvas-app mirror of the TikTok example).
Every file in `examples/` is pinned lint-clean by a regression test.

## Knowledge on disk

```
~/.tapyr/
├── workflows/              # agent-written task recipes (yaml)
├── apps/                   # per-app declarative notes (markdown)
├── calibration/            # rail-icon calibrations for canvas apps (json)
├── history/                # jsonl audit log of every action
├── cache/                  # last screenshot, dump, annotated PNG
├── debug/                  # per-step snapshots from `kb run --debug`
└── index.sqlite            # FTS index — rebuildable from the above
```

Everything is git-friendly. You can version-control your `~/.tapyr/`
directory and share workflows + calibrations across machines or with
teammates.

## Usage (from an agent, via MCP)

Once installed into your client, the agent can call the tools directly.
A typical loop for a well-known app:

1. `perceive` → read the element list + annotated PNG
2. `tap` with a selector that matches an element
3. `perceive` → confirm the new screen
4. `kb_save_workflow` once the task completes, so the next agent is faster

For a canvas-rendered app (TikTok etc.) where `uiautomator dump` returns
nothing: **vision-first fallback (v0)** automatically kicks in — the
screenshot is analyzed via grid-based variance detection to synthesize
clickable regions even without a UI tree. The agent sees numbered boxes
on the annotated PNG and can `tap --id N`. For higher precision on
known apps, calibrate once and replay:

1. `calibrate_app` with `--rail` and `--nav` flags to scan both icon
   regions in one shot and save a unified v2 calibration file
2. `kb_save_workflow` with `tap_rail` or `tap_saved` steps referencing
   namespaced labels (`right_rail.comment`, `bottom_nav.home`)
3. `kb_run_workflow` to replay

## Status

**v0 / alpha.** Validated on a real Oppo 720x1604 device through 40+ loop
iterations of incremental hardening. The TikTok engagement workflow posts
real comments, likes, bookmarks, and shares through the coordinate-based
`tap_rail` path. **977 tests passing**, covering the executor, actions,
calibration scanners, knowledge base, MCP server, CLI dispatch, and the
workflow observability surface.

What's here (as of this README revision): the full primitive set listed
above, 10 MCP tool families, `kb stats` observability over the JSONL audit
log, soft-drift `needs_llm_fallback` markers on `expect_screen` mismatches,
idempotent `unlock` and composed `device_status`, and a cross-device
coordinate system that works on any screen that `wm size` can report.

**Vision-first perception fallback (v0)** shipped — when `uiautomator dump`
returns nothing, perception now synthesizes clickable regions from the
screenshot via grid-based variance analysis so agents can at least `tap --id N`
on numbered areas in the annotated PNG. Still directional (v1): replacing
the heuristic grid with a segmentation model (SAM / YOLO) for element-level
precision, plus OCR for text field extraction. See `BACKLOG.md` for the full
list with priority ordering.

## License

MIT.
