PICOTTY — Help & Operator Docs

Overview #

PICOTTY (branded Swarm Control in the UI) is a fleet console for headless machines. Each target has a Raspberry Pi Pico wired to it as a USB HID keyboard and a serial line; the Picos ("nodes") dial home to a hub (a FastAPI service) over TCP, and this dashboard is the hub's browser front-end.

The browser holds no authoritative state. On load it pulls the registry and recent history over REST, then a WebSocket keeps everything live: node up/down, heartbeats, command results, streamed serial output, prompt-state changes, and progress for expect / OTA / runbook jobs.

DEMO mode

If the hub API is unreachable (for example you opened the file directly), the UI drops into DEMO mode with a small placeholder fleet so every screen still renders and every affordance is clickable. Demo carries no real addresses or credentials; actions are simulated locally and nothing is persisted.

Layout of the Nodes view

• Left rail — the searchable, filterable node list. See Nodes.
• Center — the selected node's detail header, the serial console, live job bars, and the input composer.
• Right rail — command history and the live event feed.

The top nav switches between Nodes, Macros, Runbooks, Events, and Settings, and carries a live fleet count, hub uptime, and the WebSocket status pill (green = live, amber = connecting, red = offline).

Nodes & the node list #

Where: Nodes view → left rail.

Each row is one node. The colored dot is online (green) / offline (hollow); a pulsing ring means commands are in flight. The row shows the node id, its label, a prompt-state badge when online, the group tag, IP, and how long ago it was last seen. Click a row to select it and load its console.

• Filter box — type to match on id, label, group, or IP. Purely client-side.
• All / Online / Offline — segmented status filter.

The detail header (top of center pane) shows the resolved facts the node reported in its hello: IP, firmware version, round-trip time, caps (capabilities such as hid, cdc, serial_tx, ota), group, keyboard layout, and — if assigned — the serial bridge endpoint.

A node is powered by its target's USB. When the target is off, the node is off — that is normal, not a fault.

The serial console #

Where: Nodes view → center pane.

The dark pane streams the selected node's serial output live over the WebSocket. Every byte the target emits is stored verbatim on the hub; this pane only affects the display.

Two renderers

If the operator has vendored xterm.js into hub/src/picotty/static/vendor/, the console is a real terminal emulator — ANSI colors, cursor moves and full-screen TUIs (installers, htop, nano) render correctly. Without it, the console falls back to an append-only log: terminal control sequences are stripped and full-screen apps scroll as plain text rather than repainting. Either way the raw bytes are kept, so a recording or a downloaded log is complete.

Console toolbar

The console is a log view for shell interaction, not a curses terminal — unless xterm.js is vendored, full-screen TUIs won't repaint in place.

Input modes: HID vs Serial #

Where: Console composer → the HID / Serial toggle.

The composer has two per-node input modes. They are distinct sessions and line up only at BIOS/GRUB.

How Serial mode behaves in the composer: the input field is a capture surface — keystrokes are intercepted and streamed to the getty; the field itself stays empty (the target's echo is what you see in the console). Fast typing is debounced into a few frames per second; Enter/Backspace/Tab/Esc/Ctrl-C/D/Z flush immediately.

The common surprise: while watching the Serial console, HID keystrokes appear to do nothing — they are landing on tty1, not the serial getty you're viewing. Pick the mode that matches what you're looking at.

Serial mode is offered only for nodes whose firmware advertises the serial_tx capability; on older firmware the toggle is greyed and the composer stays HID-only. A node that reconnects on older firmware while set to Serial falls back to HID automatically.

Control bytes & keys #

Where: Console composer → Keys / Chords (HID) and Serial rows.

HID keys & chords

In HID mode the composer offers single keys (Enter Tab ESC ↑ ↓ DEL F2 F12) and chords. Two toggles tune typing:

• append ⏎ — append a newline after typed text (usually on, to "press enter").
• char delay (ms) — per-character delay; raise it for a target that drops fast keystrokes.

Dangerous chords. CTRL+ALT+DEL and ALT+SysRq+B reset the target machine. When "require confirm for destructive keys" is on (see Settings), these prompt before firing.

Serial control bytes

Because Serial mode writes raw bytes, common control characters map straight onto the send command's raw (hex) field. The composer's Serial row exposes them as buttons:

You can also just press these keys while focused in the Serial field — they are intercepted and sent as the right bytes.

Custom chords #

Where: Console composer → HID Custom row → “＋ Chords”.

Beyond the built-in chords, you can save your own reusable key combinations — a dedicated store separate from macros. Each is a label plus a chord string like CTRL+ALT+F2; the string is split on + and upper-cased into a keys list.

• Manage: the “＋ Chords” button opens a small manager to add, edit and delete chords.
• Use: every saved chord shows up as a quick button on the composer's Custom row. Clicking it sends the chord as HID keys to the selected node.
• Safety: a custom chord matching a destructive pattern (e.g. contains CTRL+ALT+DEL or SysRq) still honours the confirm-before-firing toggle.

Keyboard layout #

Where: Detail header → kbd: <layout> (read-only).

The node types into the target as a USB HID keyboard, and the character-to-keycode mapping is layout-specific and lives on the node, not the hub. A node hardcoded to US typing into a target set to a German/UK/French layout mistypes symbols (the classic "my password has a / and the node typed -" bug).

• Layout is a per-node firmware setting: KEYBOARD_LAYOUT in the node's settings.toml (us, de, uk, fr, …). Default us.
• A requested layout whose library isn't staged on the board logs a warning and falls back to US rather than failing.
• The node reports its active, resolved layout in hello; the hub shows it read-only in the header so you can see at a glance what each node is set to.

Only literal text (type / send-as-text) is layout-sensitive; named chords (keys, e.g. CTRL+C) map to keycodes directly and are unaffected.

Ping · Read serial · Reboot node #

Where: Detail header → action buttons.

Reboot node prompts for confirmation. It reboots the node hardware (HID + serial re-enumerate on reconnect); it does not reboot or power-cycle the target machine. To reset the target, use Reboot machine.

Reboot machine #

Where: Detail header → “Reboot machine” button (distinct from “Reboot node”).

“Reboot machine” reboots the attached target machine, not the Pico node. It opens a menu of three methods, each behind its own confirmation dialog:

These reset the target machine, not the node. Serial reboot is the graceful option; Ctrl+Alt+Del depends on the target's config; Magic SysRq is the hard, last-resort reboot for a hung machine.

Prompt-state badges #

Where: Node rows & detail header (colored badge, online nodes only).

Online/offline tells you a node is reachable; it doesn't tell you where the target is. The hub classifies each node's serial output into a coarse prompt-state so a per-node badge shows, across the whole fleet, which boxes sit at a login prompt, which are mid-boot, and which have panicked.

It is computed hub-side from the freshest bytes of the rolling output tail (first match wins, most-urgent first), so no firmware change is needed. It is live status, never persisted: null while offline, exposed read-only as prompt_state on GET /api/nodes/{id}, and broadcast as a node_state event to every browser so badges update without any console open. It's the same matcher the expect engine waits on.

Machine up/dead badge #

Where: Node rows & detail header (badge next to the node status, online nodes only).

Each node (Pico) is powered by the target machine's USB, so "node online" only tells you the Pico is reachable — not whether the attached machine is actually up. The hub derives a separate target liveness signal and surfaces it as its own badge, distinct from the green node dot, so you can tell a running box from a hung one at a glance.

The badge only appears for online nodes; when a node goes offline entirely (its target lost power and took the Pico with it), the whole row dims and the node dot goes hollow instead. Target liveness is live status, never persisted: it rides the node's heartbeat as target and is exposed read-only on GET /api/nodes/{id}.

A node that vanishes completely usually means the target lost power (the Pico is bus-powered from it) — see Nodes. A machine dead badge is the more interesting case: the node is alive and talking, but its host stopped enumerating — a hang, a power-off, or a target mid-reset.

Macros #

Where: Macros view, and quick-run chips in the HID composer.

A macro is a named, reusable HID sequence stored on the hub and replayed on any node in one click. Steps run top-to-bottom as a single sequence and stop on the first error.

Creating & editing

• New macro / Edit opens the editor. Give it a name and optional group.
• Add steps of three kinds: Type text (optionally append ⏎), Key / chord (e.g. ENTER, CTRL+ALT+DELETE), and Wait (a fixed pause in ms). Reorder with ↑ ↓, remove with ×.
• Dangerous checkbox — require a confirmation prompt before this macro runs (for reboots, power keys, destructive commands).

Running & deleting

• Run (table row or detail panel) replays the macro on the selected node. The composer's macro chips run the first few macros without leaving the console.
• Delete removes it after a confirmation.

Sequences & steps #

A sequence is the fire-and-forget primitive underneath macros: an ordered list of discrete steps the firmware's run_sequence understands. It is the exact shape used by the macro editor and, extended with waits, by the expect engine and runbooks.

A plain sequence just runs the steps in order and stops on the first error. To make it react to the target — wait for a prompt before responding — add wait_for steps and you have an expect job.

Expect engine #

Where: Detail header → Expect button; live progress bar under the console.

An expect job alternates action steps with wait_for steps that block until a regex appears in the node's serial output within a timeout. That is what turns PICOTTY from a keyboard into automation: "wait for login:, send the user, wait for Password:, send the password, wait for a shell prompt." All the waiting is hub-side; the node just keeps emitting output and running the dispatched steps.

Building a job

The Expect builder lets you add, reorder, and remove three step kinds:

• Send — text into the serial line (use \n / \t escapes).
• Wait-for — a regex, a timeout_ms, and on_timeout = fail (stop the job) or continue (log it, move on — for an optional prompt).
• Delay — a fixed pause.

A wait_for searches only output that arrived after the preceding action, so it can't match a stale prompt. Run expect starts the job; the live bar shows step/total, phase, and detail, with a Cancel button. Terminal states: done, failed, timeout, cancelled.

send needs a serial getty (the serial_tx capability); on a node without it, use type/keys actions to drive tty1 instead.

POST /api/nodes/{id}/expect
{"steps": [
  {"wait_for": {"regex": "login:\\s*$", "timeout_ms": 30000}},
  {"type": "send", "data": "root\n"},
  {"wait_for": {"regex": "[Pp]assword:\\s*$", "timeout_ms": 10000}},
  {"type": "send", "data": "<password>\n"},
  {"wait_for": {"regex": "[#$]\\s*$"}},
  {"type": "send", "data": "uptime\n"}
]}

Offline command queue #

Where: Detail header → Queue button.

Dispatching to an offline node normally fails. But a node is powered by its target, so it is off exactly when you often most want to act — "press ENTER at GRUB" is something you want staged before you power the box on, delivered the instant it dials in.

• Add a command to type, with an optional TTL (minutes; default 1 hour, 0/blank = no expiry). It is validated at enqueue time.
• The Pending list shows queued commands with their remaining TTL; Cancel removes one.
• On the node's next hello, the hub drains the queue in issue order — after announcing the node online. Expired rows are dropped. If the node is already online, it drains immediately.

The queue is an explicit opt-in — the default dispatch path never buffers silently. Queued commands live in a durable table, so a hub restart doesn't lose them.

Runbooks #

Where: Runbooks view.

A macro is a single sequence on one node. A runbook is the fleet-scale version: a named, durable YAML expect flow run across a whole node group, with per-node staggering and a live progress view. Each target node gets its own expect job driven by the runbook's steps.

Authoring

New runbook / Edit opens a YAML editor. A runbook is a mapping with a name and a non-empty steps list; each step is one of wait_for (with optional timeout_ms / on_timeout), send, type, keys, or delay_ms. The hub validates the YAML on save (bad_runbook on failure).

name: log-in-and-check
steps:
  - wait_for: "login:"
    timeout_ms: 30000
  - send: "root\n"
  - wait_for: "[Pp]assword:"
  - send: "<password>\n"
  - wait_for: "[#$] $"
  - type: "uptime\n"

Running

Run targets an explicit set of nodes or a whole group, with a stagger (ms between nodes). Up to 128 nodes per run. Per target: offline → skipped; already running an expect job → rejected (busy); otherwise it goes running. One node failing does not abort the fleet. The Runs pane streams a per-node status summary live.

Bulk / fleet actions #

Fleet-scale operations fan a single action out across many nodes with an optional stagger, so one slow node never blocks the rest:

• Runbook run across a group — the primary fleet-automation path (see Runbooks).
• Canary OTA rollout — update one node, wait for it to come back healthy, then stagger the rest (see OTA).
• Bulk command — dispatch one command to a list of nodes, with skip_offline and stagger_ms.

Fleet actions touch many machines at once. Prefer a staggered rollout and a small canary batch over "all at once," especially for anything destructive or for firmware.

Session recording & replay #

Where: Detail header → Replay button.

The hub stores every output chunk with a hub-stamped timestamp, so a past session — an install, a crash, a boot — can be replayed exactly as it happened, with no separate capture path. The hub exports it as an asciicast v2 recording.

• Replay plays the recording inline when the asciinema player is vendored; otherwise it offers a .cast download you can play with asciinema play <id>.cast.
• The window is bounded by since/before (ms on the hub's clock). Node timestamps are deliberately ignored — the Picos have no RTC.

It is read-only over existing data: no schema, firmware, or protocol change.

Raw serial bridge #

Where: Detail header → Bridge button; enabled hub-wide in Settings.

The bridge exposes an assigned node's serial line as a plain TCP socket, so any tool that speaks raw serial — minicom, PuTTY, conserver, esptool — attaches to a node unchanged:

minicom -D tcp:<hub-host>:<port>      # interactive serial to that node

It is off by default and opt-in per node:

• Enable the subsystem with the serial_bridge_enabled setting (Settings). Toggling binds/unbinds listeners immediately.
• Assign a node a TCP port (1024–65535, not a hub face port, not already taken). Reassign changes it; Unassign removes it. The map is durable — one port per node.
• The hub pipes socket bytes → the node's serial line and the node's output → the socket. A node with no serial_tx gets a read-only bridge (writes dropped).

Security boundary. The bridge is raw and unauthenticated — anything that can reach the port gets an interactive serial session to that node. Bind it on the management interface only and never expose a bridge port outside the isolated segment. See Security.

OTA firmware updates #

Where: Detail header → Firmware button (ota-capable nodes only); bundles + rollout via "Manage bundles…".

OTA is a hub-driven, chunked, checksummed firmware push with a known-good fallback so a bad update self-reverts. It is the highest-risk feature — a bad push deploys to hardware you may not be able to reach — so the safety rails matter.

Using it in the dashboard

• Firmware opens the per-node update sheet: pick a bundle, Update firmware, watch the live byte-progress bar. The Pico flashes, reboots, and must report healthy to confirm.
• Manage bundles… creates a bundle from a name + files (read in the browser, base64'd into the manifest) and lists existing bundles with their short SHA.
• Upload .zip stages a whole firmware at once: pick a .zip and the hub decompresses it and stages every file — no picking files one by one. The bundle name defaults to the file's basename (sanitized to [A-Za-z0-9._-]) when the name field is blank. A bad archive is rejected with a 422.
• Canary rollout updates one node, waits for it to come back healthy, then staggers the rest across picked nodes or a group.

Provenance: flashed: vs fw

The detail header shows two different things. fw is the running firmware's FW_VERSION read from the node's code.py — the version the firmware reports about itself. The muted flashed: <bundle> line (when present) is OTA provenance: the last bundle name pushed to that node (the trailing @ <timestamp> is stripped in the UI). They are independent — the bundle name is a hub-side label, not the firmware's own version — so don't read one as the other.

The safety model (four rails)

1. Writable filesystem only — a node that can't write never advertises ota, so the hub never pushes to it.
2. Integrity verified before swap — every staged file's size + SHA-256 must match before a live file is touched; a corrupt bundle is discarded and firmware is left untouched.
3. Swap keeps the old files — replaced files become .bak and a pending marker is written, then the node reloads.
4. Crash-loop auto-revert — if the new firmware hangs and the watchdog resets the node, boot recovery restores the .bak set automatically; the node returns on the old, known-good firmware with no hands-on. Once the new firmware boots, connects, and runs, it finalizes (deletes .bak + marker) so a later unrelated reset can't revert it.

When OTA fails: wipe & reflash by hand

Some failures need hands-on recovery. An OTA node's CIRCUITPY drive is read-only over USB, so use firmware/scripts/wipe-pico.sh — it reaches the node over the CircuitPython REPL on the console serial port, disables the watchdog, and reformats the filesystem back to an empty, host-writable drive; then redeploy the firmware. If the REPL can't be reached, fall back to holding BOOTSEL while plugging in and dropping a fresh CircuitPython .uf2 (or flash_nuke.uf2) onto RPI-RP2.

Alerts #

Where: Settings → Alerts toggle + endpoint fields.

For a fleet you stop watching the dashboard, so the hub tells you when something matters. Every notable event already passes through the audit path, so outbound notifications hang off it — no polling.

• Off by default. Enable alerts_enabled and set a webhook URL (receives JSON) and/or an ntfy topic URL (receives a plain-text body) in Settings.
• Which events fire: a node going node_down, a watchdog-recovery error event, and a command result whose status is failed.
• Dedup: the same (kind, node) alert is suppressed for 60 s, so a flapping node sends one alert, not a storm.
• Never blocks the loop: each notification is its own task with a 5 s timeout and a single retry; a dead endpoint can't stall the hub.

Keep the alert endpoints reachable from the isolated segment, or route them through the same tunnel you use for the dashboard. Use a placeholder like https://<your-endpoint> in these fields — never paste a secret you don't want stored in hub settings.

Telegram bot #

Where: Settings → Telegram sidecar card.

The Telegram sidecar is a separate process that reaches the hub only through its REST + WebSocket API and bridges it to a Telegram bot over outbound-only long polling — no inbound port, so it works from an isolated management VLAN. This card configures it; the bot itself runs from telegram-bot/.

• Bot token — from @BotFather. On save the hub validates it with Telegram's getMe; it is write-only and never shown back.
• Allowed chat IDs — a comma-separated allowlist checked on every update; an unknown chat gets silence, not a reply. Message the bot once and read the sidecar log to find your numeric id.
• Shell tier (break-glass) — enables /shell, /reboot and /sysrq over chat, gated behind a TOTP. Use Generate to mint a base32 secret (add it to an authenticator app), then in chat /arm <code> arms the shell for the arm window before it auto-disarms. The shell needs a node advertising serial_tx; the getty's non-echo keeps passwords out of the chat.
• Alerts — the sidecar pushes node down/up, watchdog recovery, command failures, and hub restarts, with per-node debounce and /mute.
• Install / start sidecar — one click: the hub runs the sidecar's installer (uv sync + the shared .env) and enables its service, streaming the output into the card. The status line shows sidecar running / stopped. (The service step needs passwordless sudo on the hub; if it's missing, the output tells you the one command to run.)

The hub and sidecar share one credentials file — ~/.config/picotty/telegram.env (chmod 600) — so there's nothing to align by hand. Saving from this card writes it; the sidecar reads the same file and hot-reloads the allowlist and shell/alert settings on change (a bot-token change needs a sidecar restart, and the log says so). Full reference: docs/telegram.md and telegram-bot/README.md.

Not end-to-end encrypted. Bot traffic is TLS to Telegram's servers, which can read it. Fine for stats; for the shell this is why the terminal is gated behind break-glass TOTP and the getty keeps autologin off.

Settings #

Where: Settings view.

Hub configuration. Live values apply immediately; port changes need a restart. Save config writes the changes (PATCH /api/settings).

Safety toggles

• Require confirm for destructive keys — prompt before CTRL+ALT+DEL, SysRq, and reboot.
• Serial bridge listeners — master switch for the serial bridge.
• Alerts — enable outbound notifications.

Events & audit #

Where: Events view, and the right-rail Events / Command history tabs in the Nodes view.

The hub journal: registrations, heartbeats, stale sweeps, errors, and failures, plus a Command audit of every command sent, its target, and its outcome. Failures and node-down events are tagged in accent for quick scanning.

• Export CSV downloads the event log (/api/events/export).
• The right rail's Command history tab lets you Re-run a past command on its node.

Security model #

Network isolation is the real security boundary. Both hub and nodes belong on an isolated management VLAN.

• Reach the dashboard through a tunnel — never by exposing the web port. The node-facing swarm port must never be reachable from outside the segment.
• The hello token is a second line of defense, not a substitute for isolation.
• The raw serial bridge widens the boundary: each assigned port is an unauthenticated interactive serial session. Keep bridge ports inside the segment; bind on the management interface only.
• Auth is scaffolded but off by default (network-gated). Enable it only if you ever place the hub outside an isolated segment.
• PICOTTY types real credentials into targets — treat the whole segment, the hub host, and its stored settings as sensitive. This help page uses placeholders like <hub-host> and <password>; never hardcode a real address or secret into a shared config.

← Back to the dashboard