Remote Nodes

Install Wactorz on any machine — Raspberry Pi, VM, edge device — start it with --node, and spawn agents on it from the main Wactorz chat. Agents on a node heartbeat back to the central dashboard exactly like local ones.

Overview

A node is the same package, started in a different role. It connects to the shared MQTT broker, listens for spawn commands from the main machine, and runs the agents it is sent — the same DynamicAgent, compiled from the same code, against the same agent API, under the same OTP supervisor.

[Main machine]                        [Edge device — Raspberry Pi, VM, etc.]

MainActor  ──MQTT──►  nodes/{name}/spawn  ──►  wactorz-node --node {name}
                                                   │  compiles + runs agent
                                                   │  local ONE_FOR_ONE supervisor
Dashboard  ◄──MQTT──  agents/{id}/heartbeat  ◄──┘  heartbeats every 10 s

Agents on a node appear in the central dashboard alongside local ones. The only visual difference is a node field in their heartbeat payload showing which machine they run on.

These differ on a node, and nothing else does:

Earlier releases deployed a single self-contained remote_runner.py instead, which carried its own copy of the agent contract. A node deployed that way keeps working — main reads its heartbeat as runtime: runner — and redeploying it installs the package. Agent state files carry over untouched.


Setup on the edge device

1. Install Wactorz

python3 -m venv ~/wactorz/venv
~/wactorz/venv/bin/pip install wactorz

Install the same version main is running, and at least the one these docs ship with — earlier releases have no node runtime. The two exchange spawn configs, manifests and signed control messages, and a node a release apart from main is the kind of mismatch that surfaces days later as an agent that will not start. A deploy from the dashboard pins the version for you.

Both sides hold each other to it. Main and a node work together when they are on the same release series, the same major and minor number: 1.4.0 and 1.4.3 do, 1.4.3 and 1.5.0 do not, so a patch release of the server does not mean deploying every node again.

Either way the message names the fix, /deploy <node>. The agents a node is already running are left alone, they come back after a reboot, and a stop is always obeyed.

No extra is needed: everything a node uses — aiomqtt, psutil, aiohttp — is a core dependency. An agent that needs more says so in its spawn config's install list, and the node installs it on the spot.

2. Start it as a node

~/wactorz/venv/bin/wactorz-node --node rpi-livingroom --mqtt-broker 192.168.1.10

Replace 192.168.1.10 with the IP of the machine running the MQTT broker. The --node value is the node identifier — it must be unique across all nodes and is used to address this device when spawning agents.

Command-line options

Flag Default Description
--node [NAME] $WACTORZ_NODE, else a random node-<hex> The node's name. /deploy starts a node through its own command, wactorz-node, which takes only a node's flags — an older release, which has no such command, then fails to start rather than coming up as a second server. wactorz --node still works for a hand-written launcher. /deploy writes WACTORZ_NODE into the node's .env, so wactorz --node with no name comes up under the right one. The flag is what chooses the role — the variable on its own does not, or a server that inherited it would become a node.
--mqtt-broker $MQTT_HOST MQTT broker hostname or IP. Also reads $WACTORZ_BROKER, which wins — main's own MQTT_HOST is usually localhost, and a node that adopted it would dial itself.
--mqtt-port 1883, or 8883 with MQTT_TLS=1 MQTT broker port. Also reads $WACTORZ_PORT.
--loglevel INFO DEBUG | INFO | WARNING | ERROR

The single-file runner's spellings — --name, --broker, --port — still work, so a unit written against them survives the upgrade.

Run as a service (systemd)

A deploy from the dashboard installs this for you. It picks the least privilege the node supports — root, then a user unit with lingering enabled, then passwordless sudo — and falls back to nohup only when the node offers none of them. The deploy result says which it got, so a node reported as nohup — unsupervised is one that will not come back after a reboot.

The unit it writes, for a node deployed as user pi:

[Unit]
Description=Wactorz remote node
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=5

[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/wactorz
EnvironmentFile=/home/pi/wactorz/.env
ExecStart=/home/pi/wactorz/venv/bin/wactorz-node --mqtt-broker ${WACTORZ_BROKER} --mqtt-port ${WACTORZ_PORT} --node ${WACTORZ_NODE}
Restart=on-failure
RestartSec=5
WatchdogSec=300
NotifyAccess=main
WatchdogSignal=SIGKILL
RestartPreventExitStatus=2

[Install]
WantedBy=multi-user.target

A user unit is the same minus User=, with WantedBy=default.target, and lives at ~/.config/systemd/user/wactorz-node.service.

Deploy as a user without sudo. Main keeps the deploy key for every node it deploys, so whoever holds main's environment has a shell on each one as the deploy user: as root, when that user is root or has passwordless sudo. A user unit needs neither. On the node, as root, enable lingering for the user once, and remove the system unit an earlier deploy may have installed, which a user without sudo cannot:

sudo loginctl enable-linger pi
sudo systemctl disable --now wactorz-node
sudo rm -f /etc/systemd/system/wactorz-node.service

Then take away the user's passwordless sudo, if it has one, and /deploy the node again: it installs a user unit. What needs root on the node is then yours to do there: system packages, such as python3-venv, and a /nodes run that starts with sudo. A node on main's own machine (see below) is set up differently: its user may restart its own unit and nothing else.

Four details worth knowing if you write one by hand:

Logs

Under a unit, the runner logs to the journal rather than to ~/wactorz/<node>.log:

journalctl -u wactorz-node -f              # system unit
journalctl --user-unit=wactorz-node -f     # user unit

Note the second form: --user-unit= queries the system journal for a user unit. journalctl --user -u wactorz-node looks like the same thing and is not — --user reads the invoking user's own journal, which on a default Debian or Raspberry Pi OS install does not exist (no per-user journal files are split out, and the account is not in the systemd-journal group), so it reports "No journal files were found" while the logs sit in the system journal.

The ~/wactorz/<node>.log file is only written on the nohup fallback.


Spawning agents on a remote node

From the main Wactorz chat, add a "node" field to any spawn request. The planner and main_actor both support this — or you can do it manually.

Natural language (via planner)

"deploy a temperature sensor agent to rpi-livingroom"
"spawn an agent on rpi-bedroom that reads the door sensor every 30 seconds"
"alert me when the CPU of rpi-livingroom stays above 80%"

The planner is told which nodes there are, what each machine is and what it runs. An agent that needs a machine itself -- a camera on it, or its own CPU, memory, disk or temperature -- goes on that node, and reads the machine with psutil, which every node has: no MQTT topic or Home Assistant entity has to publish those readings. A node that cannot take an agent now, being offline or deployed before nodes signed what they publish, is listed as such, and a plan that needs it says so instead.

Manual spawn via chat

{
  "name":          "temp-sensor-agent",
  "node":          "rpi-livingroom",
  "type":          "dynamic",
  "description":   "Reads temperature from DHT22 sensor",
  "poll_interval": 30,
  "max_restarts":  5,
  "restart_delay": 3.0,
  "install":       ["adafruit-circuitpython-dht"],
  "code": "
    async def setup(agent):
        await agent.log('DHT22 sensor agent ready')

    async def process(agent):
        import random   # replace with real adafruit_dht read
        temp = round(20 + random.uniform(-2, 2), 1)
        await agent.publish('sensors/temperature', {'value': temp, 'unit': 'C', 'node': agent.node})
        await agent.log(f'Temperature: {temp}C')
  "
}

The main machine publishes this config to nodes/rpi-livingroom/spawn. The runner picks it up, installs any declared "install" packages, compiles the code, and starts the agent under a local supervisor.

The packages an agent installs

The "install" list is usually written by the model, and a plausible guess at a package name is exactly what a typosquat is published to catch. So main clears the list before it sends the spawn, because the node installs whatever the spawn lists:

Only the dashboard's own button counts: the installer is handed the press by the web server in the same process and does not listen for answers on the broker, where any agent could publish one. The same applies to an agent spawned on main, and to a move to a node with /migrate, which clears the list before the agent stops. An agent put back where it already ran — restored at start, or after a move that failed — is not asked about again: its packages were cleared when it was first placed.

WACTORZ_PIP_INDEX_URL, set on main, sends the installs main makes, on main and on a node over SSH, to that index instead of PyPI; set it on a node too for the installs its runner makes. A node install goes into the node's virtualenv at ~/wactorz/venv, made there if missing, and is refused rather than made into the system Python when one cannot be made.

A library agent (type: module)

An agent the deployment brings — a function declared with @wactorz.agent, an Actor subclass — is spawned on a node by its import path, as it is on main, and built there from the package installed on the node:

{"name": "imu-anomaly", "node": "rpi-kitchen", "type": "module", "target": "imu_anomaly.agent:detect", "options": {"threshold": 4.0}}

Main sends the agent's code with the spawn: the folder its module is in, .py files only, which the node writes under ~/wactorz/state/..bundles/, checks against the hash the signed spawn names, and imports the agent from. The packages it lists in requires={"packages": [...]} are installed first. An agent's own code has to live in that folder: one that imports code of the main machine's from anywhere else is refused before it moves, naming the import — see Using Wactorz as a library. A node that has the package installed itself, with the target named in WACTORZ_AGENTS or as a wactorz.agents entry point, builds it from that and is sent no code; the targets it can build and whether it takes code (code_bundles) are in its manifest. A node from before this release takes no code, and main refuses to move a library agent there until it is deployed again. On the node the agent runs as it does on main, with no model: me.llm is None there. The models it names with models= (or MODELS on a class) go with it as artifacts: main pins them when it sends the agent, and the node fetches them before it builds it, as for any agent whose spawn names artifacts. Using Wactorz as a library has the rest.

ℹ replace flag — If an agent with the same name is already running on the node, the spawn is ignored by default. Pass "replace": true in the config to stop the old instance and spawn fresh.


Automated deploy from chat

MainActor can deploy a node to a new machine over SSH, but only to a machine you have configured as a deploy target. Add the node to your environment first:

DEPLOY_TARGETS=rpi-bedroom
DEPLOY_RPI_BEDROOM_HOST=192.168.1.52
DEPLOY_RPI_BEDROOM_USER=pi
DEPLOY_RPI_BEDROOM_KEY=/path/to/id_ed25519    # preferred; or _PASSWORD
DEPLOY_RPI_BEDROOM_BROKER=192.168.1.10        # broker as seen FROM the Pi

The block is keyed by the node name upper-cased, with every run of non-alphanumerics collapsed to one underscore — rpi-bedroom → DEPLOY_RPI_BEDROOM_*. Restart Wactorz, then from the chat:

/deploy rpi-bedroom

The installer agent SSHes in, creates ~/wactorz/, writes the node's .env, installs wactorz at main's own version into a venv, and starts it under a systemd unit. After that, the node is available for agent spawning.

The package comes from PyPI. Two things send the deploy to a wheel built on the main machine and uploaded over SFTP instead: a version that is not published yet, and a published one that turns out not to carry the node runtime. The second happens while upgrading — a release from before nodes ran the package answers to its own version number — and it is checked rather than assumed, because an older CLI ignores flags it does not know and the node would otherwise come up as a second server. A node that cannot run as one fails the deploy rather than being started.

⚠ Credentials never go through chat. /deploy takes a node name and nothing else. The older /deploy <node> <host> <user> <password> form is refused: chat messages are written to the conversation history and the chat log, so a password typed there stays on disk long after the deploy. For the same reason, don't ask an agent to SSH somewhere with a password in the request — the installer reads credentials from the environment and ignores any supplied in a task payload.

Leaving DEPLOY_<NODE>_HOST unset makes the deploy resolve <node>.local over mDNS instead. That is a single name lookup; earlier versions fell back to scanning the local /24 for open SSH ports, which is gone.

Checking on a node, and running a command there

The assistant can check on a configured node for you — "how full is the Pi's disk?", "what did rpi-kitchen log last?" — by asking the installer for one of a fixed set of operations, each of which only reads:

Operation What it shows
uptime how long the node has been up, and its load
disk free and used space on each filesystem
memory free and used memory
temperature each thermal zone's temperature
os the operating system and kernel
network each network interface and its addresses
python the Python the node's runner uses
packages the Python packages installed for the runner
node-log the last lines the runner logged (1–200, default 50), from the journal or its log file

It cannot send a node a command of its own. The assistant reads mail, web pages and device names, and a command it wrote would be only as trustworthy as everything it read; a named operation runs its fixed command whatever the request says. A command of your own you type yourself:

/nodes run rpi-bedroom df -h /home
/nodes run rpi-bedroom sudo apt-get update

It runs over SSH with the node's deploy-target credentials, as the deploy does, and its output is the reply. A slash command is read from what you typed, never from a reply the assistant wrote, and the social channels refuse them. The whole command is logged, at warning level. Only main may ask the installer for anything that reaches a node: an agent running beside it is refused.

The operations are defined in wactorz.core.node_operations, which nodes install too.

The broker has to be reachable from the node

A remote node connects back to the MQTT broker over the network, so broker in its target block is the address the node should dial — your main machine's LAN IP, not localhost.

The Home Assistant add-on's embedded broker publishes no port by default, so nothing outside the add-on can reach it: assign 1883 a host port under the add-on's Network settings, or 8883 for TLS, before a remote node can connect to it at all — whatever credentials that node holds.

Broker credentials travel with the deploy. They are written to ~/wactorz/.env on the node (mode 0600) and sourced when the runner starts, so they appear in no command line — SSH runs the launch command through a shell whose own arguments any local user can read with ps, which is why they are not passed that way. A node uses DEPLOY_<NODE>_BROKER_USER / _BROKER_PASSWORD if set, and this server's MQTT_USERNAME / MQTT_PASSWORD otherwise.

Sharing the server's account has a cost worth stating: a stolen node holds full broker access, and the broker carries the code spawned agents run. An account per node is the answer, and Wactorz can issue them.

An account per node

WACTORZ_NODE_ACCOUNTS=1 gives every deployed node an account named after it. The compose files turn it on for their own broker, and the add-ons for their embedded one; WACTORZ_NODE_ACCOUNTS=0 in .env keeps compose's nodes on the shared account. A node already deployed keeps the shared account until its next /deploy, so change MQTT_PASSWORD once every node has been deployed again. The password is derived from the same secret the signing keys come from, so nothing new is stored and a leaked password file says nothing about a signing key, and /deploy writes it into the node's ~/wactorz/.env as before. A DEPLOY_<NODE>_BROKER_USER / _BROKER_PASSWORD you set still wins.

For the broker Wactorz configures — compose's, and the add-on's embedded one — it also writes an access list into MQTT_BROKER_DIR, beside the TLS certificate. The list names what a node may use, and a topic it does not name is refused. A node may:

Everything else is closed to it: another node's nodes/<other>/... in either direction, so it can neither drive, impersonate nor watch another node; system/; and whatever else shares the broker, such as zigbee2mqtt/ or Home Assistant's discovery topics.

agents/# being open means a node can publish under any agent's topics. What main takes from there as an agent's word -- the reply to a task, its manifest, the data samples it records for the planner -- it takes only signed by the node it places that agent on (see "What a node publishes is signed" below). Main does not take an agent's withdrawal, the empty manifest that says it is gone, from that alone: one for an agent still running on main is ignored, and one for an agent on a node waits for that node's next heartbeat, which only the node can publish, and is taken only if the heartbeat no longer lists the agent. A node that has gone quiet counts too: the withdrawal waits until it speaks again, or until main gives up on it after its grace period, which removes its agents anyway. Only for a node main does not know at all, such as one that was down when main started, is a withdrawal taken at once. What else a node can publish under another agent's topics, such as its status or logs, is taken as given.

That includes tasks. A task on agents/by-name/<name>/task reaches the agent of that name on whichever node runs it, and any node may publish one, so an agent on a node is asked by whoever publishes there, another node included. Its handle_task should treat the payload as untrusted input, as it would text from the web: check what it acts on, and never run or install what a task names.

Two settings shape the list, both read by whatever writes it (the server, and compose's mqtt-certs; the add-ons offer them as node_topics and broker_accounts):

Setting What it does
WACTORZ_NODE_TOPICS More data topics for every node, comma-separated: zigbee2mqtt/#, read:weather/#. A filter prefixed read: may be read and not written. One under nodes/, agents/, main/, system/ or $ is refused, with a message, because it would open for every node what the list exists to close.
WACTORZ_BROKER_ACCOUNTS Other accounts on this broker that keep all of it, comma-separated: homeassistant, zigbee2mqtt. The server's own account (MQTT_USERNAME) is always one of them.

An agent on a node that uses a topic outside the list gets no error. The broker drops the publish and delivers nothing to the subscription, which is how MQTT refuses, so the agent looks idle rather than broken. If an agent that works on the server goes quiet on a node, add its topic prefix to WACTORZ_NODE_TOPICS.

List every other account once a node is deployed. Mosquitto gives an account the list does not mention no access at all, so with a node deployed, Home Assistant's or zigbee2mqtt's account on this broker stops working until it is in WACTORZ_BROKER_ACCOUNTS. The server names the accounts that keep the whole broker in its log whenever it writes the list. With no node deployed nothing is closed: the list gives every account the whole broker.

The compose broker picks up a new node's account on its own: it watches that folder, reloads for a new account, and restarts itself if the access list or the certificate appeared for the first time, since mosquitto reads those only at startup.

Before any node is deployed, two lines in the broker log are expected:

Warning: ACL pattern '#' does not contain '%c' or '%u'.
Warning: ACL pattern '$SYS/#' does not contain '%c' or '%u'.

They are the two lines that give every account the whole broker while there is no node to keep out of anything. Mosquitto notes that neither names a client, which is the point — "everything" is not something %u can spell — and the alternative spelling (topic instead of pattern) applies to anonymous clients only, which would leave every named account with no access at all.

Only turn this on where the broker has those accounts. On a broker you run yourself, create them there first (or keep using DEPLOY_<NODE>_BROKER_USER). A node presenting an account its broker has never heard of is simply refused.

The rollout is per node. Every node keeps the shared account until its next /deploy, and nothing changes for it until then — so an install with the option on and nodes not yet redeployed is exactly as contained as it was. Once the last node has been redeployed, change MQTT_PASSWORD (and the broker's own account) so the account they used to share no longer opens anything.

What it does not do: a node's own agents share the node's account, so the containment boundary is the machine, not the agent. Nothing stops an agent publishing readings another agent reads — that traffic is a commons by design. The access list names the nodes you have configured, so it covers every node that exists; a node can still write under a nodes/<name>/... that no node has yet, and leave a message waiting for one deployed later. Mosquitto cannot express "every node's topics except your own" — a deny beats every allow, including the node's own — so naming them is the only shape available. Two things close the rest: /deploy clears whatever is retained on that node's control topics before the node starts and subscribes, and WACTORZ_NODE_SIGNING=enforce makes a node refuse anything main did not sign for it.

Signed commands

Everything main tells a node to do — spawn an agent, apply a desired state, stop, restart, migrate — is signed with a key derived for that node. /deploy writes the key to the node's ~/wactorz/.env with its broker credentials, so it travels over SSH and never over the broker. The signature rides in the message's MQTT v5 user properties, so the payload is unchanged.

What a node does with a command that is not signed for it is set by WACTORZ_NODE_SIGNING on the server, and written to the node when it is deployed:

A node takes the mode at its next /deploy: one deployed while the default was warn keeps warning until then.

A node deployed before signing holds no key and acts on everything, as it always did; deploy it again to give it one. A node remembers the commands it has accepted, so a captured one cannot be replayed, and refuses one published before its latest deploy. Once a node reports that it checks, main republishes its desired state signed, replacing one retained from before that the node would otherwise refuse on every reboot.

The keys are derived from a secret in <WACTORZ_STATE_DIR>/node_signing.key. Back it up with the rest of the state directory. To rotate the keys, delete it and deploy every node again: until a node is redeployed, it reports or refuses what the new secret signs.

What a node publishes is signed

The other direction uses the same key. A node signs everything it publishes, and main checks the signature against the node it expects to hear from: for a node's heartbeat and reports, the node whose topic it is; for an agent's task reply, manifest or data samples, the node main's own records place the agent on, never the one a heartbeat claims. An agent on main is spoken for by no node: main reads its manifest from the agent itself. An agent main has no record of, one a node started by itself, belongs to the node that first speaks for it.

So code running on one node, the sandbox included, can speak only for that node's agents. It cannot answer a task sent to another node's agent before that agent does, since main waits for the reply its own node signed. And it cannot describe another agent to the planner. A signature names the machine, not the agent on it: an agent can sign as any other agent on its own node.

A message main cannot check follows WACTORZ_NODE_SIGNING, as a command does:

Each heartbeat is taken once, so one captured and delivered again does not make a node that has gone away look up.

Encrypted connections (TLS)

A node's broker connection carries its account's password and the code of every agent spawned on it, so it should be encrypted on any network you do not fully trust. Wactorz sets that up with no certificate to buy or renew:

DEPLOY_<NODE>_BROKER_TLS=on uses TLS even when the check fails — the broker may not be up yet — off never does, and DEPLOY_<NODE>_BROKER_TLS_PORT changes the port. The node has to be able to reach that port.

A client trusting the generated CA checks that the broker's certificate came from it, but not the host name inside it: that CA signs nothing but this broker, and a node should not lose its connection because the broker's LAN address changed.

A certificate of your own. Set MQTT_TLS_CA on the server to the CA that signed your broker's certificate, or to system for one from a public CA such as Let's Encrypt. /deploy hands that to the node instead, and the host name is then checked, so give each target a broker name the certificate carries. MQTT_TLS_CHECK_HOSTNAME=1 or 0 decides the host name check either way.

The server's own connection uses TLS with MQTT_TLS=1: the server then dials MQTT_TLS_PORT (default 8883) instead of MQTT_PORT, under compose too. It creates the generated CA and certificate when it starts if they are missing, and writes the broker's copy to MQTT_BROKER_DIR (infra/mosquitto/generated in .env.template). A CA that cannot be loaded stops it at startup, naming the file it looked for. Without MQTT_TLS the connection stays plain, which suits a broker beside the server rather than across a network.

A node started by hand reads the same settings from its environment: MQTT_TLS=1, MQTT_TLS_CA naming a copy of <WACTORZ_STATE_DIR>/mqtt_tls/ca.crt, and MQTT_TLS_CHECK_HOSTNAME=0, with --port 8883. A runner told to use TLS that cannot load its CA refuses to start rather than connecting unverified.

A broker you run yourself can serve the generated certificate: python -m wactorz.broker_certificates --export <dir> writes broker.crt, with the CA after it, and broker.key there, for its certfile and keyfile; --name adds an address it is reached by.

Back up mqtt_tls/ with the rest of the state directory. Should ca.key be lost, delete ca.crt and ca.key and deploy every node again: a new CA is trusted by no node holding the old one.

Host key verification

SSH host keys are checked on every connection. A machine that has not been connected to before has its key recorded on first contact — the same trust-on-first-use that interactive ssh does — and any later change to that key fails the connection instead of being accepted.

Learned keys live in <WACTORZ_STATE_DIR>/known_hosts, or wherever DEPLOY_KNOWN_HOSTS points. Set DEPLOY_STRICT_HOST_KEYS=1 to turn off first-use learning entirely; every target's key must then already be in the file:

ssh-keyscan 192.168.1.52 >> "$DEPLOY_KNOWN_HOSTS"   # after verifying the fingerprint

A node on main's own machine

Main holds every credential the deployment has: the model's API key, Home Assistant's token, the SSH keys for your nodes, the key that signs commands to them. Code a model writes, possibly from an email or a web page it read, is safer run somewhere that holds none of them. A node holds none of them. A node on main's own machine, under a user of its own, gives that code a place to run without a second machine.

On Linux with systemd, set it up once, as root, from the directory main runs in:

sudo .venv/bin/wactorz-node setup-local

It makes a system user, wactorz-node, with its home in /var/lib/wactorz-node. It makes an SSH key in main's state directory and lets that user accept it from this machine only. It lets that user restart its own unit and run nothing else as root. It installs the unit, and it prints the lines to add to main's .env:

DEPLOY_TARGETS=sandbox          # with your other nodes, if you have any
DEPLOY_SANDBOX_HOST=127.0.0.1
DEPLOY_SANDBOX_USER=wactorz-node
DEPLOY_SANDBOX_KEY=/path/to/state/sandbox_node_ed25519

Restart main, then /deploy sandbox. The node is called sandbox because local and main already mean main itself: /migrate <agent> local moves an agent to main, and the dashboard lists main's agents under local. No node can be deployed under either name. From there it is a node like any other: /deploy sandbox again upgrades it, and the packages its agents install reach it the way they reach every node. SSH has to be running on the machine, and AllowUsers in sshd_config, if you use it, has to include wactorz-node.

The unit is sandboxed, which the units on other nodes are not: only ~/wactorz/state is writable, the venv it runs from is read-only, the homes and main's .env and state directory are hidden, and it has no devices, no capabilities and no way to gain privilege. A deploy restarts this unit and never rewrites it, so the sandbox can be changed only by running setup-local again. An agent that needs a camera or GPIO belongs on a node with the device.

The node's user must not be able to read main's secrets. Inside the unit they are hidden, but a package install runs outside it, as that user. So setup stops while main's .env or state directory is readable by wactorz-node. Run it again with --fix-permissions to make them readable by main's user alone. --main-user, --env-file and --state-dir name main's user and files, where sudo and the current directory do not.

The node's log is in the system journal, which its user cannot read, so a deploy that fails cannot bring back the log's last lines. Read them with journalctl -u wactorz-node.

In the compose stack the node is a container instead: see the sandbox node. It needs no DEPLOY_TARGETS entry, since nothing reaches it over SSH: it gets a broker account, as the node WACTORZ_MODEL_CODE_NODE names does.

Sending model-written code there

Once there is a node named sandbox — a deploy target, or a node main hears from — a <spawn> whose code the model wrote, naming no node, runs there rather than in main's process. The chat says so: "Spawned 'scraper' on node 'sandbox'". WACTORZ_MODEL_CODE_NODE names another node instead, and main keeps the code on main.

The model's agents already running on main move to the sandbox the first time it reports after a start, as /migrate would move them, state and all, and the chat lists them. One stays on main, named in the chat with the reason, when it reads the history without await (agent.query_ts(...) rather than await agent.query_ts(...)), which only main answers that way, or when its move is refused. The agents move one after another, each as /migrate would move it, so with many it takes a while. A move clears the agent's packages as /migrate does: an agent with a package not on the list asks you about it on the dashboard, and one whose question goes unanswered stays on main, which the chat says. An agent that runs main's own classes is not moved: an LLM agent, a rule, a schedule, a catalogue or library agent. Nor is an agent restored from main's records, or one you move with /migrate. The agents the planner spawns for a plan go the same way: one sent to the node is waited for until the node reports it running, and the plan's steps reach it through main.

A spawn the model wrote that names main is asking for main's credentials, and WACTORZ_MODEL_CODE_ON_MAIN decides what happens to it. With ask, the default, the dashboard asks you, with Run on main and Refuse, and only the dashboard's button counts, never an answer on the broker. Refused, or not answered within two minutes, it is not spawned, and for ten minutes the same spawn is refused without asking again. allow lets it run on main without asking, and never refuses it.

If the node named is neither one of DEPLOY_TARGETS nor a node main has heard from, such a spawn is refused, never quietly run on main. If it is, but the node is not running, the agent is sent all the same and starts when the node does; the chat says so.


MQTT topics

The runner subscribes to a set of control topics scoped to its node name, and publishes heartbeats for itself and all its agents.

Topic Direction Description
nodes/{name}/spawn → runner Spawn a new agent. Payload: full agent config dict. Not retained — a node that was away catches up from desired_state. Signed.
nodes/{name}/desired_state → runner Every agent the node should be running. Retained; the runner starts any that are missing when it connects, and stops (keeping its state) any it runs that the list leaves out, except one a migration placed there and main has not confirmed yet. Signed.
nodes/{name}/stop → runner Stop a named agent. Payload: {"name": "agent-name"}; "delete": true also removes its state file and clears its retained topics, and "keep_topics": true keeps those for an agent that lives on elsewhere under the same id, as a rolled-back migration does. Signed.
nodes/{name}/stop_all → runner Stop all agents and shut down the runner. Signed.
nodes/{name}/list → runner Request the list of running agents. Response on nodes/{name}/agents.
nodes/{name}/agents ← runner Response to list. Contains agent names and actor IDs.
nodes/{name}/heartbeat ← runner Runner heartbeat every 10 s. Contains node name, Wactorz version, runtime kind, agent count, broker address, whether the node checks signed commands, and how close the machine is to running out: CPU, memory (within a container's or a unit's memory limit), swap, load, free disk where the state is kept, the CPU's temperature and, on a Raspberry Pi, what is throttling it. A reading the node cannot take is null.
nodes/{name}/manifest ← runner What the machine is, retained: architecture, system, Python, model, whether it is a container, CPUs and memory (within any limit), disk, accelerators, the devices an agent can use, the installed packages, the library agents it can build (agents, by import path), and whether it clears its agents' copies on a reset (resets). Sent when the runner starts and after an install; cleared by /nodes remove.
nodes/{name}/migrate → runner Hand a running agent back to main, which places it. Payload: {"name": "...", "target_node": "@main", "return_token": "...", "force": false, "max_state_bytes": 8388608}. Signed.
nodes/{name}/migrate_result ← runner Result of a migration request. A failure, including a refusal, is shown on the dashboard.
nodes/{name}/code_changed ← runner An agent here repaired its own program. Carries the agent's name and no code.
nodes/{name}/code_request → runner Asks for the program an agent is actually running. Payload: {"agent": "...", "token": "..."}. Signed.
nodes/{name}/code_return ← runner The program, quoting the token it was asked with.
nodes/{name}/reset → runner Clear one agent's chat, state or counters here, or every agent's, as main has cleared its own records. Payload: {"scope": "chat" \| "state" \| "metrics", "agent": "..." or null, "request": "..."}. Signed.
nodes/{name}/reset_done ← runner The reset done, quoting request and naming the agents it cleared ("cleared": [...]); a scope the node does not reach into is answered with none.
nodes/{name}/reply/{id} ← runner Reply routing for agent.send_to() calls originating on this node.
agents/{id}/heartbeat ← agent Per-agent heartbeat every 10 s. Includes "node": "{name}" field.
agents/{id}/logs ← agent Log messages from agent.log() and agent.alert().
agents/by-name/{name}/task → agent Task addressed to a named agent on any node. Runner routes to local agents by name. Any node may publish one, so its payload is untrusted input.

Why a repair travels in two steps

An agent can be repaired by the LLM while it runs, and main's spawn registry has to learn about it or the next reboot sends the node the program that failed.

The node never volunteers the code. It publishes a notice that something changed; main then asks for the program on a signed topic, quoting a token it minted, and files the answer only for that token, that agent and that node, spending the token on use.

The asymmetry is the point. Main executes what a node sends it, so a node that could publish code straight into the registry could put it there for any agent it hosts — and that code would run wherever the agent went next, including on main. It is the same reason node-to-node migration is routed through main, and why an agent leaving a node is placed from main's copy of its program rather than the one the node sends back. A forged notice buys nothing: main asks the node, and the node answers with what it is really running.

A program filed this way is no longer trusted, even for a catalogue agent: a repair is model-written, and the node is not trusted to say what it wrote. It meets the safety scanner wherever the agent is spawned next, main included.

Two limits keep a notice from being worth sending. Main asks about one agent at a time and no more often than every ten seconds, because each question is a signed message and signing records a sequence number to disk — a rate meant for an operator's commands, not for whatever a node chooses to announce. And an answer that quotes a valid token but names the wrong agent or arrives from the wrong node is dropped without spending the question, so someone who reads a token off the broker cannot use it to cancel the exchange and leave the real answer refused.


Supervisor behaviour

Each agent on a remote node runs under a local ONE_FOR_ONE supervisor — identical semantics to the main machine. If an agent crashes, the supervisor restarts it with exponential back-off:

delay = min(restart_delay * (2 ** (crashes_in_a_row - 1)), 60.0)
# restart_delay=3.0: 3s → 6s → 12s → 24s → 48s → 60s (cap)

An agent that stays up for restart_window (60 s) after a restart has recovered: its next crash starts from restart_delay again.

Scenario Behaviour
Crash in process() Back-off, restart. After 5 consecutive failures in one run, escalates to supervisor for a clean restart.
Crash in setup() Fatal — supervisor stops. Broken code won't fix itself on retry.
Compile error Fatal — supervisor stops immediately.
max_restarts crashes in a row Restarts slow down: 5 minutes, then doubling up to an hour. Main says so in chat, once, and again when the agent recovers. It is never given up on automatically — delete it to stop it.
Deliberate stop command No restart — clean shutdown.

Default values: max_restarts=5, restart_delay=3.0. Override per agent in the spawn config.


Agent API on the edge

The agent object available inside remote agent code mirrors the local DynamicAgent API. All of the following work identically:

Method Description
await agent.publish(topic, data) Publish to any MQTT topic via the shared broker.
await agent.log(message) Log to agents/{id}/logs — visible in the central dashboard.
await agent.alert(message, severity) Publish an alert. Levels: info, warning, error.
agent.persist(key, value) Write to ~/wactorz/state/<name>_state.json (JSON, not pickle — portable, and what a migration carries). Bytes, numpy arrays, torch tensors and modules, and scikit-learn-family models are kept as files of their own beside it. A datetime, date, time or timedelta, a set or frozenset, a tuple, and a dict whose keys are not strings are written in a marked form and come back as what they were. Any other value that cannot be written as JSON is not kept: it is dropped with a warning naming it, and the rest is saved.
agent.recall(key) Read a persisted value.
agent.state In-memory dict, not persisted.
await agent.send_to(name, payload) Send a task to any agent (local or remote) via MQTT request/reply. Times out after 30 s by default.
agent.node The node name this agent is running on (e.g. "rpi-livingroom").
agent.agents() List of all agents running on this node.

⚠ No agent.subscribe() on edge — The remote runner does not implement agent.subscribe(). For MQTT subscriptions in remote agents, open an aiomqtt.Client directly inside setup() — the broker address is available as the machine's IP passed to --broker.


Agent migration

A running agent can be moved between main and a node, or from one node to another, from the chat:

/migrate temp-sensor rpi-bedroom     # to a node
/migrate temp-sensor local           # back to main
/migrate temp-sensor rpi-bedroom --force

Main routes every migration; a node never spawns on another node.

An agent leaving a node, for main or for another node, is placed from main's own copy of its program: the node hands back its state, and the program it sends with it — generated code, or a library agent's bundle — is compared and discarded, never run. A move on to another node sends the bundle main builds from its own copy of the agent's code, and is checked before the source stops the agent, as a move from main is. A program the node repaired reaches main's copy through the exchange in Why a repair travels in two steps; one main has not fetched yet is lost, and the announcement of the move says the node's program was discarded. An agent main holds no program for — one in no spawn registry entry, and not a library agent installed on main, though a node runs it — cannot leave its node, and /migrate says so before anything stops.

A node runs an agent whose program goes with it: generated code, or an LLM agent (type: llm), for which main writes the code. A library agent — a decorated function or an Actor subclass registered with the system, type: module — it builds from the package installed there, so one moves only to a node whose manifest names its target; see A library agent. An agent built into the server — a native catalogue agent such as flic or weather-agent, a Home Assistant actuator, a scheduled or rule agent — has no program in its config, so moving or spawning one on a node is refused with the reason, and it stays where it is. The agent is stopped where it runs, its persisted state is sent to where it is going with main's copy of its config, and the source keeps its own copy until the destination confirms the agent started. Only then is the source's copy deleted. A destination that never confirms within five minutes is told to drop the agent, and the agent is started again where it was, from the copy that was kept.

The state is taken after the agent has stopped, so what its on_stop writes last (a final counter, an LLM agent's last turn) goes with it.

State travels as JSON — a date, a set, a tuple or a dict keyed by numbers in the marked form that brings it back as it was — and the agent's blobs — bytes, numpy arrays, torch tensors and modules, scikit-learn-family models — travel beside it, each in chunks on a topic of its own, checked against its SHA-256 when it arrives. A migration is refused, and the agent keeps running where it is, when:

A destination waits for an agent's blobs before starting it, for at least a minute and longer for large ones. One that does not arrive leaves the agent where it was, as a destination that never confirms does.

A node checks its own agents against the same two terms, which main sends with the request. A node running an earlier release does not check them: it sends what it can, and main names the keys it could not send.


Debugging

Verbose diagnostics

Run with --loglevel DEBUG to see every connection event, subscribe and publish:

~/wactorz/venv/bin/wactorz-node --node rpi-test --mqtt-broker 192.168.1.10 --loglevel DEBUG

A node refuses to start, with exit status 2, in two cases it can detect up front: a node name containing an MQTT wildcard, which the broker would refuse on every operation; and MQTT_TLS turned on with a CA that cannot be read. Both would otherwise be a runner reconnecting every three seconds for ever.

Watch node traffic from the main machine

mosquitto_sub -h localhost -t 'nodes/#' -v
mosquitto_sub -h localhost -t 'agents/+/heartbeat' -v

List agents on a node

mosquitto_pub -h localhost -t "nodes/rpi-livingroom/list" -m '{}'
mosquitto_sub -h localhost -t "nodes/rpi-livingroom/agents" -C 1

Stop a specific remote agent

mosquitto_pub -h localhost -t "nodes/rpi-livingroom/stop" \
  -m '{"name": "temp-sensor-agent"}'

Stop the entire runner

mosquitto_pub -h localhost -t "nodes/rpi-livingroom/stop_all" -m '{}'