Remote Nodes
Install Wactorz on any machine — Raspberry Pi, VM, edge device — start it with --node, and spawn agents on it from the main Wactorz chat. Agents on a node heartbeat back to the central dashboard exactly like local ones.
Overview
A node is the same package, started in a different role. It connects to the shared MQTT broker, listens for spawn commands from the main machine, and runs the agents it is sent — the same DynamicAgent, compiled from the same code, against the same agent API, under the same OTP supervisor.
[Main machine] [Edge device — Raspberry Pi, VM, etc.]
MainActor ──MQTT──► nodes/{name}/spawn ──► wactorz-node --node {name}
│ compiles + runs agent
│ local ONE_FOR_ONE supervisor
Dashboard ◄──MQTT── agents/{id}/heartbeat ◄──┘ heartbeats every 10 s
Agents on a node appear in the central dashboard alongside local ones. The only visual difference is a node field in their heartbeat payload showing which machine they run on.
These differ on a node, and nothing else does:
- Publishing goes through a bounded in-memory queue rather than the server's SQLite-backed outbox. A node has no outbox, so the queue is sized for a long outage and every dropped message is counted.
- Agent state is JSON under
~/wactorz/state/rather than a pickle, so it survives the node and main running different Python versions — and so a migration can ship it over MQTT. - The LLM stays on main.
agent.llm.chat(...)works on a node; the request travels to main, which makes the call. No API key is ever deployed to a node. Main answers each node up toWACTORZ_BRIDGE_CALLS_PER_MINUTEcalls a minute (60), and history pages up toWACTORZ_BRIDGE_HISTORY_PAGES_PER_MINUTE(600); past that, a call comes back as an[LLM error: ...]text saying why, and a history read fails and says so in the node's log. - The history stays on main too.
await agent.query_ts(...),query_detections,query_ha_statesandts_statsread main's database over the broker (main/history_request), signed like an LLM request, a page at a time. On a node they must be awaited. Any node main deployed may read all of main's recorded history: sensor readings, detections and Home Assistant state changes. - Tasks arrive on an MQTT topic rather than in a mailbox, and the reply goes back the same way.
Earlier releases deployed a single self-contained
remote_runner.pyinstead, which carried its own copy of the agent contract. A node deployed that way keeps working — main reads its heartbeat asruntime: runner— and redeploying it installs the package. Agent state files carry over untouched.
Setup on the edge device
1. Install Wactorz
python3 -m venv ~/wactorz/venv
~/wactorz/venv/bin/pip install wactorz
Install the same version main is running, and at least the one these docs ship with — earlier releases have no node runtime. The two exchange spawn configs, manifests and signed control messages, and a node a release apart from main is the kind of mismatch that surfaces days later as an agent that will not start. A deploy from the dashboard pins the version for you.
Both sides hold each other to it. Main and a node work together when they are on the same release series, the same major and minor number: 1.4.0 and 1.4.3 do, 1.4.3 and 1.5.0 do not, so a patch release of the server does not mean deploying every node again.
- Main reads the version in each node's heartbeat and refuses to spawn an agent on, or migrate one to, a node on another series.
/nodesmarks such a node. - Main states its own version on every command it sends a node, and a node refuses a new agent from a server on another series, with an error in the dashboard's log. That covers the moments main's check cannot: it has not yet heard the node's version, or heard it before the node was installed again.
Either way the message names the fix, /deploy <node>. The agents a node is already running are left alone, they come back after a reboot, and a stop is always obeyed.
No extra is needed: everything a node uses — aiomqtt, psutil, aiohttp — is a core dependency. An agent that needs more says so in its spawn config's install list, and the node installs it on the spot.
2. Start it as a node
~/wactorz/venv/bin/wactorz-node --node rpi-livingroom --mqtt-broker 192.168.1.10
Replace 192.168.1.10 with the IP of the machine running the MQTT broker. The --node value is the node identifier — it must be unique across all nodes and is used to address this device when spawning agents.
Command-line options
| Flag | Default | Description |
|---|---|---|
--node [NAME] |
$WACTORZ_NODE, else a random node-<hex> |
The node's name. /deploy starts a node through its own command, wactorz-node, which takes only a node's flags — an older release, which has no such command, then fails to start rather than coming up as a second server. wactorz --node still works for a hand-written launcher. /deploy writes WACTORZ_NODE into the node's .env, so wactorz --node with no name comes up under the right one. The flag is what chooses the role — the variable on its own does not, or a server that inherited it would become a node. |
--mqtt-broker |
$MQTT_HOST |
MQTT broker hostname or IP. Also reads $WACTORZ_BROKER, which wins — main's own MQTT_HOST is usually localhost, and a node that adopted it would dial itself. |
--mqtt-port |
1883, or 8883 with MQTT_TLS=1 |
MQTT broker port. Also reads $WACTORZ_PORT. |
--loglevel |
INFO |
DEBUG | INFO | WARNING | ERROR |
The single-file runner's spellings — --name, --broker, --port — still work, so a unit written against them survives the upgrade.
Run as a service (systemd)
A deploy from the dashboard installs this for you. It picks the least
privilege the node supports — root, then a user unit with lingering enabled,
then passwordless sudo — and falls back to nohup only when the node offers
none of them. The deploy result says which it got, so a node reported as
nohup — unsupervised is one that will not come back after a reboot.
The unit it writes, for a node deployed as user pi:
[Unit]
Description=Wactorz remote node
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=5
[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/wactorz
EnvironmentFile=/home/pi/wactorz/.env
ExecStart=/home/pi/wactorz/venv/bin/wactorz-node --mqtt-broker ${WACTORZ_BROKER} --mqtt-port ${WACTORZ_PORT} --node ${WACTORZ_NODE}
Restart=on-failure
RestartSec=5
WatchdogSec=300
NotifyAccess=main
WatchdogSignal=SIGKILL
RestartPreventExitStatus=2
[Install]
WantedBy=multi-user.target
A user unit is the same minus User=, with WantedBy=default.target, and lives
at ~/.config/systemd/user/wactorz-node.service.
Deploy as a user without sudo. Main keeps the deploy key for every node it deploys, so whoever holds main's environment has a shell on each one as the deploy user: as root, when that user is root or has passwordless sudo. A user unit needs neither. On the node, as root, enable lingering for the user once, and remove the system unit an earlier deploy may have installed, which a user without sudo cannot:
sudo loginctl enable-linger pi
sudo systemctl disable --now wactorz-node
sudo rm -f /etc/systemd/system/wactorz-node.service
Then take away the user's passwordless sudo, if it has one, and /deploy the
node again: it installs a user unit. What needs root on the node is then yours to do there: system packages,
such as python3-venv, and a /nodes run that starts with sudo. A node on main's
own machine (see below) is set up differently: its user may restart its own unit
and nothing else.
Four details worth knowing if you write one by hand:
Restart=on-failure, notalways./nodes shutdownexits cleanly on purpose. UnderRestart=alwaysthat command restarts the node instead of stopping it.WatchdogSec=300. The node tells systemd, from its event loop, that the loop is running, and systemd restarts it when it has not heard for five minutes. Without it a node frozen in blocking code is a process that exists and does nothing, whichRestart=never sees.WatchdogSignal=SIGKILLbecause systemd's default aborts the process and writes a core dump each time. Before it gets that far the node's own log says where the loop is stuck, in a line starting[loop]. A node not started by systemd sends nothing.RestartPreventExitStatus=2. Exit 2 means a node name that can never work — it contains an MQTT wildcard — or a malformedExecStart. Neither succeeds on retry, so restarting is just noise.- Every argument comes from
~/wactorz/.env, which the deploy writes at mode 0600 withWACTORZ_NODE,WACTORZ_BROKER,WACTORZ_PORT, when the broker needs themMQTT_USERNAME/MQTT_PASSWORD, and theMQTT_TLSsettings when the node reaches the broker over TLS. Values are quoted so the same file is safe both sourced by a shell and read by systemd — but note that aVAR=$(cmd)you add by hand is executed by the shell and taken literally by systemd, so avoid them.
Logs
Under a unit, the runner logs to the journal rather than to
~/wactorz/<node>.log:
journalctl -u wactorz-node -f # system unit
journalctl --user-unit=wactorz-node -f # user unit
Note the second form: --user-unit= queries the system journal for a user
unit. journalctl --user -u wactorz-node looks like the same thing and is not —
--user reads the invoking user's own journal, which on a default Debian or
Raspberry Pi OS install does not exist (no per-user journal files are split out,
and the account is not in the systemd-journal group), so it reports
"No journal files were found" while the logs sit in the system journal.
The ~/wactorz/<node>.log file is only written on the nohup fallback.
Spawning agents on a remote node
From the main Wactorz chat, add a "node" field to any spawn request. The planner and main_actor both support this — or you can do it manually.
Natural language (via planner)
"deploy a temperature sensor agent to rpi-livingroom"
"spawn an agent on rpi-bedroom that reads the door sensor every 30 seconds"
"alert me when the CPU of rpi-livingroom stays above 80%"
The planner is told which nodes there are, what each machine is and what it runs.
An agent that needs a machine itself -- a camera on it, or its own CPU, memory, disk
or temperature -- goes on that node, and reads the machine with psutil, which every
node has: no MQTT topic or Home Assistant entity has to publish those readings. A
node that cannot take an agent now, being offline or deployed before nodes signed
what they publish, is listed as such, and a plan that needs it says so instead.
Manual spawn via chat
{
"name": "temp-sensor-agent",
"node": "rpi-livingroom",
"type": "dynamic",
"description": "Reads temperature from DHT22 sensor",
"poll_interval": 30,
"max_restarts": 5,
"restart_delay": 3.0,
"install": ["adafruit-circuitpython-dht"],
"code": "
async def setup(agent):
await agent.log('DHT22 sensor agent ready')
async def process(agent):
import random # replace with real adafruit_dht read
temp = round(20 + random.uniform(-2, 2), 1)
await agent.publish('sensors/temperature', {'value': temp, 'unit': 'C', 'node': agent.node})
await agent.log(f'Temperature: {temp}C')
"
}
The main machine publishes this config to nodes/rpi-livingroom/spawn. The runner picks it up, installs any declared "install" packages, compiles the code, and starts the agent under a local supervisor.
The packages an agent installs
The "install" list is usually written by the model, and a plausible guess at a package name is exactly what a typosquat is published to catch. So main clears the list before it sends the spawn, because the node installs whatever the spawn lists:
- A package on the list of packages installed without asking goes through at once. That list holds what the catalogue and the examples install, the packages the installer knows by their import names (
cv2→opencv-python), those named inWACTORZ_PIP_ALLOWon main, and those you have allowed since. - A package already installed where the agent runs is not asked about on main, since nothing is installed. A node installs whatever the spawn lists, so there every package not on the list is asked about.
- Anything else is asked about on the dashboard: a notice that stays until you answer names the agent and the packages, with Install and Refuse. The same question is in the installer's thread in Chat, which opens while it asks, so a notice you dismissed is not lost. Install allows them from then on, and lets through any other spawn waiting for the same packages. Refuse, or no answer within two minutes, means the agent is not sent, and the dashboard says why; a question nobody answered closes as "No answer in time". For ten minutes after either, a request for the same package is refused without asking (main's log says when it was refused), so a model retrying the spawn does not put the question to you again;
/packages allowlifts that at once. /packagesshows the list,/packages allow <name> …adds to it and/packages forget <name> …takes off what you added. Forgetting applies to agents placed from then on: an agent already saved keeps the packages it was placed with, and is reinstalled with them when it is restored, so that a restart never waits on you. Delete the agent to be rid of them. Typing/deploy-pkgin the terminal allows the packages it names.
Only the dashboard's own button counts: the installer is handed the press by the web server in the same process and does not listen for answers on the broker, where any agent could publish one. The same applies to an agent spawned on main, and to a move to a node with /migrate, which clears the list before the agent stops. An agent put back where it already ran — restored at start, or after a move that failed — is not asked about again: its packages were cleared when it was first placed.
WACTORZ_PIP_INDEX_URL, set on main, sends the installs main makes, on main and on a node over SSH, to that index instead of PyPI; set it on a node too for the installs its runner makes. A node install goes into the node's virtualenv at ~/wactorz/venv, made there if missing, and is refused rather than made into the system Python when one cannot be made.
A library agent (type: module)
An agent the deployment brings — a function declared with @wactorz.agent, an Actor subclass — is spawned on a node by its import path, as it is on main, and built there from the package installed on the node:
{"name": "imu-anomaly", "node": "rpi-kitchen", "type": "module", "target": "imu_anomaly.agent:detect", "options": {"threshold": 4.0}}
Main sends the agent's code with the spawn: the folder its module is in, .py files only, which the node writes under ~/wactorz/state/..bundles/, checks against the hash the signed spawn names, and imports the agent from. The packages it lists in requires={"packages": [...]} are installed first. An agent's own code has to live in that folder: one that imports code of the main machine's from anywhere else is refused before it moves, naming the import — see Using Wactorz as a library. A node that has the package installed itself, with the target named in WACTORZ_AGENTS or as a wactorz.agents entry point, builds it from that and is sent no code; the targets it can build and whether it takes code (code_bundles) are in its manifest. A node from before this release takes no code, and main refuses to move a library agent there until it is deployed again. On the node the agent runs as it does on main, with no model: me.llm is None there. The models it names with models= (or MODELS on a class) go with it as artifacts: main pins them when it sends the agent, and the node fetches them before it builds it, as for any agent whose spawn names artifacts. Using Wactorz as a library has the rest.
ℹ replace flag — If an agent with the same name is already running on the node, the spawn is ignored by default. Pass
"replace": truein the config to stop the old instance and spawn fresh.
Automated deploy from chat
MainActor can deploy a node to a new machine over SSH, but only to a machine you have configured as a deploy target. Add the node to your environment first:
DEPLOY_TARGETS=rpi-bedroom
DEPLOY_RPI_BEDROOM_HOST=192.168.1.52
DEPLOY_RPI_BEDROOM_USER=pi
DEPLOY_RPI_BEDROOM_KEY=/path/to/id_ed25519 # preferred; or _PASSWORD
DEPLOY_RPI_BEDROOM_BROKER=192.168.1.10 # broker as seen FROM the Pi
The block is keyed by the node name upper-cased, with every run of non-alphanumerics collapsed to one underscore — rpi-bedroom → DEPLOY_RPI_BEDROOM_*. Restart Wactorz, then from the chat:
/deploy rpi-bedroom
The installer agent SSHes in, creates ~/wactorz/, writes the node's .env, installs wactorz at main's own version into a venv, and starts it under a systemd unit. After that, the node is available for agent spawning.
The package comes from PyPI. Two things send the deploy to a wheel built on the main machine and uploaded over SFTP instead: a version that is not published yet, and a published one that turns out not to carry the node runtime. The second happens while upgrading — a release from before nodes ran the package answers to its own version number — and it is checked rather than assumed, because an older CLI ignores flags it does not know and the node would otherwise come up as a second server. A node that cannot run as one fails the deploy rather than being started.
⚠ Credentials never go through chat.
/deploytakes a node name and nothing else. The older/deploy <node> <host> <user> <password>form is refused: chat messages are written to the conversation history and the chat log, so a password typed there stays on disk long after the deploy. For the same reason, don't ask an agent to SSH somewhere with a password in the request — the installer reads credentials from the environment and ignores any supplied in a task payload.
Leaving DEPLOY_<NODE>_HOST unset makes the deploy resolve <node>.local over mDNS instead. That is a single name lookup; earlier versions fell back to scanning the local /24 for open SSH ports, which is gone.
Checking on a node, and running a command there
The assistant can check on a configured node for you — "how full is the Pi's disk?", "what did rpi-kitchen log last?" — by asking the installer for one of a fixed set of operations, each of which only reads:
| Operation | What it shows |
|---|---|
uptime |
how long the node has been up, and its load |
disk |
free and used space on each filesystem |
memory |
free and used memory |
temperature |
each thermal zone's temperature |
os |
the operating system and kernel |
network |
each network interface and its addresses |
python |
the Python the node's runner uses |
packages |
the Python packages installed for the runner |
node-log |
the last lines the runner logged (1–200, default 50), from the journal or its log file |
It cannot send a node a command of its own. The assistant reads mail, web pages and device names, and a command it wrote would be only as trustworthy as everything it read; a named operation runs its fixed command whatever the request says. A command of your own you type yourself:
/nodes run rpi-bedroom df -h /home
/nodes run rpi-bedroom sudo apt-get update
It runs over SSH with the node's deploy-target credentials, as the deploy does, and its output is the reply. A slash command is read from what you typed, never from a reply the assistant wrote, and the social channels refuse them. The whole command is logged, at warning level. Only main may ask the installer for anything that reaches a node: an agent running beside it is refused.
The operations are defined in wactorz.core.node_operations, which nodes install too.
The broker has to be reachable from the node
A remote node connects back to the MQTT broker over the network, so broker in
its target block is the address the node should dial — your main machine's
LAN IP, not localhost.
The Home Assistant add-on's embedded broker publishes no port by default, so
nothing outside the add-on can reach it: assign 1883 a host port under the
add-on's Network settings, or 8883 for TLS, before a remote node can
connect to it at all — whatever credentials that node holds.
Broker credentials travel with the deploy. They are written to ~/wactorz/.env
on the node (mode 0600) and sourced when the runner starts, so they appear in
no command line — SSH runs the launch command through a shell whose own
arguments any local user can read with ps, which is why they are not passed
that way. A node uses DEPLOY_<NODE>_BROKER_USER / _BROKER_PASSWORD if set,
and this server's MQTT_USERNAME / MQTT_PASSWORD otherwise.
Sharing the server's account has a cost worth stating: a stolen node holds full broker access, and the broker carries the code spawned agents run. An account per node is the answer, and Wactorz can issue them.
An account per node
WACTORZ_NODE_ACCOUNTS=1 gives every deployed node an account named after it.
The compose files turn it on for their own broker, and the add-ons for their
embedded one; WACTORZ_NODE_ACCOUNTS=0 in .env keeps compose's nodes on the
shared account. A node already deployed keeps the shared account until its next
/deploy, so change MQTT_PASSWORD once every node has been deployed again.
The password is derived from the same secret the signing keys come from, so
nothing new is stored and a leaked password file says nothing about a signing
key, and /deploy writes it into the node's ~/wactorz/.env as before. A
DEPLOY_<NODE>_BROKER_USER / _BROKER_PASSWORD you set still wins.
For the broker Wactorz configures — compose's, and the add-on's embedded one —
it also writes an access list into MQTT_BROKER_DIR, beside the TLS
certificate. The list names what a node may use, and a topic it does not name is
refused. A node may:
- publish and read its own
nodes/<name>/...; - publish and read the agent traffic every host shares,
agents/#— exceptagents/+/commands, which stops agents on the server; - write, and not read,
main/llm_request,main/history_requestandmain/reply/#: it asks main's model, reads main's history and answers main's tasks, without seeing what another node asks or answers; - publish and read the data topics agents use by convention —
custom/#,sensors/#,home/#,schedule/#— and readhomeassistant/state_changes/#.
Everything else is closed to it: another node's nodes/<other>/... in either
direction, so it can neither drive, impersonate nor watch another node;
system/; and whatever else shares the broker, such as zigbee2mqtt/ or Home
Assistant's discovery topics.
agents/# being open means a node can publish under any agent's topics. What
main takes from there as an agent's word -- the reply to a task, its manifest,
the data samples it records for the planner -- it takes only signed by the node
it places that agent on (see "What a node publishes is signed" below). Main
does not take an
agent's withdrawal, the empty manifest that says it is gone, from that alone:
one for an agent still running on main is ignored, and one for an agent on a
node waits for that node's next heartbeat, which only the node can publish,
and is taken only if the heartbeat no longer lists the agent. A node that has
gone quiet counts too: the withdrawal waits until it speaks again, or until
main gives up on it after its grace period, which removes its agents anyway.
Only for a node main does not know at all, such as one that was down when main
started, is a withdrawal taken at once. What else a node can publish under
another agent's topics, such as its status or logs, is taken as given.
That includes tasks. A task on agents/by-name/<name>/task reaches the agent of
that name on whichever node runs it, and any node may publish one, so an agent
on a node is asked by whoever publishes there, another node included. Its
handle_task should treat the payload as untrusted input, as it would text from
the web: check what it acts on, and never run or install what a task names.
Two settings shape the list, both read by whatever writes it (the server, and
compose's mqtt-certs; the add-ons offer them as node_topics and
broker_accounts):
| Setting | What it does |
|---|---|
WACTORZ_NODE_TOPICS |
More data topics for every node, comma-separated: zigbee2mqtt/#, read:weather/#. A filter prefixed read: may be read and not written. One under nodes/, agents/, main/, system/ or $ is refused, with a message, because it would open for every node what the list exists to close. |
WACTORZ_BROKER_ACCOUNTS |
Other accounts on this broker that keep all of it, comma-separated: homeassistant, zigbee2mqtt. The server's own account (MQTT_USERNAME) is always one of them. |
An agent on a node that uses a topic outside the list gets no error. The
broker drops the publish and delivers nothing to the subscription, which is how
MQTT refuses, so the agent looks idle rather than broken. If an agent that works
on the server goes quiet on a node, add its topic prefix to
WACTORZ_NODE_TOPICS.
List every other account once a node is deployed. Mosquitto gives an account
the list does not mention no access at all, so with a node deployed, Home
Assistant's or zigbee2mqtt's account on this broker stops working until it is in
WACTORZ_BROKER_ACCOUNTS. The server names the accounts that keep the whole
broker in its log whenever it writes the list. With no node deployed nothing is
closed: the list gives every account the whole broker.
The compose broker picks up a new node's account on its own: it watches that folder, reloads for a new account, and restarts itself if the access list or the certificate appeared for the first time, since mosquitto reads those only at startup.
Before any node is deployed, two lines in the broker log are expected:
Warning: ACL pattern '#' does not contain '%c' or '%u'.
Warning: ACL pattern '$SYS/#' does not contain '%c' or '%u'.
They are the two lines that give every account the whole broker while there is
no node to keep out of anything. Mosquitto notes that neither names a client,
which is the point — "everything" is not something %u can spell — and the
alternative spelling (topic instead of pattern) applies to anonymous clients
only, which would leave every named account with no access at all.
Only turn this on where the broker has those accounts. On a broker you run
yourself, create them there first (or keep using DEPLOY_<NODE>_BROKER_USER).
A node presenting an account its broker has never heard of is simply refused.
The rollout is per node. Every node keeps the shared account until its next
/deploy, and nothing changes for it until then — so an install with the option
on and nodes not yet redeployed is exactly as contained as it was. Once the last
node has been redeployed, change MQTT_PASSWORD (and the broker's own account)
so the account they used to share no longer opens anything.
What it does not do: a node's own agents share the node's account, so the
containment boundary is the machine, not the agent. Nothing stops an agent
publishing readings another agent reads — that traffic is a commons by design.
The access list names the nodes you have configured, so it covers every node that
exists; a node can still write under a nodes/<name>/... that no node has yet, and
leave a message waiting for one deployed later. Mosquitto cannot express "every
node's topics except your own" — a deny beats every allow, including the node's
own — so naming them is the only shape available. Two things close the rest:
/deploy clears whatever is retained on that node's control topics before the node
starts and subscribes, and WACTORZ_NODE_SIGNING=enforce makes a node refuse
anything main did not sign for it.
Signed commands
Everything main tells a node to do — spawn an agent, apply a desired state, stop,
restart, migrate — is signed with a key derived for that node. /deploy writes the
key to the node's ~/wactorz/.env with its broker credentials, so it travels over
SSH and never over the broker. The signature rides in the message's MQTT v5 user
properties, so the payload is unchanged.
What a node does with a command that is not signed for it is set by
WACTORZ_NODE_SIGNING on the server, and written to the node when it is deployed:
enforce(the default) refuses it. Main signs everything it sends a node, so an unsigned command came from something else with access to the broker.warnacts on it and counts it. The node reports the count in its heartbeat, and main says in chat when it goes up — the way to find out what sends unsigned commands on an install before refusing them.
A node takes the mode at its next /deploy: one deployed while the default was
warn keeps warning until then.
A node deployed before signing holds no key and acts on everything, as it always did; deploy it again to give it one. A node remembers the commands it has accepted, so a captured one cannot be replayed, and refuses one published before its latest deploy. Once a node reports that it checks, main republishes its desired state signed, replacing one retained from before that the node would otherwise refuse on every reboot.
The keys are derived from a secret in <WACTORZ_STATE_DIR>/node_signing.key. Back
it up with the rest of the state directory. To rotate the keys, delete it and deploy
every node again: until a node is redeployed, it reports or refuses what the new
secret signs.
What a node publishes is signed
The other direction uses the same key. A node signs everything it publishes, and main checks the signature against the node it expects to hear from: for a node's heartbeat and reports, the node whose topic it is; for an agent's task reply, manifest or data samples, the node main's own records place the agent on, never the one a heartbeat claims. An agent on main is spoken for by no node: main reads its manifest from the agent itself. An agent main has no record of, one a node started by itself, belongs to the node that first speaks for it.
So code running on one node, the sandbox included, can speak only for that node's agents. It cannot answer a task sent to another node's agent before that agent does, since main waits for the reply its own node signed. And it cannot describe another agent to the planner. A signature names the machine, not the agent on it: an agent can sign as any other agent on its own node.
A message main cannot check follows WACTORZ_NODE_SIGNING, as a command does:
enforce(the default) refuses it. A node deployed before nodes signed what they publish still counts as up, so its agents are not removed, but main believes nothing else its heartbeats say, sends it no tasks, refuses to spawn or move an agent there, and says in chat that it needs/deploy <node>. The dashboard shows the node's agents as main knows them. After upgrading main, deploy every node again.warntakes it, and says so in chat and the log.
Each heartbeat is taken once, so one captured and delivered again does not make a node that has gone away look up.
Encrypted connections (TLS)
A node's broker connection carries its account's password and the code of every agent spawned on it, so it should be encrypted on any network you do not fully trust. Wactorz sets that up with no certificate to buy or renew:
- The broker gets a certificate. Wactorz keeps a private certificate authority
(CA) in
<WACTORZ_STATE_DIR>/mqtt_tls/, created the first time it is needed, and issues the broker's certificate from it. The compose broker serves TLS on port8883beside plain MQTT on1883once the certificate is ininfra/mosquitto/generated/: thepythonandfullprofiles put it there, and so does awactorzrun on the host withMQTT_TLS=1, ormake mqtt-certs— restart the broker after the first time. The Home Assistant add-on's embedded broker serves TLS too. The certificate is issued again before it expires, or when it no longer names the broker's addresses; the CA stays, so nodes keep trusting it. /deploychecks before it switches. It copies the CA to the node and tries a TLS connection to the broker on8883from the node itself. If that works, the node uses TLS from then on; if not, it stays on plain MQTT, and the deploy log says why. A node deployed before this keeps plain MQTT until it is deployed again.
DEPLOY_<NODE>_BROKER_TLS=on uses TLS even when the check fails — the broker may
not be up yet — off never does, and DEPLOY_<NODE>_BROKER_TLS_PORT changes the
port. The node has to be able to reach that port.
A client trusting the generated CA checks that the broker's certificate came from it, but not the host name inside it: that CA signs nothing but this broker, and a node should not lose its connection because the broker's LAN address changed.
A certificate of your own. Set MQTT_TLS_CA on the server to the CA that signed
your broker's certificate, or to system for one from a public CA such as Let's
Encrypt. /deploy hands that to the node instead, and the host name is then
checked, so give each target a broker name the certificate carries.
MQTT_TLS_CHECK_HOSTNAME=1 or 0 decides the host name check either way.
The server's own connection uses TLS with MQTT_TLS=1: the server then dials
MQTT_TLS_PORT (default 8883) instead of MQTT_PORT, under compose too. It creates
the generated CA and certificate when it starts if they are missing, and writes the
broker's copy to MQTT_BROKER_DIR (infra/mosquitto/generated in .env.template). A CA
that cannot be loaded stops it at startup, naming the file it looked for. Without
MQTT_TLS the connection stays plain, which suits a broker beside the server rather
than across a network.
A node started by hand reads the same settings from its environment:
MQTT_TLS=1, MQTT_TLS_CA naming a copy of <WACTORZ_STATE_DIR>/mqtt_tls/ca.crt,
and MQTT_TLS_CHECK_HOSTNAME=0, with --port 8883. A runner told to use TLS that
cannot load its CA refuses to start rather than connecting unverified.
A broker you run yourself can serve the generated certificate:
python -m wactorz.broker_certificates --export <dir> writes broker.crt, with the
CA after it, and broker.key there, for its certfile and keyfile; --name adds
an address it is reached by.
Back up mqtt_tls/ with the rest of the state directory. Should ca.key be lost,
delete ca.crt and ca.key and deploy every node again: a new CA is trusted by no
node holding the old one.
Host key verification
SSH host keys are checked on every connection. A machine that has not been connected to before has its key recorded on first contact — the same trust-on-first-use that interactive ssh does — and any later change to that key fails the connection instead of being accepted.
Learned keys live in <WACTORZ_STATE_DIR>/known_hosts, or wherever DEPLOY_KNOWN_HOSTS points. Set DEPLOY_STRICT_HOST_KEYS=1 to turn off first-use learning entirely; every target's key must then already be in the file:
ssh-keyscan 192.168.1.52 >> "$DEPLOY_KNOWN_HOSTS" # after verifying the fingerprint
A node on main's own machine
Main holds every credential the deployment has: the model's API key, Home Assistant's token, the SSH keys for your nodes, the key that signs commands to them. Code a model writes, possibly from an email or a web page it read, is safer run somewhere that holds none of them. A node holds none of them. A node on main's own machine, under a user of its own, gives that code a place to run without a second machine.
On Linux with systemd, set it up once, as root, from the directory main runs in:
sudo .venv/bin/wactorz-node setup-local
It makes a system user, wactorz-node, with its home in /var/lib/wactorz-node.
It makes an SSH key in main's state directory and lets that user accept it from
this machine only. It lets that user restart its own unit and run nothing else
as root. It installs the unit, and it prints the lines to add to main's .env:
DEPLOY_TARGETS=sandbox # with your other nodes, if you have any
DEPLOY_SANDBOX_HOST=127.0.0.1
DEPLOY_SANDBOX_USER=wactorz-node
DEPLOY_SANDBOX_KEY=/path/to/state/sandbox_node_ed25519
Restart main, then /deploy sandbox. The node is called sandbox because
local and main already mean main itself: /migrate <agent> local moves an
agent to main, and the dashboard lists main's agents under local. No node can
be deployed under either name. From there it is a node like any other:
/deploy sandbox again upgrades it, and the packages its agents install reach it
the way they reach every node. SSH has to be running on the machine, and
AllowUsers in sshd_config, if you use it, has to include wactorz-node.
The unit is sandboxed, which the units on other nodes are not: only
~/wactorz/state is writable, the venv it runs from is read-only, the homes
and main's .env and state directory are hidden, and it has no devices, no
capabilities and no way to gain privilege. A deploy restarts this unit and
never rewrites it, so the sandbox can be changed only by running
setup-local again. An agent that needs a camera or GPIO belongs on a node
with the device.
The node's user must not be able to read main's secrets. Inside the unit they
are hidden, but a package install runs outside it, as that user. So setup
stops while main's .env or state directory is readable by wactorz-node.
Run it again with --fix-permissions to make them readable by main's user
alone. --main-user, --env-file and --state-dir name main's user and
files, where sudo and the current directory do not.
The node's log is in the system journal, which its user cannot read, so a
deploy that fails cannot bring back the log's last lines. Read them with
journalctl -u wactorz-node.
In the compose stack the node is a container instead: see
the sandbox node. It needs no DEPLOY_TARGETS
entry, since nothing reaches it over SSH: it gets a broker account, as the node
WACTORZ_MODEL_CODE_NODE names does.
Sending model-written code there
Once there is a node named sandbox — a deploy target, or a node main hears
from — a <spawn> whose code the model wrote, naming no node, runs there
rather than in main's process. The chat says so: "Spawned 'scraper' on node
'sandbox'". WACTORZ_MODEL_CODE_NODE names another node instead, and main
keeps the code on main.
The model's agents already running on main move to the sandbox the first time
it reports after a start, as /migrate would move them, state and all, and the
chat lists them. One stays on main, named in the chat with the reason, when it
reads the history without await (agent.query_ts(...) rather than
await agent.query_ts(...)), which only main answers that way, or when its move
is refused. The agents move one after another, each as /migrate would move
it, so with many it takes a while. A move clears the agent's packages as
/migrate does: an agent with a package not on the list asks you about it on
the dashboard, and one whose question goes unanswered stays on main, which the
chat says. An
agent that runs main's own classes is not moved: an LLM agent, a rule, a
schedule, a catalogue or library agent. Nor is an agent restored from main's
records, or one you move with /migrate. The agents the planner spawns for a
plan go the same way: one sent to the node is waited for until the node reports
it running, and the plan's steps reach it through main.
A spawn the model wrote that names main is asking for main's credentials, and
WACTORZ_MODEL_CODE_ON_MAIN decides what happens to it. With ask, the default,
the dashboard asks you, with Run on main and Refuse, and only the
dashboard's button counts, never an answer on the broker. Refused, or not
answered within two minutes, it is not spawned, and for ten minutes the same
spawn is refused without asking again. allow lets it run on main without
asking, and never refuses it.
If the node named is neither one of DEPLOY_TARGETS nor a node main has
heard from, such a spawn is refused, never quietly run on main. If it is, but the node is not running, the agent is
sent all the same and starts when the node does; the chat says so.
MQTT topics
The runner subscribes to a set of control topics scoped to its node name, and publishes heartbeats for itself and all its agents.
| Topic | Direction | Description |
|---|---|---|
nodes/{name}/spawn |
→ runner | Spawn a new agent. Payload: full agent config dict. Not retained — a node that was away catches up from desired_state. Signed. |
nodes/{name}/desired_state |
→ runner | Every agent the node should be running. Retained; the runner starts any that are missing when it connects, and stops (keeping its state) any it runs that the list leaves out, except one a migration placed there and main has not confirmed yet. Signed. |
nodes/{name}/stop |
→ runner | Stop a named agent. Payload: {"name": "agent-name"}; "delete": true also removes its state file and clears its retained topics, and "keep_topics": true keeps those for an agent that lives on elsewhere under the same id, as a rolled-back migration does. Signed. |
nodes/{name}/stop_all |
→ runner | Stop all agents and shut down the runner. Signed. |
nodes/{name}/list |
→ runner | Request the list of running agents. Response on nodes/{name}/agents. |
nodes/{name}/agents |
← runner | Response to list. Contains agent names and actor IDs. |
nodes/{name}/heartbeat |
← runner | Runner heartbeat every 10 s. Contains node name, Wactorz version, runtime kind, agent count, broker address, whether the node checks signed commands, and how close the machine is to running out: CPU, memory (within a container's or a unit's memory limit), swap, load, free disk where the state is kept, the CPU's temperature and, on a Raspberry Pi, what is throttling it. A reading the node cannot take is null. |
nodes/{name}/manifest |
← runner | What the machine is, retained: architecture, system, Python, model, whether it is a container, CPUs and memory (within any limit), disk, accelerators, the devices an agent can use, the installed packages, the library agents it can build (agents, by import path), and whether it clears its agents' copies on a reset (resets). Sent when the runner starts and after an install; cleared by /nodes remove. |
nodes/{name}/migrate |
→ runner | Hand a running agent back to main, which places it. Payload: {"name": "...", "target_node": "@main", "return_token": "...", "force": false, "max_state_bytes": 8388608}. Signed. |
nodes/{name}/migrate_result |
← runner | Result of a migration request. A failure, including a refusal, is shown on the dashboard. |
nodes/{name}/code_changed |
← runner | An agent here repaired its own program. Carries the agent's name and no code. |
nodes/{name}/code_request |
→ runner | Asks for the program an agent is actually running. Payload: {"agent": "...", "token": "..."}. Signed. |
nodes/{name}/code_return |
← runner | The program, quoting the token it was asked with. |
nodes/{name}/reset |
→ runner | Clear one agent's chat, state or counters here, or every agent's, as main has cleared its own records. Payload: {"scope": "chat" \| "state" \| "metrics", "agent": "..." or null, "request": "..."}. Signed. |
nodes/{name}/reset_done |
← runner | The reset done, quoting request and naming the agents it cleared ("cleared": [...]); a scope the node does not reach into is answered with none. |
nodes/{name}/reply/{id} |
← runner | Reply routing for agent.send_to() calls originating on this node. |
agents/{id}/heartbeat |
← agent | Per-agent heartbeat every 10 s. Includes "node": "{name}" field. |
agents/{id}/logs |
← agent | Log messages from agent.log() and agent.alert(). |
agents/by-name/{name}/task |
→ agent | Task addressed to a named agent on any node. Runner routes to local agents by name. Any node may publish one, so its payload is untrusted input. |
Why a repair travels in two steps
An agent can be repaired by the LLM while it runs, and main's spawn registry has to learn about it or the next reboot sends the node the program that failed.
The node never volunteers the code. It publishes a notice that something changed; main then asks for the program on a signed topic, quoting a token it minted, and files the answer only for that token, that agent and that node, spending the token on use.
The asymmetry is the point. Main executes what a node sends it, so a node that could publish code straight into the registry could put it there for any agent it hosts — and that code would run wherever the agent went next, including on main. It is the same reason node-to-node migration is routed through main, and why an agent leaving a node is placed from main's copy of its program rather than the one the node sends back. A forged notice buys nothing: main asks the node, and the node answers with what it is really running.
A program filed this way is no longer trusted, even for a catalogue agent: a repair is model-written, and the node is not trusted to say what it wrote. It meets the safety scanner wherever the agent is spawned next, main included.
Two limits keep a notice from being worth sending. Main asks about one agent at a time and no more often than every ten seconds, because each question is a signed message and signing records a sequence number to disk — a rate meant for an operator's commands, not for whatever a node chooses to announce. And an answer that quotes a valid token but names the wrong agent or arrives from the wrong node is dropped without spending the question, so someone who reads a token off the broker cannot use it to cancel the exchange and leave the real answer refused.
Supervisor behaviour
Each agent on a remote node runs under a local ONE_FOR_ONE supervisor — identical semantics to the main machine. If an agent crashes, the supervisor restarts it with exponential back-off:
delay = min(restart_delay * (2 ** (crashes_in_a_row - 1)), 60.0)
# restart_delay=3.0: 3s → 6s → 12s → 24s → 48s → 60s (cap)
An agent that stays up for restart_window (60 s) after a restart has recovered: its next
crash starts from restart_delay again.
| Scenario | Behaviour |
|---|---|
Crash in process() |
Back-off, restart. After 5 consecutive failures in one run, escalates to supervisor for a clean restart. |
Crash in setup() |
Fatal — supervisor stops. Broken code won't fix itself on retry. |
| Compile error | Fatal — supervisor stops immediately. |
max_restarts crashes in a row |
Restarts slow down: 5 minutes, then doubling up to an hour. Main says so in chat, once, and again when the agent recovers. It is never given up on automatically — delete it to stop it. |
Deliberate stop command |
No restart — clean shutdown. |
Default values: max_restarts=5, restart_delay=3.0. Override per agent in the spawn config.
Agent API on the edge
The agent object available inside remote agent code mirrors the local DynamicAgent API. All of the following work identically:
| Method | Description |
|---|---|
await agent.publish(topic, data) |
Publish to any MQTT topic via the shared broker. |
await agent.log(message) |
Log to agents/{id}/logs — visible in the central dashboard. |
await agent.alert(message, severity) |
Publish an alert. Levels: info, warning, error. |
agent.persist(key, value) |
Write to ~/wactorz/state/<name>_state.json (JSON, not pickle — portable, and what a migration carries). Bytes, numpy arrays, torch tensors and modules, and scikit-learn-family models are kept as files of their own beside it. A datetime, date, time or timedelta, a set or frozenset, a tuple, and a dict whose keys are not strings are written in a marked form and come back as what they were. Any other value that cannot be written as JSON is not kept: it is dropped with a warning naming it, and the rest is saved. |
agent.recall(key) |
Read a persisted value. |
agent.state |
In-memory dict, not persisted. |
await agent.send_to(name, payload) |
Send a task to any agent (local or remote) via MQTT request/reply. Times out after 30 s by default. |
agent.node |
The node name this agent is running on (e.g. "rpi-livingroom"). |
agent.agents() |
List of all agents running on this node. |
⚠ No
agent.subscribe()on edge — The remote runner does not implementagent.subscribe(). For MQTT subscriptions in remote agents, open anaiomqtt.Clientdirectly insidesetup()— the broker address is available as the machine's IP passed to--broker.
Agent migration
A running agent can be moved between main and a node, or from one node to another, from the chat:
/migrate temp-sensor rpi-bedroom # to a node
/migrate temp-sensor local # back to main
/migrate temp-sensor rpi-bedroom --force
Main routes every migration; a node never spawns on another node.
An agent leaving a node, for main or for another node, is placed from main's own copy of its program: the node hands back its state, and the program it sends with it — generated code, or a library agent's bundle — is compared and discarded, never run. A move on to another node sends the bundle main builds from its own copy of the agent's code, and is checked before the source stops the agent, as a move from main is. A program the node repaired reaches main's copy through the exchange in Why a repair travels in two steps; one main has not fetched yet is lost, and the announcement of the move says the node's program was discarded. An agent main holds no program for — one in no spawn registry entry, and not a library agent installed on main, though a node runs it — cannot leave its node, and /migrate says so before anything stops.
A node runs an agent whose program goes with it: generated code, or an LLM agent (type: llm), for which main writes the code. A library agent — a decorated function or an Actor subclass registered with the system, type: module — it builds from the package installed there, so one moves only to a node whose manifest names its target; see A library agent. An agent built into the server — a native catalogue agent such as flic or weather-agent, a Home Assistant actuator, a scheduled or rule agent — has no program in its config, so moving or spawning one on a node is refused with the reason, and it stays where it is. The agent is stopped where it runs, its persisted state is sent to where it is going with main's copy of its config, and the source keeps its own copy until the destination confirms the agent started. Only then is the source's copy deleted. A destination that never confirms within five minutes is told to drop the agent, and the agent is started again where it was, from the copy that was kept.
The state is taken after the agent has stopped, so what its on_stop writes last (a final counter, an LLM agent's last turn) goes with it.
State travels as JSON — a date, a set, a tuple or a dict keyed by numbers in the marked form that brings it back as it was — and the agent's blobs — bytes, numpy arrays, torch tensors and modules, scikit-learn-family models — travel beside it, each in chunks on a topic of its own, checked against its SHA-256 when it arrives. A migration is refused, and the agent keeps running where it is, when:
- its state holds a value that can go neither way (an open capture, a socket, an object of the agent's own class). The refusal names the keys.
--forcemoves the agent without them, and the announcement when it arrives names what was left behind; - it is on a node and keeps a value that runs code when it loads — a torch module, a scikit-learn, XGBoost, LightGBM or CatBoost model saved with joblib. Main sends these to a node, which runs what main sends it anyway; it does not load them from a node, and one node does not pass them to another. They are refused like the values above, and
--forceleaves them on the node. Keep a model'sstate_dict()instead of the module, and it travels; - its state is larger than
WACTORZ_MIGRATION_MAX_STATE_BYTES(8 MiB by default, set on main), or its blobs together are larger thanWACTORZ_MIGRATION_MAX_BLOB_BYTES(64 MiB by default, read on main and on each node).--forcedoes not change this. The blobs' size is known only once they are encoded, after the agent stops, so an agent over it is started again where it was.
A destination waits for an agent's blobs before starting it, for at least a minute and longer for large ones. One that does not arrive leaves the agent where it was, as a destination that never confirms does.
A node checks its own agents against the same two terms, which main sends with the request. A node running an earlier release does not check them: it sends what it can, and main names the keys it could not send.
Debugging
Verbose diagnostics
Run with --loglevel DEBUG to see every connection event, subscribe and publish:
~/wactorz/venv/bin/wactorz-node --node rpi-test --mqtt-broker 192.168.1.10 --loglevel DEBUG
A node refuses to start, with exit status 2, in two cases it can detect up front: a node name containing an MQTT wildcard, which the broker would refuse on every operation; and MQTT_TLS turned on with a CA that cannot be read. Both would otherwise be a runner reconnecting every three seconds for ever.
Watch node traffic from the main machine
mosquitto_sub -h localhost -t 'nodes/#' -v
mosquitto_sub -h localhost -t 'agents/+/heartbeat' -v
List agents on a node
mosquitto_pub -h localhost -t "nodes/rpi-livingroom/list" -m '{}'
mosquitto_sub -h localhost -t "nodes/rpi-livingroom/agents" -C 1
Stop a specific remote agent
mosquitto_pub -h localhost -t "nodes/rpi-livingroom/stop" \
-m '{"name": "temp-sensor-agent"}'
Stop the entire runner
mosquitto_pub -h localhost -t "nodes/rpi-livingroom/stop_all" -m '{}'