You typed a Python script, clicked Dispatch, and minutes later it ran on a machine somewhere else. Here's every step of what just happened.
Imagine you just dispatched a job from the dashboard. Before you even finish your coffee, the script has traveled through five distinct systems. Click Next Step to trace the path.
Dashboard
Puppeteer
PostgreSQL
Polls & picks up
Runs script
Here's something clever: the Puppeteer server never pushes work to nodes. The nodes poll (repeatedly ask) for work. This is called the pull model, and it's a deliberate choice.
Since nodes initiate all connections, there's no need to poke holes in firewalls or expose ports. A node on your home laptop can talk to a Puppeteer server in the cloud.
If a node disconnects, no work is lost — jobs just wait in the database. When the node comes back, it picks up from where it left off.
Nodes identify themselves by a certificate — a cryptographic identity card. IP addresses can change; certificates stay the same.
The backend framework that handles every API request — from dispatching jobs to enrolling new nodes.
The frontend framework for the dashboard — the window you use to manage everything.
The database that stores everything: jobs, nodes, users, audit logs. SQLite is used locally for development.
Two cryptographic systems: one for verifying node identity, one for verifying that scripts haven't been tampered with. More on this in Module 5.
Three separate programs, each with a distinct role — like a three-act play where every character has exactly one job.
Axiom is not one program — it's three that work together. Understanding who's who is the foundation for everything else.
The FastAPI server. Owns all state: jobs, nodes, users, certificates. The brain of the operation. Lives in puppeteer/agent_service/
The React app. Renders job queues, node status, charts, and forms. Talks exclusively to the Puppeteer API via authenticated fetch calls. Lives in puppeteer/dashboard/
The Python agent that runs on any machine you want to control. Polls for jobs, verifies signatures, executes scripts in containers, reports results. Lives in puppets/environment_service/
Splitting a system into components that each have one job is one of the most important patterns in software engineering. When something breaks, you immediately know which component to look in — and you can change one without breaking the others.
You don't need to memorize every file. But knowing where things live lets you tell your AI coding assistant exactly what to change.
When you log in, the server creates a JWT (a digital ticket) and sends it to your browser. Every subsequent request includes that ticket. Here's the code that creates it:
def create_access_token(data: dict, expires_delta=None):
to_encode = data.copy()
if expires_delta:
expire = datetime.now(UTC) + expires_delta
else:
expire = datetime.now(UTC) + timedelta(minutes=15)
to_encode.update({"exp": expire})
encoded_jwt = jwt.encode(
to_encode, SECRET_KEY, algorithm=ALGORITHM
)
return encoded_jwt
Take a dictionary of user info (like username and role) and make a copy of it.
Add an expiry time — this ticket becomes invalid after 24 hours, so stolen tokens don't work forever.
Cryptographically sign the whole bundle using a secret key only the server knows. This signature means: "I, the server, verified this and vouch for it."
Return the signed ticket — a compact string the browser stores and sends back with every request.
The server can verify any JWT just by checking the signature — without looking it up in the database. That's much faster than a database lookup on every request. The trade-off: you can't revoke a token until it expires (unless you increment the user's token_version field).
Before a node can receive any work, it must prove it belongs to your network. Here's how a stranger on the internet earns trust — automatically.
Imagine you get a phone call from someone claiming to be from your bank. How do you know they're really from the bank? You need a shared secret — something only the real bank would know.
Same problem here. When a new machine connects and says "I'm a puppet node — give me work," how does the server know it's really yours and not a random intruder?
You could use a shared password — every node gets the same API key. But if one node is compromised, every node is compromised. Axiom uses something much stronger: unique certificates per node.
The Puppeteer server runs its own Certificate Authority — think of it as a private passport office that only you control. Every node gets its own certificate. The server only trusts nodes with valid certificates it issued.
A new node gets a JOIN_TOKEN — a string you paste into the node's environment. It looks like random characters, but it's actually a Base64-encoded JSON document containing two things: the server's Root CA certificate, and a one-time enrollment token.
decoded_bytes = base64.b64decode(top_token)
decoded_str = decoded_bytes.decode('utf-8')
payload = json.loads(decoded_str, strict=False)
if "t" in payload and "ca" in payload:
ca_content = payload["ca"]
with open("secrets/root_ca.crt", "w") as f:
f.write(ca_content)
self.join_token = payload["t"]
print(f"✅ Trust Bootstrapped")
Decode the JOIN_TOKEN from text back into raw bytes, then parse it as a JSON document.
Check if this is an "enhanced token" containing both a CA certificate and an enrollment token.
Save the CA certificate to disk. This is the server's identity card — from now on, the node will only trust the Puppeteer server if it presents a certificate signed by this CA.
Extract the real enrollment token. The node is now ready to register.
The node generates its own private key and creates a CSR — a formal request saying "I am node-abc1234, please issue me a certificate."
The node sends its CSR to the server along with the enrollment token (the "hiring letter" from the JOIN_TOKEN). The server verifies the token is valid and unused.
The Puppeteer's Certificate Authority signs the node's CSR, creating a unique client certificate. This cert says: "I, the Puppeteer CA, vouch that this is node-abc1234." The cert is sent back to the node.
From now on, every connection the node makes presents its certificate. The server verifies it was signed by the CA — and the node verifies the server's certificate too. Both sides prove identity. This mutual verification is called mTLS.
A stolen certificate can be revoked — the server adds it to a CRL (Certificate Revocation List) and immediately stops trusting it. No password resets needed. The node simply can't connect anymore.
Like a relay race, a job passes through multiple hands before it's done. Each handoff has a specific protocol — and a security check.
Nodes execute code that arrives over the network. If anyone could dispatch a job, an attacker could run anything on your machines. So the system requires every script to be signed with a private key whose public counterpart is registered in the system.
Here's the verification code that runs on the node before any script executes:
with open(self.verify_key_path, "rb") as f:
public_key_bytes = f.read()
public_key = serialization.load_pem_public_key(
public_key_bytes
)
sig_bytes = base64.b64decode(signature)
public_key.verify(sig_bytes, script.encode('utf-8'))
print(f"✅ Signature Verified for Job {guid}")
Load the public verification key from disk. This key was fetched from the server during bootstrap — the server controls which keys are trusted.
Decode the signature that arrived with the job from text back into raw bytes.
The critical check: use the public key to verify that the signature matches the script. If even one character of the script was changed after signing, this fails. If the wrong key signed it, this fails.
Only if this passes does execution continue. Otherwise, the job is immediately rejected with a security error.
Nodes poll the server every few seconds asking "any work for me?" Here's the code that does it, using the mTLS certificate for authentication:
async with httpx.AsyncClient(
verify=VERIFY_SSL,
cert=(self.cert_file, self.key_file)
) as client:
headers = {
"X-Node-ID": self.node_id,
"X-Node-Secret-Hash": secret_hash,
}
resp = await client.post(
f"{AGENT_URL}/work/pull",
headers=headers,
timeout=10.0
)
if resp.status_code == 200:
data = resp.json()
if data: return data
Open a network connection using the node's certificate and private key. This is the mTLS handshake — the server sees this cert and knows exactly which node is connecting.
Include the Node ID and a secret hash in the request headers as extra identity confirmation.
POST to /work/pull — the server's job-dispensing endpoint. It checks if there's any queued work that matches this node's capabilities.
If the server sends back a job (status 200 with data), return it so the node can start executing.
Here's the complete exchange between all the components when you dispatch a job. Click Next Message to see each step.
This system has three completely different security mechanisms — each solving a different problem. Using all three is why it's robust.
Think of a high-security venue with three different checkpoints — not because they're paranoid, but because each checkpoint catches a different kind of threat.
Problem: Are you a logged-in user with permission to do this?
Solution: A signed ticket you present on every browser request. Expires after 24 hours.
Problem: Is this a trusted node that the Puppeteer itself enrolled?
Solution: Mutual certificate verification — both sides prove identity before any data is exchanged.
Problem: Was this script approved by someone with a registered signing key?
Solution: Cryptographic signatures on every script. Unsigned scripts are always rejected.
Each layer catches different attacks. An attacker who steals a JWT still can't enroll rogue nodes (no mTLS cert). A compromised node still can't run unauthorized scripts (no signing key). Breaking one layer doesn't break the system — you'd need to break all three simultaneously.
The POST /work/pull endpoint that nodes call is a perfect example. It enforces multiple checks in a single line:
@app.post("/work/pull", response_model=PollResponse)
async def pull_work(
request: Request,
node_id: str = Depends(verify_node_secret),
api_key: str = Depends(verify_api_key),
db: AsyncSession = Depends(get_db)
):
r = await db.execute(
select(Node).where(Node.node_id == node_id)
)
n = r.scalar_one_or_none()
if n and n.status == "REVOKED":
raise HTTPException(status_code=403)
This route requires three things before it will even run: verify_node_secret (checks the mTLS cert + secret hash — proves this is a real enrolled node), verify_api_key (checks the shared API key), and a database session.
FastAPI's Depends() system means: "before calling this function, run these checks. If any fail, return a 401 or 403 error automatically."
Even after passing the automated checks, there's a manual check: if this node was revoked (e.g., stolen hardware), it gets a 403 Forbidden — it cannot receive any work.
Jobs can carry secrets (like API keys) that get injected as environment variables during execution. These are stored encrypted in the database using Fernet encryption.
ENCRYPTION_KEY
Server env var — Fernet key for secrets at rest. Never stored in the DB.
SECRET_KEY
Server env var — JWT signing key. Change this in production.
JOIN_TOKEN
Node env var — contains the Root CA cert + enrollment token. Generated fresh per node.
ADMIN_PASSWORD
Only seeds the admin user on first run — DB password is the source of truth after that.
You've seen the engine room. Here's how to use that knowledge to build better, debug faster, and steer AI more precisely.
Check: Are any nodes online? Do any nodes have the capabilities the job requires? Is the node's concurrency limit full? Look at job_service.py → capability matching logic.
Check: Is the JOIN_TOKEN correct and unmodified? Is AGENT_URL pointing to the right server? Does the secrets/ folder have a leftover .crt from a previous enrollment? Node ID is determined by scanning secrets/ for node-*.crt files.
For browser requests: JWT expired (re-login) or user's token_version incremented (password was changed). For node requests: cert might be revoked, or mTLS not configured correctly.
SQLAlchemy's create_all only creates tables — it won't ALTER existing ones. For existing deployments, run the migration SQL manually or drop+recreate the database.
Three separate programs. mTLS enrollment. JWT auth. Ed25519 script signing. The pull model. Capability matching. You can read the code, describe what each file does, spot where a bug might live, and give precise instructions to an AI coding assistant. That's exactly what this course was for.