01

The 20-Second Mystery

You typed a Python script, clicked Dispatch, and minutes later it ran on a machine somewhere else. Here's every step of what just happened.

A script's journey across the network

Imagine you just dispatched a job from the dashboard. Before you even finish your coffee, the script has traveled through five distinct systems. Click Next Step to trace the path.

1
You

Dashboard

2
API

Puppeteer

3
Database

PostgreSQL

4
Node

Polls & picks up

5
Container

Runs script

Click "Next Step" to begin tracing the journey.
Step 0 / 5

Why nodes call home — not the other way around

Here's something clever: the Puppeteer server never pushes work to nodes. The nodes poll (repeatedly ask) for work. This is called the pull model, and it's a deliberate choice.

Nodes can live behind firewalls

Since nodes initiate all connections, there's no need to poke holes in firewalls or expose ports. A node on your home laptop can talk to a Puppeteer server in the cloud.

Nodes can go offline gracefully

If a node disconnects, no work is lost — jobs just wait in the database. When the node comes back, it picks up from where it left off.

The server doesn't need to know node IPs

Nodes identify themselves by a certificate — a cryptographic identity card. IP addresses can change; certificates stay the same.

The four technologies that power this

FastAPI

The backend framework that handles every API request — from dispatching jobs to enrolling new nodes.

⚛️

React

The frontend framework for the dashboard — the window you use to manage everything.

🗄️

PostgreSQL

The database that stores everything: jobs, nodes, users, audit logs. SQLite is used locally for development.

🔒

mTLS + Ed25519

Two cryptographic systems: one for verifying node identity, one for verifying that scripts haven't been tampered with. More on this in Module 5.

02

Meet the Cast

Three separate programs, each with a distinct role — like a three-act play where every character has exactly one job.

Three programs, one system

Axiom is not one program — it's three that work together. Understanding who's who is the foundation for everything else.

🖥️
Puppeteer — the control plane

The FastAPI server. Owns all state: jobs, nodes, users, certificates. The brain of the operation. Lives in puppeteer/agent_service/

📊
Dashboard — your window in

The React app. Renders job queues, node status, charts, and forms. Talks exclusively to the Puppeteer API via authenticated fetch calls. Lives in puppeteer/dashboard/

🤖
Puppet Node — the worker

The Python agent that runs on any machine you want to control. Polls for jobs, verifies signatures, executes scripts in containers, reports results. Lives in puppets/environment_service/

💡
Separation of concerns

Splitting a system into components that each have one job is one of the most important patterns in software engineering. When something breaks, you immediately know which component to look in — and you can change one without breaking the others.

The files that matter most

You don't need to memorize every file. But knowing where things live lets you tell your AI coding assistant exactly what to change.

📂 puppeteer/agent_service/
main.pyEvery API route — the entry point for all requests
db.pyDatabase table definitions (Job, Node, User, etc.)
auth.pyJWT creation and verification — handles logins
pki.pyCertificate Authority — issues certs to new nodes
services/
job_service.pyJob assignment, node selection, capability matching
scheduler_service.pyCron-based scheduled job definitions
foundry_service.pyBuilds custom Docker images for nodes
📂 puppets/environment_service/
node.pyThe entire puppet agent — enrolls, polls, executes, reports
runtime.pyRuns scripts inside Docker or Podman containers

How the server proves you're logged in

When you log in, the server creates a JWT (a digital ticket) and sends it to your browser. Every subsequent request includes that ticket. Here's the code that creates it:

auth.py
def create_access_token(data: dict, expires_delta=None):
    to_encode = data.copy()
    if expires_delta:
        expire = datetime.now(UTC) + expires_delta
    else:
        expire = datetime.now(UTC) + timedelta(minutes=15)
    to_encode.update({"exp": expire})
    encoded_jwt = jwt.encode(
        to_encode, SECRET_KEY, algorithm=ALGORITHM
    )
    return encoded_jwt
Plain English

Take a dictionary of user info (like username and role) and make a copy of it.

Add an expiry time — this ticket becomes invalid after 24 hours, so stolen tokens don't work forever.

Cryptographically sign the whole bundle using a secret key only the server knows. This signature means: "I, the server, verified this and vouch for it."

Return the signed ticket — a compact string the browser stores and sends back with every request.

💡
Why signing matters

The server can verify any JWT just by checking the signature — without looking it up in the database. That's much faster than a database lookup on every request. The trade-off: you can't revoke a token until it expires (unless you increment the user's token_version field).

03

The Secret Handshake

Before a node can receive any work, it must prove it belongs to your network. Here's how a stranger on the internet earns trust — automatically.

The trust problem

Imagine you get a phone call from someone claiming to be from your bank. How do you know they're really from the bank? You need a shared secret — something only the real bank would know.

Same problem here. When a new machine connects and says "I'm a puppet node — give me work," how does the server know it's really yours and not a random intruder?

⚠️
The naive solution (don't do this)

You could use a shared password — every node gets the same API key. But if one node is compromised, every node is compromised. Axiom uses something much stronger: unique certificates per node.

The solution: certificates issued by a private authority

The Puppeteer server runs its own Certificate Authority — think of it as a private passport office that only you control. Every node gets its own certificate. The server only trusts nodes with valid certificates it issued.

The JOIN_TOKEN — a hiring letter encoded as data

A new node gets a JOIN_TOKEN — a string you paste into the node's environment. It looks like random characters, but it's actually a Base64-encoded JSON document containing two things: the server's Root CA certificate, and a one-time enrollment token.

node.py — bootstrap_trust()
decoded_bytes = base64.b64decode(top_token)
decoded_str = decoded_bytes.decode('utf-8')
payload = json.loads(decoded_str, strict=False)

if "t" in payload and "ca" in payload:
    ca_content = payload["ca"]
    with open("secrets/root_ca.crt", "w") as f:
        f.write(ca_content)
    self.join_token = payload["t"]
    print(f"✅ Trust Bootstrapped")
Plain English

Decode the JOIN_TOKEN from text back into raw bytes, then parse it as a JSON document.

Check if this is an "enhanced token" containing both a CA certificate and an enrollment token.

Save the CA certificate to disk. This is the server's identity card — from now on, the node will only trust the Puppeteer server if it presents a certificate signed by this CA.

Extract the real enrollment token. The node is now ready to register.

From stranger to trusted worker in 4 steps

1
Generate a private key and CSR

The node generates its own private key and creates a CSR — a formal request saying "I am node-abc1234, please issue me a certificate."

2
Submit CSR + enrollment token to /api/enroll

The node sends its CSR to the server along with the enrollment token (the "hiring letter" from the JOIN_TOKEN). The server verifies the token is valid and unused.

3
Server signs the certificate

The Puppeteer's Certificate Authority signs the node's CSR, creating a unique client certificate. This cert says: "I, the Puppeteer CA, vouch that this is node-abc1234." The cert is sent back to the node.

4
mTLS active — all future connections are verified

From now on, every connection the node makes presents its certificate. The server verifies it was signed by the CA — and the node verifies the server's certificate too. Both sides prove identity. This mutual verification is called mTLS.

💡
Why this beats passwords

A stolen certificate can be revoked — the server adds it to a CRL (Certificate Revocation List) and immediately stops trusting it. No password resets needed. The node simply can't connect anymore.

04

The Job Relay

Like a relay race, a job passes through multiple hands before it's done. Each handoff has a specific protocol — and a security check.

Why every script needs a signature

Nodes execute code that arrives over the network. If anyone could dispatch a job, an attacker could run anything on your machines. So the system requires every script to be signed with a private key whose public counterpart is registered in the system.

Here's the verification code that runs on the node before any script executes:

node.py — execute_task()
with open(self.verify_key_path, "rb") as f:
    public_key_bytes = f.read()
    public_key = serialization.load_pem_public_key(
        public_key_bytes
    )
sig_bytes = base64.b64decode(signature)
public_key.verify(sig_bytes, script.encode('utf-8'))
print(f"✅ Signature Verified for Job {guid}")
Plain English

Load the public verification key from disk. This key was fetched from the server during bootstrap — the server controls which keys are trusted.

Decode the signature that arrived with the job from text back into raw bytes.

The critical check: use the public key to verify that the signature matches the script. If even one character of the script was changed after signing, this fails. If the wrong key signed it, this fails.

Only if this passes does execution continue. Otherwise, the job is immediately rejected with a security error.

The polling loop — a node's heartbeat

Nodes poll the server every few seconds asking "any work for me?" Here's the code that does it, using the mTLS certificate for authentication:

node.py — poll_for_work()
async with httpx.AsyncClient(
    verify=VERIFY_SSL,
    cert=(self.cert_file, self.key_file)
) as client:
    headers = {
        "X-Node-ID": self.node_id,
        "X-Node-Secret-Hash": secret_hash,
    }
    resp = await client.post(
        f"{AGENT_URL}/work/pull",
        headers=headers,
        timeout=10.0
    )
    if resp.status_code == 200:
        data = resp.json()
        if data: return data
Plain English

Open a network connection using the node's certificate and private key. This is the mTLS handshake — the server sees this cert and knows exactly which node is connecting.

Include the Node ID and a secret hash in the request headers as extra identity confirmation.

POST to /work/pull — the server's job-dispensing endpoint. It checks if there's any queued work that matches this node's capabilities.

If the server sends back a job (status 200 with data), return it so the node can start executing.

The full conversation — watch it happen

Here's the complete exchange between all the components when you dispatch a job. Click Next Message to see each step.

📡 Job Dispatch — Live System Conversation
0 / 9 messages

Test your understanding

Scenario A node keeps getting assigned jobs but they all fail immediately with "Signature Verification Failed." You can confirm the node is online and connected. Where's the problem?
Scenario You want to add a "GPU acceleration" requirement to a job so it only runs on nodes with a GPU. Which parts of the system need to change?
Scenario A script ran successfully on the node (exit code 0) but the Dashboard shows the job as FAILED. Based on the architecture you've learned, where would you look first?
05

Three Locks on One Door

This system has three completely different security mechanisms — each solving a different problem. Using all three is why it's robust.

Three security problems need three different solutions

Think of a high-security venue with three different checkpoints — not because they're paranoid, but because each checkpoint catches a different kind of threat.

🎫

JWT — for humans

Problem: Are you a logged-in user with permission to do this?
Solution: A signed ticket you present on every browser request. Expires after 24 hours.

🪪

mTLS — for machines

Problem: Is this a trusted node that the Puppeteer itself enrolled?
Solution: Mutual certificate verification — both sides prove identity before any data is exchanged.

✍️

Ed25519 — for code

Problem: Was this script approved by someone with a registered signing key?
Solution: Cryptographic signatures on every script. Unsigned scripts are always rejected.

💡
Defence in depth

Each layer catches different attacks. An attacker who steals a JWT still can't enroll rogue nodes (no mTLS cert). A compromised node still can't run unauthorized scripts (no signing key). Breaking one layer doesn't break the system — you'd need to break all three simultaneously.

All three checks in one route

The POST /work/pull endpoint that nodes call is a perfect example. It enforces multiple checks in a single line:

main.py — /work/pull route
@app.post("/work/pull", response_model=PollResponse)
async def pull_work(
    request: Request,
    node_id: str = Depends(verify_node_secret),
    api_key: str = Depends(verify_api_key),
    db: AsyncSession = Depends(get_db)
):
    r = await db.execute(
        select(Node).where(Node.node_id == node_id)
    )
    n = r.scalar_one_or_none()
    if n and n.status == "REVOKED":
        raise HTTPException(status_code=403)
Plain English

This route requires three things before it will even run: verify_node_secret (checks the mTLS cert + secret hash — proves this is a real enrolled node), verify_api_key (checks the shared API key), and a database session.

FastAPI's Depends() system means: "before calling this function, run these checks. If any fail, return a 401 or 403 error automatically."

Even after passing the automated checks, there's a manual check: if this node was revoked (e.g., stolen hardware), it gets a 403 Forbidden — it cannot receive any work.

Secrets stored in the database are encrypted

Jobs can carry secrets (like API keys) that get injected as environment variables during execution. These are stored encrypted in the database using Fernet encryption.

ENCRYPTION_KEY Server env var — Fernet key for secrets at rest. Never stored in the DB.
SECRET_KEY Server env var — JWT signing key. Change this in production.
JOIN_TOKEN Node env var — contains the Root CA cert + enrollment token. Generated fresh per node.
ADMIN_PASSWORD Only seeds the admin user on first run — DB password is the source of truth after that.

Security in practice

Scenario An attacker gains access to a node machine and tries to execute their own malicious script by submitting it directly to /work/pull. What stops them?
Scenario A laptop running a puppet node gets stolen. You need to ensure the attacker can't use it to receive jobs. What do you do, and what happens?
06

What You Know Now

You've seen the engine room. Here's how to use that knowledge to build better, debug faster, and steer AI more precisely.

The full picture — click to explore

🖥️ Your Browser
📊 Dashboard (React)
⚡ Puppeteer Server
🔌 API (main.py)
🧠 Job Service
🔐 PKI / CA (pki.py)
🗄️ Database
🤖 Puppet Nodes
🤖 Node Agent (node.py)
📦 Runtime (runtime.py)
👆 Click any component to learn what it does and where the code lives.

When something breaks — where to look

?
Job stuck as PENDING forever

Check: Are any nodes online? Do any nodes have the capabilities the job requires? Is the node's concurrency limit full? Look at job_service.py → capability matching logic.

?
Node won't connect / crash on startup

Check: Is the JOIN_TOKEN correct and unmodified? Is AGENT_URL pointing to the right server? Does the secrets/ folder have a leftover .crt from a previous enrollment? Node ID is determined by scanning secrets/ for node-*.crt files.

?
401 / 403 on API requests

For browser requests: JWT expired (re-login) or user's token_version incremented (password was changed). For node requests: cert might be revoked, or mTLS not configured correctly.

?
New columns not appearing after code change

SQLAlchemy's create_all only creates tables — it won't ALTER existing ones. For existing deployments, run the migration SQL manually or drop+recreate the database.

Final challenge — put it all together

Build scenario You want to add a "cost centre" field to every job so accounting can track which team ran what. The field needs to be stored and filterable. Which parts of the system definitely need to change?
Debugging scenario A user reports that after changing their password, they were immediately logged out on all devices, including their phone. Is this a bug?
Architecture decision You want to build a "dry run" feature — show what a script would do without actually executing it. Based on the architecture, which component should be responsible for the dry run logic?
🎓
You know this system now

Three separate programs. mTLS enrollment. JWT auth. Ed25519 script signing. The pull model. Capability matching. You can read the code, describe what each file does, spot where a bug might live, and give precise instructions to an AI coding assistant. That's exactly what this course was for.