Running Axiom with confidence
Everything you need to dispatch jobs, manage nodes, schedule automation, and keep your fleet healthy — without touching the code.
Know Your Dashboard
After this module you'll know what every section of the dashboard is for and be able to orient yourself on first login.
What is Axiom?
Axiom is a remote job execution platform. It lets you write a script, push it to a fleet of machines (nodes), and see the output — all from a single dashboard. Think of it as a command center for your infrastructure.
Unlike traditional SSH or push-based tools, Axiom uses a pull model: each node calls home every few seconds asking "do you have work for me?" This means nodes can sit behind firewalls, on home networks, or in a cloud VPC — as long as they can make outbound HTTPS requests, they work.
You never SSH into a node. You dispatch a job from the dashboard, and within seconds the next available node picks it up and runs it. Results appear in the dashboard automatically.
The nine sections at a glance
Use the left sidebar to navigate. Here's what each section does:
Live summary — active node count, running jobs, 7-day failure rate chart, and recent activity. Your starting point every shift.
Your fleet. See every enrolled machine, its CPU/RAM, status, tags, and runtime health. Enroll new nodes, revoke compromised ones.
The job queue. Dispatch one-off scripts, monitor their status in real time, bulk-cancel or resubmit, download output.
Live job queue monitor (/queue). See pending jobs grouped by node, queue depth per node, and DRAINING state — useful for diagnosing why a job is waiting to be picked up.
Scheduled, repeating jobs. Define a cron schedule, a signed script, and target nodes — the system fires them automatically.
Build custom Docker images for your nodes — bundling specific Python packages, tools, and network configs. Admin-only.
Register Ed25519 public keys. Every job script must be signed with a registered key before nodes will execute it — this is your tamper-proof guarantee.
Team access control. Create accounts, assign roles (admin / operator / viewer), configure permissions. Admin-only.
Immutable security trail. Every login, job dispatch, user change, and node revocation is recorded with timestamp and actor.
Your dashboard at a glance
First-time setup checklist
New to this instance? Work through these steps before dispatching your first job.
Adding & Managing Nodes
After this module you'll be able to enrol new machines, understand every node status, edit node configuration, and safely revoke access.
What is a node?
A node is any machine running the Axiom agent. Nodes can be:
- Physical servers in your data centre
- Cloud VMs (EC2, GCE, Azure VMs)
- Docker containers (via the node compose file)
- Developer workstations (for testing)
Each node has a set of tags that describe its capabilities and environment. Jobs can target a specific node by ID, or filter by tags to let the system find the best available match.
How to enrol a new node
Enrolment uses a JOIN_TOKEN — a one-time credential that lets the node trust your server and get a signed certificate. Tokens expire after 24 hours.
Navigate to Admin in the left sidebar, then click the Onboarding tab. You'll see the Node Enrollment card.
A token string appears. Click the copy button immediately — the token is shown once and expires in 24 hours.
On the machine you want to enrol, set the environment variable JOIN_TOKEN=<paste here> and start the Axiom agent using the node compose file.
Navigate to Nodes. Within 10–30 seconds the new node card appears with status ONLINE. The agent fetched its client certificate and began polling for work.
Click the gear ⚙️ icon on the node card and add tags like python:3.11, prod, or eu-west. This lets jobs target nodes by capability.
Each JOIN_TOKEN contains your server's Root CA certificate. Treat it like a password. If you accidentally expose a token, click "Revoke & Create New" before a new machine joins the fleet.
Node statuses explained
| Status | What it means | What to do |
|---|---|---|
| ONLINE | Healthy, polling for work | Nothing — it's ready |
| BUSY | Executing one or more jobs right now | Wait for jobs to complete, or check the node drawer for detail |
| OFFLINE | Missed 3+ heartbeats (≈90 seconds) | Check the machine is running and has network access to the server |
| TAMPERED | Runtime health check found unexpected changes | Review the drift details in the node card, investigate, then click "Clear tamper alert" |
| DRAINING | Admin requested graceful drain — no new jobs assigned | Wait for active jobs to finish, then the node can be safely shut down or reconfigured |
| REVOKED | Certificate revoked — node cannot receive any work | To re-enrol: generate a new JOIN_TOKEN and restart the Axiom agent on that machine |
Reading node health
Each node card shows live CPU and RAM gauges plus a sparkline — a tiny chart showing the last 20 data points (about 10 minutes). Use it to spot memory leaks or sustained high CPU.
The drawer shows: the job currently running, all eligible pending jobs for that node, recent execution history, and the full capabilities list. This is the fastest way to diagnose a stuck job.
The Runtime Health section on each card flags drift — tools missing, version mismatches, or unauthorised packages. An amber "Drift Detected" badge means something changed on the node since it enrolled. Expand the health section to see exactly which tools triggered the alert.
Editing node settings
Click the gear ⚙️ icon on any node card to enter inline edit mode. You can change:
- Concurrency limit — max jobs this node runs in parallel (default: 1)
- Memory limit — maximum RAM a single job can use on this node (e.g.
512m,2g) - Tags — capabilities and labels for job routing
- Environment tag — PROD / TEST / DEV (used for filtering in the UI)
Changing tags or concurrency while a node is actively running jobs can cause in-flight jobs to stall or get mis-routed. Use the Drain option (in the node detail drawer) first, wait for jobs to complete, then edit.
Revoking a node
Revoke a node when it's been decommissioned, compromised, or you want to permanently remove it from the fleet. Revocation adds the node's certificate to the CRL, so the machine can no longer receive jobs or enrol again with the same certificate.
Use the environment filter or scroll to find the node you want to revoke.
A confirmation dialog appears. Read it carefully — this is irreversible without a new JOIN_TOKEN.
The node card goes grey with a REVOKED badge. The CRL is updated immediately. Any in-progress jobs on that node will be abandoned and re-queued.
Revoking a node certificate cannot be undone. To bring the machine back into the fleet, you must generate a new JOIN_TOKEN and restart the Axiom agent on that machine — it will enrol as a new node.
gpu. Where do you set this requirement?
Dispatching & Monitoring Jobs
After this module you'll be able to dispatch jobs to your fleet, understand every status in the queue, read execution output, and use bulk actions efficiently.
The job lifecycle
Every job travels through this state machine from dispatch to completion:
Waiting in queue
Node executing
Output available
Check error log
Signature invalid
A FAILED job automatically retries up to 3 times (configurable) with exponential backoff — you'll see RETRYING with a countdown timer. If all retries fail it becomes DEAD_LETTER — a terminal state that requires manual action.
Dispatching your first job
Open the Jobs page. The left panel is the dispatch form. Use the Guided mode (default) for common scripts — switch to Advanced for full control.
Type a descriptive name in the Name field — e.g. disk-cleanup-web01. This appears in the queue and audit log.
Choose Python, Bash, or PowerShell depending on your script type. This tells the node which interpreter to use.
Paste the script into the Script textarea. Keep it self-contained — the node has no shared filesystem with you.
Pick a key from the Signature dropdown. The script will be signed with this key before dispatch. If the dropdown is empty, you need to register a key in Signatures first — see Module 5.
Leave blank to let any available node pick it up. Or select a specific node from the node dropdown, or add capability requirement tags like python:3.11 to filter the eligible pool.
The job appears in the queue on the right with status PENDING. Within seconds a node picks it up. The status updates live via WebSocket — no refresh needed.
Check that you have at least one ONLINE node and that its tags match your capability requirements. A job with gpu required will stay PENDING forever if no nodes have a gpu tag.
Reading the job queue
Reading a job's detail panel
Click any job row to open the detail panel. Here's what you'll find:
| Section | What's in it |
|---|---|
| Status & metadata | Current status, attempt count, assigned node, duration, created-by user |
| Dispatch diagnosis | Why this job went to this node (capability matching explanation) |
| Output logs | stdout and stderr from the script execution — scroll down for the full output |
| Execution records | History of every attempt. Pinnable — click the pin icon to keep an important attempt visible. |
| Flight recorder | Error messages and stack traces, only populated on FAILED jobs |
| Node health at execution | CPU/RAM snapshot from the moment the node picked up this job |
From the detail panel you can Cancel a PENDING or ASSIGNED job, or Resubmit a FAILED / COMPLETED job to dispatch a fresh copy.
Bulk actions
Select jobs using the checkboxes — the filter bar is replaced by a bulk action bar showing the count and available actions: Cancel, Resubmit, or Delete. Available actions depend on the selected jobs' statuses.
Deleting jobs removes them from the queue permanently, including their output and audit trail. Only delete jobs you're certain you no longer need. For completed jobs you want to keep off the queue, use the date-range filter instead of deleting.
Scheduling Jobs
After this module you'll be able to create, manage, and publish scheduled job definitions that fire automatically on a cron schedule.
What is a Job Definition?
A Job Definition is a repeating, pre-signed script with a cron schedule. Think of it as a saved template that the system fires automatically — nightly database backups, hourly health checks, weekly log rotations.
Unlike one-off jobs, definitions go through a sign → publish → activate workflow. This ensures every scheduled execution uses a verified, approved script — the DRAFT state prevents unsigned or untested scripts from running automatically.
Script edited, awaiting signature — will NOT fire automatically
Signed and published — fires on schedule automatically
Creating a scheduled job
The Job Definition modal opens with empty fields.
Give it a clear name like nightly-db-backup. Paste the script that should run. Remember: it runs unattended, so it must be fully self-contained and handle its own errors.
Enter a cron expression. Examples: 0 2 * * * = 2am daily, */30 * * * * = every 30 minutes, 0 9 * * 1 = 9am every Monday.
Optionally target a specific node, or add tags to route to the right pool. Leave blank to use any available node.
Pick from the Signature dropdown. This key signs every execution of this definition — make sure it's registered in Signatures.
Saving creates a DRAFT. Clicking the gold star icon publishes it — status changes to ACTIVE and the scheduler begins watching the cron expression.
Editing an existing definition
Click the pencil ✏️ icon to edit any definition. If you change the script content, the definition moves back to DRAFT automatically — you'll see a warning. This is by design: any script change requires a new signature before it can run automatically again.
To re-sign and re-activate after editing: click the lock icon (visible on DRAFT definitions) to open the signature picker, select your key, and confirm. The definition returns to ACTIVE.
The DRAFT state prevents a situation where someone edits a scheduled script and it immediately runs on production nodes without review. The mandatory re-sign step is your last checkpoint before automation takes over.
Viewing execution history
Click any definition row to expand its execution history. You'll see every past run grouped by date with status badges. Click the Logs button on any run to see the full output — identical to the Job Detail Panel from the Jobs page.
Use the toggle switch on each definition row to enable/disable without publishing or deleting. Disabled definitions won't fire but stay configured for when you need them again.
Scheduling Health
The Health tab on the Scheduled Jobs page (/job-definitions) shows how reliably each definition has been firing over a selected window — 24 h, 7 d, or 30 d. It is the primary tool for catching silent scheduling failures before they impact operations. If a definition is supposed to run every hour but hasn't fired in three hours, the Health tab will surface that immediately.
| Metric | What it means |
|---|---|
| fired | Scheduler dispatched the job for this slot — expected activity |
| skipped | Slot intentionally skipped — job was in DRAFT status, or a previous run was still active when the fire time arrived |
| late | Expected fire window passed but the next scheduled fire has NOT yet elapsed — job is behind; check node availability and whether the scheduler service is running |
| missed | Expected fire window passed AND the next scheduled fire has also passed — this fire slot is permanently gone; investigate the cause |
| failed | Job was dispatched but subsequently reached FAILED status — the scheduler fired it, but execution failed on the node |
LATE (amber warning) — the scheduler may still catch up. The grace period is 5 minutes: a job fired within 5 minutes of its scheduled time counts as on-time. Check whether nodes are available and whether the scheduler service is running. Action is advised but not yet urgent.
MISSED (red error) — the window is gone and cannot be recovered. Investigate: was the scheduler stopped? Were all nodes offline or DRAINING? Was the definition in DRAFT? Each missed fire is a gap in your automation record.
Each definition shows an overall health status: ok — no missed or failed events; warning — at least one late event; error — at least one missed or failed event. Use this to triage at a glance without drilling into individual fire slots.
The API endpoint GET /api/health/scheduling?window=24h|7d|30d returns the same health data in JSON format. Integrate it with your monitoring or alerting stack to get notified when any definition crosses into warning or error state.
Execution records (the ScheduledFireLog entries that power the Health tab) are pruned after retention_days — default is 14 days, configurable in Admin › Retention. If your retention setting is shorter than the selected health window (for example, 7-day retention with the 30-day window open), older fire slots will simply be absent from the data. Recommendation: set retention to at least 30 days to ensure reliable long-window health reporting.
Signing Keys & Security
After this module you'll understand why job signing exists, how to register a key, and what to do when a job is rejected for a bad signature.
Why does every job need a signature?
Job signing is your guarantee that the script that runs on a node is exactly the script you wrote. Without it, anyone who gained access to the job queue could inject malicious scripts. With signing, nodes verify the Ed25519 signature before executing — if the script was tampered with in transit or in the database, the node rejects it immediately.
The workflow is: you (or your CI/CD system) sign scripts with a private key that you keep secret. You register only the corresponding public key in the dashboard. Nodes check the signature against registered public keys — they never see your private key.
Registering a signing key
You need an Ed25519 key pair. Generate one with the admin tool, then register the public key in the dashboard.
Run: python ~/Development/toms_home/.agents/tools/admin_signer.py --generate
This creates secrets/signing.key (private) and secrets/verification.key (public). Keep signing.key safe — never commit it.
Open secrets/verification.key — copy the full PEM content including the -----BEGIN PUBLIC KEY----- header and footer.
The registration dialog opens with a Name field and a PEM textarea.
e.g. ci-deploy-key or thomas-workstation. This name appears in job dispatch forms and the audit log.
The key card appears with an Active badge. The key is now available in the signature dropdown when dispatching jobs.
The Signatures page only accepts public keys (they start with -----BEGIN PUBLIC KEY-----). Your private key (signing.key) must stay on your machine or in your secrets manager. Pasting a private key would expose it to anyone with dashboard access.
Signing a script before dispatch
The dashboard signs scripts automatically when you select a key from the Signature dropdown in the dispatch form — no manual step needed for UI dispatch. The signature is computed from the script content and attached to the job payload before sending to the server.
For CI/CD pipelines or command-line dispatch, use the admin signer tool:
python ~/Development/toms_home/.agents/tools/admin_signer.py --sign my_script.py
This outputs a base64-encoded signature that you include in the signature_payload field of the API job submission.
What SECURITY_REJECTED means
A job shows SECURITY_REJECTED when the node's signature check failed. This means:
- The script content was modified after signing (tamper attempt), or
- The signature was generated with a key that isn't registered in Signatures, or
- No signature was provided at all
SECURITY_REJECTED is a terminal status — the job will not retry. Open the Job Detail Panel and check the Flight Recorder section for the specific error. Then re-dispatch with a valid signature.
Managing Users & Roles
After this module you'll be able to invite team members, assign the right role, configure granular permissions, and manage the user lifecycle.
The three roles
Axiom uses RBAC with three built-in roles:
| Role | Who it's for | Can dispatch jobs? | Can manage nodes? | Can manage users? |
|---|---|---|---|---|
| Admin | System owner / lead operator | ✓ Yes | ✓ Yes | ✓ Yes — all operations |
| Operator | Day-to-day operations | ✓ Yes | ✓ Yes (limited) | ✗ No |
| Viewer | Read-only monitoring | ✗ No | ✗ No | ✗ No |
The admin role always has full access regardless of what's in the permissions table. Operator and viewer permissions are fully customisable — you can grant or revoke individual permissions from the Role Permissions panel in the Users page.
Creating a new user
An inline form appears at the top of the user table.
Choose a role from the dropdown: viewer, operator, or admin. The user can change their password after first login.
Click the ⏱️ icon to toggle the must-change-password flag. The user will be prompted to set a new password on their first login — they can't use the dashboard until they do.
The user row appears in the table. Share the username and temporary password securely (not by email). The amber ⚠️ badge on their row reminds you the force-change flag is active.
Customising role permissions
Scroll down on the Users page to the Role Permissions section. Each role panel shows its currently granted permissions as chips. Click the + on any missing permission chip to grant it, or the × on an existing one to revoke it.
Permission naming follows resource:action format. Key permissions:
| Permission | What it allows |
|---|---|
jobs:write | Dispatch, cancel, and resubmit jobs |
jobs:read | View jobs and their output |
nodes:write | Edit node settings, drain, revoke |
nodes:read | View nodes and their health |
signatures:write | Register and delete signing keys |
foundry:write | Build and manage node images |
users:write | Create, edit, and delete users (admin only by default) |
Changing permissions on the operator role affects every user with that role, immediately. If you need different permissions for different people, assign them different roles or create a custom permission configuration for one role before adding them.
Session security & password management
A few important security behaviours to know:
- Password change invalidates all sessions — when a user changes their password, all their other active sessions are immediately invalidated. This is intentional: if a password was compromised, changing it locks out the attacker's session instantly.
- Resetting a user's password — click the 🔑 key icon on any user row to expand an inline form. Enter the new password twice and click "Set Password". The user must re-login on all devices.
- Force change flag — the ⏱️ icon toggles
must_change_password. Use it when creating new accounts or after a suspected breach. The user sees a modal on login and cannot access the dashboard until they change it. - Deleting a user — click the 🗑️ icon and confirm. All their sessions are terminated immediately. Their jobs remain in the queue under their username for audit purposes.
When Things Go Wrong
After this module you'll have a clear diagnostic path for the most common failure modes: stuck jobs, failed scripts, node alerts, and security rejections.
Job stuck in PENDING
A job stays PENDING when no eligible node picks it up. Work through this checklist:
Go to Nodes. If all nodes are OFFLINE, the problem is your fleet — not the job. Investigate why nodes lost connectivity.
Open the Job Detail Panel and read the Dispatch Diagnosis section. It explains which capability requirements couldn't be matched. Compare against the online nodes' tag lists.
If the job requests more RAM than any node's configured memory limit, it stays PENDING. Either lower the job's memory requirement or raise the node's limit.
Occasionally a scheduler edge case leaves a job orphaned. Cancel and resubmit — a fresh dispatch usually resolves it.
Reading a FAILED job
When a job shows FAILED, open the Job Detail Panel and check these sections in order:
This shows the error message and stack trace from the last execution. Start here — it usually tells you exactly what went wrong (e.g. ModuleNotFoundError: No module named 'requests').
Full stdout/stderr output from the script. Scroll to the bottom — most scripts print their error just before exiting.
If the job retried, check each attempt's output separately. Sometimes failures are intermittent (network timeout) vs. deterministic (bad script).
If the script has a bug, fix it, re-sign it, and dispatch a new job. If the failure was environmental (missing package on node), address that first — then resubmit from the detail panel.
If a job shows DEAD_LETTER, all automatic retry attempts failed. It won't retry again automatically. Review the error, fix the underlying issue, then use Resubmit to dispatch a fresh copy.
Investigating a TAMPERED node alert
A TAMPERED status means the node's runtime health check detected unexpected changes since the node enrolled. This could be:
- New tools installed that weren't in the original image
- Version drift — a tool was upgraded or downgraded
- Actual tampering — an unauthorised change was made to the machine
Click the node card to open it. The health section shows which specific tools triggered the alert — tool name, expected version vs found version.
Log into the machine and verify the changes. Were they authorised maintenance? An automated system update? Or something unexpected?
Click the amber "Clear tamper alert" button on the node card. The node returns to ONLINE and can receive jobs again.
If you can't explain the changes, treat the node as compromised. Revoke its certificate immediately. Investigate the machine out-of-band. Re-enrol only after clean reimaging.
SECURITY_REJECTED — signature failure
See Module 5 for the full explanation. Quick diagnostic steps:
| Symptom | Likely cause | Fix |
|---|---|---|
| All jobs from one CI pipeline rejected | Signing key deleted from Signatures | Re-register the public key in Signatures |
| Jobs rejected after script edit | Script changed without re-signing | Re-sign the script with the private key, dispatch again |
| One-off job rejected, same key works elsewhere | Script copy-paste introduced extra whitespace or encoding change | Re-sign the exact script content that will be dispatched |
Using the Audit Log for incident investigation
The Audit Log page records every security-relevant action: logins, job dispatches, user changes, node revocations, permission grants. Use it to reconstruct what happened during an incident.
Automated actions from CI/CD pipelines using API keys appear in the audit log with the prefix sp: (service principal). This lets you distinguish human actions from automated ones when reviewing incident timelines.
You now have everything needed to confidently operate Axiom — from node enrolment to job scheduling, security management, and incident response.
Questions? Check the in-app Docs section or the CLAUDE.md reference.