# ![The cronstable wordmark; its l is a live self-balancing double pendulum: it sways through the theme glitches, collapses when the signal drops, and swings itself back upright](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/logo-balance.webp)

[![PyPI version](https://img.shields.io/pypi/v/cronstable.svg?logo=pypi&logoColor=white&color=0073b7)](https://pypi.org/project/cronstable/)
[![GitHub release](https://img.shields.io/github/v/release/ptweezy/cronstable?logo=github&color=8a2be2)](https://github.com/ptweezy/cronstable/releases/latest)
[![Release downloads](https://img.shields.io/github/downloads/ptweezy/cronstable/total?logo=github&label=binary%20downloads&color=fb8c00)](https://github.com/ptweezy/cronstable/releases)
[![Python versions](https://img.shields.io/pypi/pyversions/cronstable.svg?logo=python&logoColor=ffd343&color=306998)](https://pypi.org/project/cronstable/)
[![PyPI status](https://img.shields.io/pypi/status/cronstable.svg?color=2ea44f)](https://pypi.org/project/cronstable/)
[![Platforms](https://img.shields.io/badge/platforms-Linux%20%7C%20macOS%20%7C%20Windows-00bcd4)](https://github.com/ptweezy/cronstable/releases/latest)
[![Architectures](https://img.shields.io/badge/arch-amd64%20%7C%20arm64%20%7C%20armv7%20%7C%20armv6%20%7C%20i686%20%7C%20ppc64le%20%7C%20s390x%20%7C%20riscv64-c2185b)](https://github.com/ptweezy/cronstable/releases/latest)
[![CI](https://github.com/ptweezy/cronstable/actions/workflows/release.yml/badge.svg)](https://github.com/ptweezy/cronstable/actions/workflows/release.yml)
[![Coverage](https://img.shields.io/codecov/c/github/ptweezy/cronstable?logo=codecov&logoColor=white&color=f01f7a)](https://codecov.io/gh/ptweezy/cronstable)
[![Container image](https://img.shields.io/badge/ghcr.io-ptweezy%2Fcronstable-2496ed?logo=docker&logoColor=white)](https://github.com/ptweezy/cronstable/pkgs/container/cronstable)
[![Docker Hub](https://img.shields.io/badge/docker.io-ptweezy%2Fcronstable-2496ed?logo=docker&logoColor=white)](https://hub.docker.com/r/ptweezy/cronstable)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-2a6db2)](https://mypy-lang.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

# cronstable™
/ kraahn-stuh-bl /

A stability-focused, container-friendly, optionally-distributed, fault-tolerant, highly-available, leader-electing, configurable, precompiled, multi-architecture, portable, batteries-inlcuded, security-hardened, production-ready cron replacement.

## Why cronstable?

* **Built for locked-down containers.** Runs unmodified under restricted
  Kubernetes PodSecurity: non-root, read-only root filesystem,
  `RuntimeDefault` seccomp, every Linux capability dropped (see
  [Production container deployment](#production-container-deployment)).
* **Prebuilt for practically everything.** Multi-architecture images on GHCR
  and Docker Hub. Also, self-contained binaries for Linux (glibc and musl),
  macOS (signed and notarized) and Windows, so Python on the host is optional
  (see [Installation](#installation)).
* **Observability and durability**: A live
  [web dashboard](#web-dashboard), native [Prometheus metrics](#metrics),
  per-job [resource monitoring](#resource-monitoring), and opt-in
  [durable state](https://github.com/ptweezy/cronstable/wiki/Durable-State),
  [orchestration DAGs](https://github.com/ptweezy/cronstable/wiki/Orchestration-and-DAGs)
  and [leader-elected clustering](#clustering-and-leader-election).

## Features

* "Crontab" is in YAML format; classic crontab files are accepted as-is too
  (see [Classic crontab files](#classic-crontab-files))
* **Business-day schedules**: `LW` (the month's last weekday), `L-3` (three
  days before month-end), `15W` (the weekday nearest the 15th), and `5#3`
  (the third Friday) express payroll/billing-style cadences directly, and
  Quartz day expressions largely paste straight in (see the
  [Business-Day Schedules](https://github.com/ptweezy/cronstable/wiki/Business-Day-Schedules)
  wiki page)
* Built-in **schedule linting**: dead schedules that can never fire again are
  called out loudly (never silently dropped), and common footguns (AND day
  semantics, uneven `*/n` steps, day-31-in-April, schedules that DST skips or
  repeats) are flagged at config load, in the dashboards, and over the API
  (see [Schedule introspection](#schedule-introspection))
* Builtin sending of Sentry, Mail, and webhook (Slack-compatible)
  notifications when cron jobs fail
* **End-to-end encrypted push notifications**: a fifth reporter seals each
  alert to a paired device's own key (libsodium sealed box), so the relay
  that forwards it to the platform push service never sees job names,
  hostnames, or log lines; pairing is a dashboard QR scan or one API call,
  and an opt-in Bonjour/mDNS advert lets a companion app find the daemon on
  the LAN (see [Push notifications](#push-notifications) and the
  [Push Notifications](https://github.com/ptweezy/cronstable/wiki/Push-Notifications)
  and [LAN Discovery](https://github.com/ptweezy/cronstable/wiki/LAN-Discovery)
  wiki pages)
* Flexible configuration: you decide how to determine if a cron job fails or not
* Designed for running in Docker, Kubernetes, or 12 factor environments:
  * Runs in the foreground
  * Logs everything to stdout/stderr
  * Production-ready for locked-down corporate container platforms: runs as a
    non-root user, under a restricted seccomp profile, with a read-only root
    filesystem, an `fsGroup`-mounted config, and all Linux capabilities
    dropped, so no writable paths or elevated privileges are required (see
    [Production container deployment](#production-container-deployment))
* Option to automatically retry failing cron jobs, with exponential backoff
* **Per-job SLA monitoring**: an `sla:` block declares thresholds for the runs
  that did not happen: too long without a success, a due slot that never
  started, a run exceeding its runtime bound. A breach fires a dedicated
  `onLate` reporting hook once (mail, Sentry, shell, webhook), gauges and
  counters land in the metrics, and the dashboards badge the job OVERDUE
  (see [Late-run detection](#late-run-detection-sla-monitoring) and the
  [Late-Run Detection](https://github.com/ptweezy/cronstable/wiki/Late-Run-Detection)
  wiki page)
* **Runtime pause/resume**: pause any job's scheduled fires for a bounded
  window (an hour by default, thirty days at most) over the API, the
  dashboards, or MCP, without touching the config. Skipped slots are recorded
  visibly, pending retries defer, catch-up owes nothing for the window, and
  with a `state:` store the pause survives restarts and is honored by every
  node (see the
  [Pausing Jobs](https://github.com/ptweezy/cronstable/wiki/Pausing-Jobs)
  wiki page)
* **Opt-in durable state**: point a single `state:` config block at a local
  directory (or an Amazon S3 Files / EFS mount to share it fleet-wide) and jobs
  gain durability across restarts: missed-run catch-up after downtime and
  retries that survive a daemon restart. The same store is handed to the jobs
  themselves over a loopback endpoint, so a job command can reach for durable
  key/value, an ETL cursor/watermark, a fleet-wide mutex or semaphore,
  idempotency keys, a shared artifact store and run-scoped secrets with
  `cronstable state|cursor|lock|artifact|idempotent|secret` (see the
  [Durable State](https://github.com/ptweezy/cronstable/wiki/Durable-State) wiki
  page); without it, cronstable stays stateless as before
* **Opt-in orchestration DAGs**: a `dags:` block turns the scheduler into a
  small, durable workflow engine: tasks with `dependsOn` edges, cross-task
  data hand-off (XCom), dynamic fan-out/mapping, sensors, human approval gates,
  whole-DAG backfill, and crash-resume of a partial graph, all on the same
  state store and coordinated across a fleet under a single lease so a task
  never double-launches (see the
  [Orchestration and DAGs](https://github.com/ptweezy/cronstable/wiki/Orchestration-and-DAGs)
  wiki page)
* Optional HTTP REST API, to fetch status, start jobs, cancel running jobs, and
  read per-job run history on demand
* **Native TLS on the listeners**: `web.listen` accepts `https://` addresses
  served from a `web.tls` block, mixed freely with plaintext and unix-socket
  entries on one runner, and an optional `clientCa` makes the listener require
  a client certificate signed by your own CA (mutual TLS), so it authenticates
  its callers rather than merely encrypting them. A web certificate replaced in
  place is picked up without a daemon restart. The job-facing state API gains
  the same block as `state.jobApi.tls`, and `cronstable tui` / `cronstable mcp`
  gain `--cacert`, `--client-cert`, `--client-key` and `--insecure` (see
  [Serving the API over TLS](#serving-the-api-over-tls) and the
  [Listener TLS](https://github.com/ptweezy/cronstable/wiki/Listener-TLS)
  wiki page)
* **iCal calendar export**: subscribe any calendar app to `GET /calendar.ics`
  (or one job's `/jobs/{name}/calendar.ics`) and the fleet's upcoming fires,
  enumerated by the scheduler's own engine, land on the on-call engineer's
  calendar; the dashboard draws the same data as a seven-day **week
  calendar** (see the
  [Calendar Export](https://github.com/ptweezy/cronstable/wiki/Calendar-Export)
  wiki page)
* Optional **[MCP server](https://github.com/ptweezy/cronstable/wiki/MCP)** for
  AI agents. An agent can **observe**
  cronstable, **author and debug schedules** with the daemon's own engine
  (validate/explain an expression, explain field by field why a job did not
  run at a timestamp), and **control** it when you opt in. It is read-only by
  default and exposes tools, resources, and triage prompts covering jobs,
  DAGs, the cluster/fleet, metrics, and durable state. It is served at
  `POST /mcp` on the web listeners and through a `cronstable mcp` stdio
  bridge, and is hand-rolled in pure Python with no new dependencies
* Native **Prometheus metrics** at `/metrics` (plus per-job statsd push
  metrics), covering run outcomes, durations, retries, schedules, and cluster
  health (see [Metrics](#metrics))
* Opt-in **per-job resource monitoring**: one `monitorResources: true` samples
  each run's CPU time and peak memory across its whole process tree, live and
  per run, in the dashboard, the metrics, and the failure reports (see
  [Resource monitoring](#resource-monitoring))
* A **job-set id**: an order-independent fingerprint of every job's effective
  configuration, so replicas deployed from the same config can confirm they
  hold an identical set of jobs (see [Job-set id](#job-set-id))
* **Opt-in clustering and leader election**: optionally have instances confirm
  over mutual TLS that a configured set of peers is running the same job set, and
  **elect a leader** so several replicas can run from one config without
  double-running jobs (see
  [Clustering and leader election](#clustering-and-leader-election))
* Arbitrary timezone support
* Optional **[live control panel](#web-dashboard)**: watch every job's status,
  tail its logs in real time, run or cancel jobs on demand, review run history
  and success rates, drive DAG runs and approvals, and keep an eye on the whole
  cluster, from one self-contained page with ten themes and a shortcut for
  everything and a **[terminal twin](#terminal-dashboard)**
  (`cronstable tui`) with the same keys

[![cronstable web dashboard, animated: a tour of the live job overview, the command palette, a live log tail, a DAG's task graph, the nine-node cluster and fleet matrix, the wallboard and incident timeline, the device-pairing QR panel for encrypted push alerts, and the accessibility options (a colour-vision-safe palette and larger UI scale)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-reel.webp)](#web-dashboard)

> The (optional) Web UI tour.

## Quick start

You can have a running scheduler with a live dashboard in about a minute.
Install it (see [Installation](#installation) for Docker, Homebrew, and
no-Python binary options):

```shell
pip install cronstable
```

Describe your first job in a `cronstable.yaml`:

```yaml
jobs:
  - name: hello
    command: echo "hello from cronstable on $(hostname)"
    schedule: "* * * * *"        # every minute
    captureStdout: true

web:
  listen:
    - http://127.0.0.1:8080      # optional: the REST API + dashboard
```

Run it (always in the foreground):

```shell
cronstable -c cronstable.yaml
```

Open <http://127.0.0.1:8080/> and watch `hello` fire once a minute,
with its output tailing live in the [dashboard](#web-dashboard). From there, each of
these is a few lines of config away:

* **Never miss a silent failure**: retries with backoff and a Slack/mail/Sentry
  report when a job ultimately fails ([tutorial](#tutorial-1-alert-when-a-job-fails-then-retry-it)).
* **Survive restarts**: a one-line `state:` block makes history, retries and
  missed-run catch-up durable ([tutorial](#tutorial-2-survive-restarts-catch-up-what-was-missed)).
* **Chain jobs into a pipeline**: a durable DAG with data hand-off and an
  approval gate ([tutorial](#tutorial-3-your-first-dag-a-durable-pipeline)).
* **Run replicas safely**: leader election so two copies never double-fire
  ([tutorial](#tutorial-4-two-replicas-zero-double-runs)).
* **See it all at once**: `docker compose -f example/grand-tour/docker-compose.yml up
  --build` boots a nine-node cluster running every feature together
  ([example gallery](#example-gallery)).

Already have a crontab? You don't have to translate it:
`cronstable -c /etc/crontab` runs the classic format as-is (see
[Classic crontab files](#classic-crontab-files)).

## Installation

### Run with Docker

Prebuilt multi-architecture images (seven Linux platforms) are published on
every release to two registries, the GitHub Container Registry
(`ghcr.io/ptweezy/cronstable`) and Docker Hub (`ptweezy/cronstable`). Mount your crontab and go:

```shell
docker run --rm \
  -v "$PWD/cronstable.yaml:/etc/cronstable.d/cronstable.yaml:ro" \
  ghcr.io/ptweezy/cronstable:latest
```

The image runs as a non-root user and reads its configuration from
`/etc/cronstable.d` by default. The default image is built on Debian (slim);
Alpine, Ubuntu, RHEL/UBI, Fedora, openSUSE, Amazon Linux and distroless
variants are published from the same release under a `-<distro>` tag suffix.
The platform list, the variant table, and each variant's architecture
coverage are in the
[Installation](https://github.com/ptweezy/cronstable/wiki/Installation) wiki
page. For production, pin a specific version instead of `latest` and see
[Production container deployment](#production-container-deployment) for the
hardened Kubernetes/Docker setup.

### Install using pip

cronstable requires Python >= 3.10 (for systems with an older Python, use the
binary instead). Install it in a virtual environment, or let
[pipx](https://github.com/pipxproject/pipx) create one for you:

```shell
pip install cronstable   # inside a venv
pipx install cronstable  # or isolated, via pipx
```

### Install using Homebrew or winget

Both package managers install the self-contained release binary for your
platform, so no Python is required:

```shell
brew install ptweezy/tap/cronstable  # macOS or Linux, from the cronstable tap
winget install ptweezy.cronstable    # Windows
```

Upgrade later with `brew upgrade cronstable` or
`winget upgrade ptweezy.cronstable`.

### Install using binary

Alternatively, a self-contained binary can be downloaded from github:
<https://github.com/ptweezy/cronstable/releases>. Every release attaches
binaries for Linux (glibc and musl builds for `amd64`, `arm64`, `i686`,
`armv7`, `ppc64le`, `s390x` and `riscv64`, plus a musl-only `armv6`), macOS
(`amd64` and `arm64`, signed and notarized by Apple) and Windows (`amd64` and
`arm64`). Python is not required on the target system; it is embedded in the
executable:

```shell
# pick the asset for your OS and architecture (glibc amd64 Linux shown; append
# -musl on Alpine, or use cronstable-macos-<arch> on a Mac)
curl -fsSL -o cronstable \
  https://github.com/ptweezy/cronstable/releases/latest/download/cronstable-linux-amd64
chmod +x cronstable
./cronstable --version
```

The binary unpacks an embedded Python runtime at startup, so under a
read-only root filesystem it needs a small writable and executable temp
mount; the container image and `pip`/`pipx` installs never self-extract. The
full asset table, the glibc/musl compatibility notes, and the
tmpfs/`emptyDir` recipe are in the
[Installation](https://github.com/ptweezy/cronstable/wiki/Installation) wiki
page.

## Running on Windows

cronstable runs natively on Windows (x64 and ARM64), in addition to Linux and
macOS. Install it with `pip install cronstable`, or download the self-contained
`cronstable-windows-amd64.exe` / `cronstable-windows-arm64.exe` from the
[releases page](https://github.com/ptweezy/cronstable/releases) (no Python
required). Everything else, like the YAML crontab, scheduling, reporting, retries,
the HTTP API and the [web dashboard](#web-dashboard), works the same as on
POSIX. A few platform details differ:

* **Default config location.** When `-c` is omitted, cronstable looks in
  `%APPDATA%\cronstable` (e.g. `C:\Users\you\AppData\Roaming\cronstable`), the
  Windows analog of `/etc/cronstable.d`. Point it anywhere with `-c`:

  ```shell
  cronstable -c C:\path\to\cronstable.yaml
  ```

* **Default shell.** A string `command` with no explicit `shell` runs through
  the native command processor (`%ComSpec%`, i.e. `cmd.exe`), mirroring the
  `/bin/sh` default on POSIX. For PowerShell, or any other interpreter, set
  `shell:` or pass `command` as a list (which bypasses the shell entirely):

  ```yaml
  jobs:
    - name: powershell-job
      command:
        - powershell
        - -Command
        - Get-Date
      schedule: "*/5 * * * *"
      captureStdout: true
  ```

* **Graceful shutdown.** Press `Ctrl-C` (or `Ctrl-Break`) to stop cronstable; it
  shuts down after the currently running jobs finish, just as `SIGTERM` does on
  POSIX.

* **Not supported on Windows.** Per-job `user`/`group` switching (there is no
  `setuid`/`setgid` equivalent) is rejected with a clear configuration error,
  and `unix://` web listeners are skipped with a warning. Use an `http://`
  listener instead.

## Production container deployment

cronstable is built to run unmodified under the hardened security contexts that
corporate and enterprise Kubernetes / container platforms enforce. At runtime
the daemon only *reads* its configuration and secrets and writes its output to
stdout/stderr; it never needs a writable working directory, temp files, or log
files, so it can run as an unprivileged non-root user with the `RuntimeDefault`
seccomp profile, a read-only root filesystem, all Linux capabilities dropped,
and config/secret volumes mounted with an `fsGroup`. Only the optional per-job
[user/group switching](#change-to-another-usergroup) requires root. Two
exceptions need a small writable mount: a `unix://` web listener's socket, and
the standalone binary's temp directory (see
[Install using binary](#install-using-binary)).

The published image (`ghcr.io/ptweezy/cronstable` and `docker.io/ptweezy/cronstable`)
is already built this way (non-root, with `cronstable -c /etc/cronstable.d` as its
entrypoint and no writable paths required), so for most deployments you can use
it directly and mount your crontab read-only. The
[Production Deployment](https://github.com/ptweezy/cronstable/wiki/Production-Deployment)
wiki page has the full setup: a Kubernetes `Deployment` with a fully restricted
security context, baking configuration into your own image, the writable-path
exceptions in detail, and health checks.

## Web dashboard

cronstable ships with a **built-in web dashboard**: one self-contained page (no
build step, no external assets, no database) served straight from the daemon.
Point a browser at the HTTP listener and you have a keyboard-driven control
room for every job, and, when you use them, for the cluster, the DAGs, and the
durable state store too.

[![cronstable web dashboard: a live overview of every job, showing status, live resource usage, owner node, schedule, last run, next-run countdown, and a run-trend sparkline](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-overview.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-overview.png)

The overview shows every job with its **live status**, a **countdown to its
next run**, the last run's duration and exit-code badge, and a **sparkline of
recent runs**; jobs with [resource monitoring](#resource-monitoring) add live
**CPU and memory** chips while they run, and a cluster adds each job's
**owner node**. Everything is sortable, filterable, and searchable, and when
something is failing a **verdict bar** correlates the failures into one
headline ("4 share exit=69, likely one cause"). Click any job (or press
`Enter`) to open its detail drawer:

| Live log tail | Run history | Schedule, explained |
| :---: | :---: | :---: |
| [![Live log tailing with ANSI color, timestamps, and in-log search](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-logs.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-logs.png) | [![Run history with success rate, duration chart, and per-run CPU and peak-memory columns](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-history.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-history.png) | [![A plain-English schedule with timezone-aware next-run times](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-schedule.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-schedule.png) |
| Follow a running job's output **live** over Server-Sent Events, with ANSI color, in-log **grep** (plain text or regex), per-line timestamps, line-wrap, and one-click download. | **Success rate** plus average / min / max duration over the retained history, with a color-coded per-run chart; with [resource monitoring](#resource-monitoring) on, **CPU time and peak memory** per run and in the stats. | A **plain-English** reading of the cron expression and a **timezone-aware preview of the next run times**, computed live in the browser. |

Every action has a key. A fuzzy command palette (`Ctrl-K` / `⌘K`) runs any
action or jumps to any job, `?` lists every shortcut, `/` filters, `j`/`k`
move the cursor, `r` runs the selected job and `x` cancels it. A click runs a
single job on demand, or every failing job at once.

| Fuzzy command palette | Keyboard-first, with a shortcut for everything |
| :---: | :---: |
| [![A fuzzy command palette listing run and log actions for each job](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-palette.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-palette.png) | [![The keyboard shortcut reference overlay](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-shortcuts.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-shortcuts.png) |

### Orchestration, live

[DAGs](https://github.com/ptweezy/cronstable/wiki/Orchestration-and-DAGs) get
their own card and drawer: trigger or backfill a run, watch the **task graph**
advance node by node, inspect per-task attempts, XCom values and logs, and
decide **approval gates** with a click, from any node in the fleet.

| The task graph | A human approval gate |
| :---: | :---: |
| [![The DAG drawer's graph tab: a diamond of tasks, every node green](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-dag-graph.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-dag-graph.png) | [![The DAG drawer's task list with an approval gate awaiting a decision, Approve and Reject buttons armed](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-dag-approval.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-dag-approval.png) |
| A `data-quality-gate` diamond: fan-out checks that reconverge on a `certify` task, colored by state as the run advances. | A release train **parked on a human**: the build succeeded, the approval gate is `awaiting`, and the sensor and publish tasks queue behind your decision. |

### The whole fleet on one page

With [clustering](#clustering-and-leader-election) on, a **cluster panel**
shows the quorum math, this node's role, per-peer attestation status, and,
with `cluster.observability`, every node's **whole-host CPU and memory**. The
**fleet view** goes further: a jobs × nodes matrix of the entire fleet's runs,
assembled from data that piggybacks on the gossip the nodes already exchange,
so any node can serve the single pane of glass.

| Cluster panel | Fleet view |
| :---: | :---: |
| [![The cluster panel: nine peers, all agreed, quorum met, with per-node load and per-node job ownership](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-cluster.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-cluster.png) | [![The fleet view: a jobs-by-nodes matrix with each node's last outcome and age per job](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-fleet.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-fleet.png) |
| Nine nodes, `8/8 agreed`, quorum met, per-node **load meters** and per-node **owns** counts under `distribution: spread`. | Every node's state for every job, one glance: ok / failing / running cells with ages, per-column node health, and a **failing only** filter. |

### When things break

Three panels are aimed at the 3 a.m. incident. The verdict bar's incident
timeline lays out every job's most recent finish, newest first, with the
correlated blast-radius set highlighted. The mitigate console starts or
cancels the failing set in bulk and copies a Markdown incident summary for
your ticket. The multi-tail console merges up to four jobs' live logs into one
pane, like tailing a set of pods.

| Incident timeline | Merged multi-tail |
| :---: | :---: |
| [![The incident timeline overlay: every job's most recent run, newest first, with failure reasons and exit codes](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-incident-timeline.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-incident-timeline.png) | [![The multi-tail console merging four jobs' live logs with identity colors and end-of-run markers](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-multitail.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-multitail.png) |
| "What happened, in what order": relative times, outcome glyphs, failure reasons, exit codes, durations, and a **failing only** filter. | Four streams, one pane: identity-colored prefixes, `end of run output` markers, auto re-attach on each job's next run. |

### Wallboards, heatmaps, and the state store

Press `w` for a full-screen **wallboard** built for a TV: worst-first tiles,
an incident stamp when something is failing, a `NO SIGNAL` banner when the
data goes stale (never a stale all-green), and a zen **screensaver** that
takes over when everything is healthy. The **activity heatmap** turns run
history into a punchcard (worst outcome per bucket, shaded by volume), and the
opt-in **state inspector** shows the [durable state store](https://github.com/ptweezy/cronstable/wiki/Durable-State)'s
health: record counts by kind, op latencies and errors, locks, cursors,
counters, artifacts, and quarantine.

| Wallboard / TV mode | Activity heatmap | Durable-state inspector |
| :---: | :---: | :---: |
| [![The wallboard: worst-first job tiles with an INCIDENT stamp and next-fire countdowns](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-wallboard.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-wallboard.png) | [![The activity heatmap punchcard: one row per job, cells colored by worst outcome and shaded by run volume](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-heatmap.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-heatmap.png) | [![The durable-state inspector: record counts per kind, op latencies, and per-primitive tabs](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-state.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-state.png) |

### Themes, Readability, and Accessibility

**Ten themes**: **carolina** (the default, a Carolina-blue CRT phosphor),
amber and green phosphor, and flat **modern** and **standard** looks, each in
a dark (phosphor) and a light (paper) variant. Cycle hues with `t`, flip
light/dark with `T`:

[![The same cronstable board cycling through all ten themes (carolina, amber, green, modern and standard, each in a dark phosphor and a light paper variant) and, for each, the terminal monospace and the readable proportional-sans interface font](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-themes.webp)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-themes.webp)

*(One board, ten themes, two interface fonts, animated: [WebP](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-themes.webp), [GIF](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-themes.gif). The four stills below are pulled from it.)*

| Amber phosphor CRT | Green phosphor CRT |
| :---: | :---: |
| [![The dashboard in the amber phosphor CRT theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-amber.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-amber.png) | [![The dashboard in the green phosphor CRT theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-green.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-green.png) |

| Flat modern theme | Carolina, on paper (light) |
| :---: | :---: |
| [![The dashboard in the flat modern theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-modern.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-modern.png) | [![The dashboard in the carolina light (paper) theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-carolina-light.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-theme-carolina-light.png) |

Beyond the themes: an optional proportional-sans interface font (shown per
theme in the animation above), UI scaling, deuteranopia- and tritanopia-safe
palettes, reduced-motion support, CRT-effect and notification toggles, all
remembered per browser, with status always carried by glyphs and text, not
colour or animation alone. There is also an optional (on by default, once per
12 hours) BIOS-style boot self-test that checks the daemon, job set, cluster,
and schedules for real while it types:

| Settings | Startup self-test |
| :---: | :---: |
| [![The settings panel: theme picker with carolina selected, CRT toggles, notifications, zen, and refresh interval](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-settings.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-settings.png) | [![The boot self-test screen: firmware version, job-set id, cluster role, and schedule scan, all OK](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-boot.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-boot.png) |

The `l` in the header's "cronstable" is a live cart-and-double-pendulum
simulation. I like to call him double-P, Peter Parker, or PP.

Run history and live logs are kept **in memory only** (unless you opt into the
durable state store), and the page is served with a strict
Content-Security-Policy. A one-line `web:` block turns it on: the
[**web dashboard tour**](https://github.com/ptweezy/cronstable/wiki/Web-Dashboard)
in the wiki is the full walkthrough, and
[Remote web/HTTP interface](#remote-webhttp-interface) below shows how to
enable it.

**Try it:** `docker compose -f example/zen-demo/docker-compose.yml up` boots a single node with a demo job set, and `docker compose -f example/cluster/docker-compose.yml up` boots a 3-node cluster (`cronstable-a`/`cronstable-b`/`cronstable-c`) so you can open each node's dashboard and watch the cluster panel and leader election live. For **every feature at once** (a 9-node mutual-TLS cluster sharing one durable state store and running the classic job set, durable-state jobs, orchestration DAGs and second-level probes together, with all five failure reporters wired to live sinks), run `docker compose -f example/grand-tour/docker-compose.yml up --build` (the [grand tour](example/grand-tour); see its [README](example/grand-tour/README.md)). More one-command demos are in the [example gallery](#example-gallery).

## Terminal dashboard

The dashboard has a **TUI sibling**: `cronstable tui` opens the board in
your terminal, over SSH, in a tmux pane, or on a box where a browser is
not an option. It is a client of the same HTTP control API (nothing extra
to enable on the daemon), and the shortcut table is the same one as the
web page's: `j`/`k` move, `Enter` opens a job's drawer, `r` runs, `x`
cancels, `/` filters, `Ctrl-K` opens the fuzzy command palette, and `?`
lists everything.

[![The cronstable TUI: a live 59-job board with status glyphs, next-fire countdowns, run sparklines, live CPU/memory chips, cluster owner column, and the verdict bar correlating a staged failure](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-overview.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-overview.png)

Press `Enter` on any job for its drawer, the same three tabs as the
web page, plus resources for monitored jobs:

| Live log tail | Run history | Schedule, explained |
| :---: | :---: | :---: |
| [![The Logs tab: a live SSE tail with per-line timestamps, in-log search with match highlighting, and end-of-run markers between runs](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-logs.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-logs.png) | [![The History tab: success rate, duration stats, and per-run rows with duration bars and CPU seconds](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-history.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-history.png) | [![The Schedule tab: the cron expression in plain English with the exact next fire instants from the daemon's own engine](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-schedule.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-schedule.png) |

| Fuzzy command palette | Keyboard-first, with the web page's keys |
| :---: | :---: |
| [![The command palette fuzzy-matching "run": global actions plus per-job and per-DAG commands](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-palette.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-palette.png) | [![The shortcut overlay: the web dashboard's shortcut table verbatim, with terminal extras grouped below](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-shortcuts.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-shortcuts.png) |

DAGs get the same drawer as the browser, and approval gates are decided
with a keypress:

| The task graph, mid-flight | A human approval gate |
| :---: | :---: |
| [![The DAG drawer's graph tab: the data-quality-gate diamond as topological layers, states coloring as the run advances](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-dag-graph.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-dag-graph.png) | [![The DAG drawer's tasks tab: release-train parked on its approval gate, with a approve / R reject armed](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-dag-approval.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-dag-approval.png) |

With clustering on, the cluster panel and the full fleet matrix render
in the terminal too:

| Cluster panel | Fleet view |
| :---: | :---: |
| [![The cluster panel: nine gossiping peers, all agreed, with per-node load and the lease detail](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-cluster.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-cluster.png) | [![The fleet view: a 59-job by 9-node matrix of live cells: ok, failing, and running with ages](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-fleet.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-fleet.png) |

The same incident tools are here, from the timeline to the multi-tail:

| Incident timeline | Merged multi-tail |
| :---: | :---: |
| [![The incident timeline: every job's most recent finish, newest first, with failure reasons, exit codes, and the blast-radius set flagged](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-incident-timeline.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-incident-timeline.png) | [![The multi-tail console merging four jobs' live logs with identity-colored prefixes and end-of-run markers](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-multitail.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-multitail.png) |

So are the wallboard, the heatmap, and the state inspector:

| Wallboard / TV mode | Activity heatmap | Durable-state inspector |
| :---: | :---: | :---: |
| [![The wallboard: worst-first tiles with failure ages and exit codes, run sparklines, and the tally foot](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-wallboard.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-wallboard.png) | [![The activity heatmap: one row per job, one cell per hour, worst outcome colored and shaded by volume](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-heatmap.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-heatmap.png) | [![The state inspector: store inventory, record streams, and document namespaces from the durable state store](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-state.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-state.png) |

The same ten themes as the browser (`t` cycles the hue, `T` flips
phosphor ↔ paper), with the same colour-vision-safe remaps and an
`--ascii` glyph mode:

| Amber phosphor | Green phosphor |
| :---: | :---: |
| [![The TUI in the amber phosphor theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-amber.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-amber.png) | [![The TUI in the green phosphor theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-green.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-green.png) |

| Flat modern | Carolina, on paper (light) |
| :---: | :---: |
| [![The TUI in the flat modern theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-modern.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-modern.png) | [![The TUI in the carolina light paper theme](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-carolina-light.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-theme-carolina-light.png) |

The TUI runs the same BIOS-style boot self-test, next to the settings
sheet:

| Startup self-test | Settings |
| :---: | :---: |
| [![The TUI boot self-test: link latency, firmware, job set, schedules, and cluster probed live, all OK](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-boot.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-boot.png) | [![The TUI settings panel: theme, color vision, refresh interval, log toggles, zen, and the boot self-test](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-settings.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/tui-settings.png) |

Run `cronstable tui` against the local daemon, or point it elsewhere with
`--url` and `--token-env`; `--tv` starts on the wallboard and `--job`
deep-links a drawer. The
[**Terminal Dashboard**](https://github.com/ptweezy/cronstable/wiki/Terminal-Dashboard)
wiki page is the full reference (options, every key, the panel tour).

## Tutorials

Four short walkthroughs you can copy and run, each built on the
[quick start](#quick-start) config and each pointing at the wiki page that
covers it in full.

### Tutorial 1: Alert when a job fails, then retry it

Classic cron mails root and hopes. Instead: retry with exponential backoff,
and page a Slack channel only if the job *ultimately* fails.

```yaml
jobs:
  - name: nightly-backup
    command: /usr/local/bin/backup --incremental
    schedule: "0 3 * * *"
    captureStderr: true            # include stderr in the report
    onFailure:
      retry:
        maximumRetries: 5
        initialDelay: 5            # 5s, 10s, 20s, 40s, ... capped at 300s
        maximumDelay: 300
        backoffMultiplier: 2
    onPermanentFailure:            # fires once, after the last retry is spent
      report:
        webhook:
          url:
            fromEnvVar: SLACK_WEBHOOK_URL
```

By default a job *fails* when it exits non-zero **or** writes to a captured
stderr; tune that per job with [`failsWhen`](#handling-failure). The webhook's
default body is Slack-compatible (Mattermost and Teams work as-is), and mail,
Sentry, and a shell command are equally one block away, with jinja2 templating
over the run's name, output, and exit code. Deeper:
[Failure Detection and Retries](https://github.com/ptweezy/cronstable/wiki/Failure-Detection-and-Retries)
and [Reporting](https://github.com/ptweezy/cronstable/wiki/Reporting) in the wiki.

### Tutorial 2: Survive restarts, catch up what was missed

Stateless is the default. When a deploy or a reboot lands mid-schedule, one
`state:` block gives jobs a memory:

```yaml
state:
  path: /var/lib/cronstable           # a local dir, or a shared mount for a fleet

jobs:
  - name: hourly-invoice-emit
    command: python -m billing.emit_hourly
    schedule: "0 * * * *"
    onMissed: run-all              # replay each hour missed while we were down
    startingDeadlineSeconds: 21600 # ...unless the slot is older than 6h
    onFailure:
      retry:
        maximumRetries: 10
        initialDelay: 30
        maximumDelay: 600
        backoffMultiplier: 2
```

With just the `state.path` line, run history survives restarts (the dashboard
rehydrates it), armed retries re-arm at their absolute deadlines, `@reboot`
means once per *boot* rather than once per daemon start, and Prometheus
counters stop resetting to zero. `onMissed` adds catch-up on top: `run-once`
coalesces any number of missed slots into one launch, `run-all` replays each
one, bounded by `startingDeadlineSeconds`. The same store also hands your job
*commands* durable primitives (key/value, cursors, fleet-wide locks,
idempotency keys, artifacts, run-scoped secrets) over a loopback endpoint:
`cronstable state|cursor|lock|idempotent|artifact|secret`. Deeper:
[Durable State](https://github.com/ptweezy/cronstable/wiki/Durable-State).

### Tutorial 3: Your first DAG, a durable pipeline

A `dags:` block turns the scheduler into a small, durable workflow engine.
This one builds, waits for a human, then publishes:

```yaml
state:
  path: /var/lib/cronstable           # DAGs live on the state store

dags:
  - name: release-train            # no schedule: manual-only
    tasks:
      - id: build
        command: make dist
      - id: approve
        type: approval             # parks the graph on a human decision
        dependsOn: [build]
      - id: publish
        dependsOn: [approve]
        command: make publish
        retries: 2                 # task-level retries, DAG-owned
        retryDelaySeconds: 60
```

Trigger it and approve the gate (or click **Approve** in the dashboard's DAG
drawer):

```shell
curl -X POST http://127.0.0.1:8080/dags/release-train/trigger
# -> {"dag": "release-train", "runKey": "manual-..."}
curl -X POST http://127.0.0.1:8080/dags/release-train/runs/<runKey>/tasks/approve/decision \
     -H 'Content-Type: application/json' -d '{"decision": "approve", "by": "alice"}'
```

Every transition is durable: restart the daemon mid-run and the run resumes
exactly where it was, and across a fleet the run advances under a lease so a
task never launches twice. Scheduled DAGs add catch-up and `backfill` over a
date range; tasks can pass data with `cronstable xcom push/pull`, fan out
dynamically over a list an upstream task produced, and poll for conditions
with `type: sensor`. Deeper:
[Orchestration and DAGs](https://github.com/ptweezy/cronstable/wiki/Orchestration-and-DAGs).

### Tutorial 4: Two replicas, zero double-runs

Run the same config on two (or nine) hosts that share a POSIX mount, and let
them elect a leader through a fenced lease file, with no certificates and no
coordination service:

```yaml
state:
  path: /mnt/shared/cronstable/state  # shared durable state (optional but natural here)

cluster:
  backend: filesystem
  filesystem:
    path: /mnt/shared/cronstable      # the mount is the election store
  nodeName: node-a                 # unique and stable per replica!
  electLeader: true

jobs:
  - name: charge-subscriptions
    command: python -m billing.charge
    schedule: "0 6 * * *"
    clusterPolicy: Leader          # the default: exactly the leader runs it
```

Only the elected leader fires `Leader` jobs; stop it and a follower adopts the
lease within its TTL. Per job, `clusterPolicy` picks the trade-off:
`Leader` (never double-runs, may skip when quorum is lost), `PreferLeader`
(never skips, may double-run under a partition), or `EveryNode` (genuinely
per-node work). No shared mount? The `gossip` backend elects over mutual TLS
with no shared store at all, `kubernetes` uses a `coordination.k8s.io` Lease,
and `etcd` a lease-bound key; `distribution: spread` load-balances job
ownership across the fleet instead of concentrating it on one leader. Deeper:
[Clustering and Leader Election](https://github.com/ptweezy/cronstable/wiki/Clustering-and-Leader-Election).

## Example gallery

Every example in [`example/`](example) is a self-contained, annotated,
runnable project; each compose file lives in its example's folder (the `demo`
quickstart uses the root `docker-compose.yml`). Highlights:

| Example | One command | Shows off |
| --- | --- | --- |
| [`demo`](example/demo) | `docker compose up` | The dashboard playground: varied jobs, live logs, retries, a long-runner, an on-demand job. |
| [`grand-tour`](example/grand-tour) | `docker compose -f example/grand-tour/docker-compose.yml up --build` | **Everything at once**: a 9-node mTLS cluster, shared durable state, five DAG patterns, second-level probes, all five reporters wired to live sinks. |
| [`cluster`](example/cluster) | `docker compose -f example/cluster/docker-compose.yml up` | A 3-node gossip cluster: peer attestation, quorum, leader election, live failover. |
| [`cluster-large`](example/cluster-large) | `docker compose -f example/cluster-large/docker-compose.yml up` | A 10-node, CPU-heavy fleet for watching `distribution: spread` and the load meters. |
| [`dag`](example/dag) | `cronstable -c example/dag` | Orchestration alone, single node: dependencies, XCom, fan-out, a sensor, an approval gate. |
| [`dag-cluster`](example/dag-cluster) | `docker compose -f example/dag-cluster/docker-compose.yml up` | DAGs coordinating across three nodes on one shared store: crash-resume, exactly-once tasks. |
| [`job-state`](example/job-state) | `cronstable -c example/job-state` | The job-facing state primitives: KV, cursors, locks, idempotency keys, artifacts, secrets. |
| [`mcp`](example/mcp) | `docker compose -f example/mcp/docker-compose.yml up --build` | The MCP server: an AI agent (Claude, Cursor, Copilot) observing and driving the scheduler over `POST /mcp`, or the `cronstable mcp` stdio bridge. |
| [`pulse-monitor`](example/pulse-monitor) | `docker compose -f example/pulse-monitor/docker-compose.yml up` | Second-level scheduling as a real-time uptime / SLA monitor. |
| [`pulse-cluster`](example/pulse-cluster) | `docker compose -f example/pulse-cluster/docker-compose.yml up` | The same probes fanned across a 3-node leader-electing cluster. |
| [`zen-demo`](example/zen-demo) | `docker compose -f example/zen-demo/docker-compose.yml up` | A deliberately calm board, for the wallboard's zen screensaver. |
| [`crontab`](example/crontab) | `cronstable -c example/crontab` | Classic Vixie crontabs running as-is next to YAML jobs. |
| [`kubernetes`](example/kubernetes) | `kubectl apply -f example/kubernetes/deployment.yaml` | Leader election through a `coordination.k8s.io/v1` Lease. |
| [`etcd`](example/etcd) | `docker compose -f example/etcd/docker-compose.yml up` | Leader election through an etcd lease, over plain HTTP. |
| [`docker`](example/docker) | `docker build` | The minimal "add cronstable to your own image" recipe. |

## Usage

Configuration is in YAML format.  To start cronstable, give it a configuration file
or directory path as the `-c` argument.  For example:

```shell
cronstable -c /tmp/my-crontab.yaml
```

This starts cronstable (always in the foreground!), reading
`/tmp/my-crontab.yaml` as configuration file.  If the path is a directory,
any `*.yaml` or `*.yml` files inside this directory are taken as
configuration files, along with any classic crontabs (`*.crontab`, `*.cron`,
or a file named `crontab`; see
[Classic crontab files](#classic-crontab-files)).

### Configuration basics

This configuration runs a command every 5 minutes:

```yaml
jobs:
  - name: test-01
    command: echo "foobar"
    shell: /bin/bash
    schedule: "*/5 * * * *"
```

The command can be a string or a list of strings.  If command is a string,
cronstable runs it through a shell, which is `/bin/bash` in the above example, but
is `/bin/sh` by default.

If the command is a list of strings, the command is executed directly, without a
shell.  The ARGV of the command to execute is extracted directly from the
configuration:

```yaml
jobs:
  - name: test-01
    command:
      - echo
      - foobar
    schedule: "*/5 * * * *"
```

The `schedule` option can be a string in the classic crontab format (5, 6 or 7 fields; ranges, steps, lists, `jan`/`mon` names, and Quartz's `?` standing alone in a day field), parsed by cronstable's built-in cron engine; see [Schedules and Timezones](https://github.com/ptweezy/cronstable/wiki/Schedules-and-Timezones) for the full dialect. Expressions in other dialects (Quartz `#`/`W`, the seconds-first 6-field layout) fail with an error naming the dialect and how to convert.
Additionally @reboot can be included , which will only run the job when cronstable is initially
executed. Further `schedule` can be an object with properties.  The following configuration
runs a command every 5 minutes, but only on the specific date 2017-07-19, and
doesn't run it in any other date:

```yaml
jobs:
  - name: test-01
    command: echo "foobar"
    schedule:
      minute: "*/5"
      dayOfMonth: 19
      month: 7
      year: 2017
      dayOfWeek: "*"
```

#### Schedule introspection

Six features answer questions about schedules, each with its own wiki page:

* Schedule linting: every schedule is linted at config load for legal
  expressions that probably do not mean what they say (no future occurrence,
  the day-of-month AND day-of-week rule, non-dividing `*/n` steps, wall
  times DST skips or repeats); findings surface on `/jobs` and `/status`,
  and `GET /schedule/preview` checks any expression before it becomes a job
  ([Schedule Linting](https://github.com/ptweezy/cronstable/wiki/Schedule-Linting)).
* Hashed schedules: an `H` field hashes a stable slot from the job's name,
  so a fleet of hourly jobs spreads across the hour instead of stampeding
  at `:00`
  ([Hashed Schedules](https://github.com/ptweezy/cronstable/wiki/Hashed-Schedules)).
* Schedule pressure: `GET /schedule/pressure` buckets the next 24 hours of
  fires into a collision heatmap, drawn in both dashboards
  ([Schedule Pressure](https://github.com/ptweezy/cronstable/wiki/Schedule-Pressure)).
* Duplicate detection: `GET /schedule/duplicates` groups jobs whose
  schedules fire on identical instants, by semantic equality
  ([Duplicate Schedule Detection](https://github.com/ptweezy/cronstable/wiki/Duplicate-Schedule-Detection)).
* Suggest a slot: `GET /schedule/suggest` recommends the least-loaded slot
  for a new job from the fleet's real fires
  ([Suggest a Slot](https://github.com/ptweezy/cronstable/wiki/Suggest-a-Slot)).
* Why didn't it run: `GET /schedule/why?job=<name>&at=<timestamp>`
  decomposes the scheduler's own match test field by field for one job and
  one instant
  ([Why Didn't It Run?](https://github.com/ptweezy/cronstable/wiki/Why-No-Run)).

#### Second-level schedules

Schedules are minute-granular by default, but cronstable can also run jobs at
**second granularity**. There are two equivalent spellings:

* a full **seven-field** crontab string, where the first field is the second
  (`second minute hour dayOfMonth month dayOfWeek year`); or
* the object form with a `second:` property.

Both of the jobs below run every 15 seconds (at seconds 0, 15, 30 and 45 of
every minute):

```yaml
jobs:
  - name: every-15s-string
    command: echo "tick"
    schedule: "*/15 * * * * * *"   # 7 fields: the leading field is seconds
  - name: every-15s-object
    command: echo "tick"
    schedule:
      second: "*/15"
```

The second field accepts the same syntax as the others (`*`, `*/5`, `0,30`,
`10-20`, ...). `second: "*"` (or `* * * * * * *`) fires every second.

While any enabled job specifies seconds, the scheduler wakes once per second
instead of once per minute; minute-granular jobs are unaffected and still fire
exactly once in their scheduled minute. If no job uses seconds, cronstable keeps
its original once-a-minute cadence, so there is no overhead for the common case.

Second-level scheduling is a YAML feature: [classic crontab files](#classic-crontab-files)
keep their standard five-field, minute-granular format. (A **six-field** string
is read as the classic five fields plus a trailing `year` column, *not* as
seconds; seconds require the full seven fields.)

For a runnable end-to-end example, see
[`example/pulse-monitor`](example/pulse-monitor), a small real-time uptime / SLA
monitor that probes a service every few seconds
(`docker compose -f example/pulse-monitor/docker-compose.yml up`), and its clustered sibling
[`example/pulse-cluster`](example/pulse-cluster), which fans the probes across a
three-node leader-electing cluster
(`docker compose -f example/pulse-cluster/docker-compose.yml up`).

Important: by default all time is interpreted to be in UTC, but you can
request to use local time instead.  For instance, the cron job below runs
every day at 19h27 *local time* because of the `utc: false` option:

```yaml
jobs:
  - name: test-01
    command: echo "hello"
    schedule: "27 19 * * *"
    utc: false
    captureStdout: true
```

You can also request that the schedule be
interpreted in an arbitrary timezone, using the `timezone` attribute:

```yaml
jobs:
  - name: test-01
    command: echo "hello"
    schedule: "27 19 * * *"
    timezone: America/Los_Angeles
    captureStdout: true
```

You can ask for environment variables to be defined for command execution:

```yaml
jobs:
  - name: test-01
    command: echo "foobar"
    shell: /bin/bash
    schedule: "*/5 * * * *"
    environment:
      - key: PATH
        value: /bin:/usr/bin
```

You can also provide an environment file to define environments for command execution:

```yaml
jobs:
  - name: test-01
    command: echo "foobar"
    shell: /bin/bash
    schedule: "*/5 * * * *"
    env_file: .env
```

The env file must be a list of `KEY=VALUE` pairs. Empty lines and lines starting with `#` will be ignored.

Variables declared in the `environment` option will override those found in the `env_file`.

### Classic crontab files

Already have a crontab?  cronstable runs it as-is.  A file named `*.crontab`,
`*.cron`, or just `crontab` (so `-c /etc/crontab` works) is read in the
classic Vixie format, whether passed directly to `-c`, dropped into a config
directory next to YAML files, or pulled in with `include:`:

```crontab
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin

# m h dom mon dow command
*/15 * * * *  /usr/local/bin/backup --incremental
30 4 * * mon-fri  /usr/local/bin/report --daily
@daily  /usr/local/bin/rotate-logs
0 0 * * *  pg_dump mydb > /backup/mydb-$(date +\%F).sql
```

Comments, `NAME=value` environment lines (position-sensitive, `SHELL` and
`CRON_TZ` honored), the `@reboot`/`@daily`/... nicknames, and `\%` escapes
all work as in `man 5 crontab`.  Each entry becomes an ordinary cronstable job
named `<file>:<line>`, configured to cronstable's standard defaults rather than
an emulation of cron's environment: schedules run in **UTC** unless the
crontab sets `CRON_TZ`, failure means a non-zero exit or stderr output (no
`MAILTO` mail), and the `%`-as-stdin feature is a load-time error instead of
a silent surprise (`\%` still gives a literal `%`).  When an entry needs
retries, reporting, timeouts, or any other per-job option, move it to YAML.
The full mapping and every deviation are documented in the
[Classic Crontabs](https://github.com/ptweezy/cronstable/wiki/Classic-Crontabs)
wiki page, and a runnable example (a config directory mixing a crontab with
YAML and the dashboard) lives in [example/crontab](example/crontab).

### Specifying defaults

There can be a special `defaults` section in the config.  Any attributes
defined in this section provide default values for cron jobs to inherit.
Although cron jobs can still override the defaults, as needed:

```yaml
defaults:
    environment:
      - key: PATH
        value: /bin:/usr/bin
    shell: /bin/bash
    utc: false
jobs:
  - name: test-01
    command: echo "foobar"  # runs with /bin/bash as shell
    schedule: "*/5 * * * *"
  - name: test-02  # runs with /bin/sh as shell
    command: echo "zbr"
    shell: /bin/sh
    schedule: "*/5 * * * *"
```

Note: if the configuration option is a directory and there are multiple configuration files in that directory, then the `defaults` section in each configuration file provides default options only for cron jobs inside that same file; the defaults have no effect beyond any individual YAML file.

### Reporting

cronstable has five built-in reporters: `sentry`, `mail`, `shell`, `webhook`
(Slack-compatible out of the box), and `push`
([end-to-end encrypted push notifications](#push-notifications), below). Each
can fire on the `onFailure`, `onPermanentFailure`, `onSuccess`, and `onLate`
hooks; the mail `subject`/`body` and sentry `body` are jinja2 templates over
the run's outcome and captured output, and secrets (DSNs, passwords, webhook
URLs) can come from `value`, `fromFile`, or `fromEnvVar`:

```yaml
- name: test-01
  command: |
    echo "hello" 1>&2
    exit 10
  schedule:
    minute: "*/2"
  captureStderr: true
  onFailure:
    report:
      sentry:
        dsn:
          fromEnvVar: SENTRY_DSN
      mail:
        from: example@foo.com
        to: example@bar.com
        smtpHost: 127.0.0.1
        subject: Cron job '{{name}}' failed
        body: |
          {{stderr}}
          (exit code: {{exit_code}})
      shell:
        shell: /bin/bash
        command: echo "Error code $CRONSTABLE_RETCODE"
      webhook:
        url:
          fromEnvVar: SLACK_WEBHOOK_URL
```

A report includes the output streams the job captures (`captureStderr` is on
by default, `captureStdout` off; see
[Output Capturing](https://github.com/ptweezy/cronstable/wiki/Output-Capturing)
for the capture options, including the `streamPrefix` line prefix). The
[Reporting](https://github.com/ptweezy/cronstable/wiki/Reporting) wiki page
documents every reporter's options (HTML mail, sentry fingerprints, webhook
method/headers/body and per-service examples), the template variables, and
the shell reporter's `CRONSTABLE_*` environment.

### Push notifications

A fifth reporter, `push`, delivers end-to-end encrypted alerts to paired
devices. Each alert is sealed to the device's X25519 public key (a libsodium
sealed box) before it leaves the daemon; the hosted relay that forwards it to
the platform push service (APNs) sees only ciphertext and routing metadata,
never job names, hostnames, or log lines. It needs the `push` extra
(`pip install "cronstable[push]"`), a daemon-global `push:` section, and an
opt-in on the reporting hooks; a config that enables push without any of
those refuses to start rather than silently not alerting:

```yaml
push:
  relay:
    url: https://relay.example.net/v1/notify
  devicesFile: /var/lib/cronstable/devices.json

defaults:
  onFailure:
    report:
      push:
        enabled: true
```

(With a `state:` section configured, `devicesFile` can be dropped: pairings
ride the durable store and are visible to every node sharing it.)

Pair a device from the dashboard ("Pair a device" in the command palette or
settings, a QR scan) or with one call:

[![The dashboard's Pair a device panel: a QR code of the connection payload, the same JSON as a copyable string, and a warning that the embedded token holds every scope](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-pair.png)](https://raw.githubusercontent.com/ptweezy/cronstable/develop/docs/img/dashboard-pair.png)

```shell
$ curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"name": "my-iphone", "platform": "ios", "publicKey": "<base64 X25519 key>", "pushToken": "<device push token>"}' \
    http://127.0.0.1:8080/push/devices
```

Setting `web.bonjour: true` (with the `discovery` extra installed)
additionally advertises the web API as a `_cronstable._tcp` mDNS service on
the local network, so a companion app finds the daemon without a typed URL;
see the
[LAN Discovery](https://github.com/ptweezy/cronstable/wiki/LAN-Discovery)
wiki page.

See
[Push Notifications](https://github.com/ptweezy/cronstable/wiki/Push-Notifications)
in the wiki for the report options, pairing and revocation, storage, size
limits, and the relay trust model.

### Metrics

Cronstable natively exposes Prometheus metrics whenever the
[HTTP REST API](https://github.com/ptweezy/cronstable/wiki/HTTP-API) is enabled;
no exporter sidecar needed:

```yaml
web:
  listen:
    - http://127.0.0.1:8080
```

`GET /metrics` then serves job run outcomes, duration histograms, retries,
next-run times, config-reload health, and cluster/leader-election state, in
both the Prometheus text format and OpenMetrics. See
[Metrics with Prometheus](https://github.com/ptweezy/cronstable/wiki/Metrics-with-Prometheus)
for the full metric reference, scrape configuration, and example alert rules.

Cronstable also has builtin support for pushing per-job metrics to
[Statsd](https://github.com/etsy/statsd):

```yaml
jobs:
  - name: test01
    command: echo "hello"
    schedule: "* * * * *"
    statsd:
      host: my-statsd.example.com
      port: 8125
      prefix: my.cron.jobs.prefix.test01
```

With this config Cronstable will write the following metrics over UDP
to the Statsd listening on `my-statsd.example.com:8125`:

```text
my.cron.jobs.prefix.test01.start:1|g  # this one is sent when the job starts
my.cron.jobs.prefix.test01.stop:1|g   # the rest are sent when the job stops
my.cron.jobs.prefix.test01.success:1|g
my.cron.jobs.prefix.test01.duration:3|ms
```

### Resource monitoring

Ever wondered which cron job is eating the box?! Turn on per-job resource
accounting with a single flag (or once under `defaults:` for every job):

```yaml
jobs:
  - name: nightly-model-refresh
    command: python -m models.refresh
    schedule: "0 4 * * *"
    monitorResources: true
```

While the job runs, cronstable samples its **whole process tree** (children and
shell-outs included) with [psutil](https://github.com/giampaolo/psutil), and
the run ends with its **total CPU time (user + system)** and **peak resident
memory**. The numbers surface everywhere the run does:

* **live** on the dashboard job row and drawer while it runs (`cpu 61% · 288 MiB`);
* per run and aggregated (avg/max CPU, peak memory) in the dashboard
  **History** tab and `GET /jobs/{name}/runs`;
* as **CPU/memory charts** in the dashboard's **Resources** tab (a live
  view of the running instance, the recorded profile of any recent run, and
  per-run trend strips), plus a node-wide history chart behind the header
  meter (`GET /jobs/{name}/resources`, `GET /node/history`);
* as Prometheus families on `GET /metrics`
  (`cronstable_job_cpu_seconds_total`, `cronstable_job_last_run_max_rss_bytes`, ...)
  and over [statsd](#metrics) when the job has a sink;
* in the durable run record's `resources` object when a
  [state store](https://github.com/ptweezy/cronstable/wiki/Durable-State) is
  configured, so it survives restarts;
* in report templates (`cpu_seconds` / `max_rss_bytes`) and the shell
  reporter's environment (`CRONSTABLE_CPU_SECONDS` / `CRONSTABLE_MAX_RSS_BYTES`),
  so a failure page can say how big the run was when it died.

It is observability only (it never changes a run's verdict), it is off by
default with zero overhead when off, and the numbers are sampled, so
short-lived runs are approximate while the long, heavy runs that matter are
sampled many times. The map form tunes the sampling cadence and how many
chart points each run keeps (`monitorResources: { interval: 0.5, history:
240 }`); series are downsampled in place so even a days-long run stays a few
KB. DAG tasks accept the same flag; their usage lands in the
task record of the `dag_run` document. On a cluster,
`cluster.observability` additionally shares each node's **whole-host**
CPU/memory so the dashboard's cluster panel and fleet view show where the
load actually is. The full semantics live in the
[Configuration Reference](https://github.com/ptweezy/cronstable/wiki/Configuration-Reference).

### Handling failure

By default, cronstable considers a job *failed* if the process exits non-zero
or writes to standard error (with stderr capturing enabled). The `failsWhen`
option tunes this per job with four booleans: `producesStdout` (default
false), `producesStderr` (default true), `nonzeroReturn` (default true), and
`always` (default false).

A `retry` option inside `onFailure` retries failing jobs with exponential
backoff, and `onPermanentFailure` reports only once all retries are
exhausted and cronstable gives up:

```yaml
- name: test-01
  command: |
    echo "hello" 1>&2
    exit 10
  schedule:
    minute: "*/10"
  captureStderr: true
  onFailure:
    retry:
      maximumRetries: 10
      initialDelay: 1
      maximumDelay: 30
      backoffMultiplier: 2
  onPermanentFailure:
    report:
      mail:
        from: example@foo.com
        to: example@bar.com
        smtpHost: 127.0.0.1
```

`maximumRetries: -1` retries forever, mostly useful with an `@reboot`
schedule to restart a long-running process when it fails. Retries are
in-memory by default (a daemon restart forgets an armed retry); with a
`state:` section configured they survive restarts and resume where they left
off. See
[Failure Detection and Retries](https://github.com/ptweezy/cronstable/wiki/Failure-Detection-and-Retries)
and [Durable State](https://github.com/ptweezy/cronstable/wiki/Durable-State)
in the wiki.

### Late-run detection (SLA monitoring)

Failure hooks only see runs that happened. An `sla:` block watches for the
runs that did not: each job can declare up to three independent thresholds,
evaluated once per minute by an in-process monitor, with a dedicated
`onLate` reporting hook that fires once when a threshold is breached and
takes the same `report` block (mail, Sentry, shell, webhook) as `onFailure`:

```yaml
- name: nightly-etl
  command: python -m etl.run
  schedule: "0 4 * * *"
  sla:
    maxTimeSinceSuccessSeconds: 129600   # no success for 36h
    lateAfterSeconds: 900                # a due slot not started within 15min
    maxRuntimeSeconds: 7200              # a run still going after 2h
  onLate:
    report:
      webhook:
        url:
          fromEnvVar: SLACK_WEBHOOK_URL
```

Breaches latch: one report per breach, not one per minute, with a recovery
log line and no report when the check clears. `maxRuntimeSeconds` observes
and never kills (use `executionTimeout` to enforce a limit); paused and
disabled jobs are excused; under leader election only the job's owning node
evaluates, so one breach pages once. Breaches surface as an OVERDUE badge in
both dashboards, an `sla` object on `GET /jobs`, and
`cronstable_job_late{job_name, check}` /
`cronstable_job_sla_breaches_total{job_name, check}` in the metrics. The
monitor runs inside the daemon and cannot report its own death, so pair it
with an external Prometheus staleness alert. See the
[Late-Run Detection](https://github.com/ptweezy/cronstable/wiki/Late-Run-Detection)
wiki page.

### Concurrency

Sometimes it may happen that a cron job takes so long to execute that when the moment its next scheduled execution is reached a previous instance may still be running.  How cronstable handles this situation is controlled by the option `concurrencyPolicy`, which takes one of the following values:

Allow
: allows concurrently running jobs (default)

Forbid
: forbids concurrent runs, skipping next run if previous hasn't finished yet

Replace
: cancels currently running job and replaces it with a new one

### Execution timeout

If you have a cron job that may possibly hang sometimes, you can instruct cronstable
to terminate the process after N seconds if it's still running by then, via the
`executionTimeout` option.  For example, the following cron job takes 2
seconds to complete, cronstable will terminate it after 1 second:

```yaml
- name: test-03
  command: |
    echo "starting..."
    sleep 2
    echo "all done."
  schedule:
    minute: "*"
  captureStderr: true
  executionTimeout: 1  # in seconds
```

When terminating a job, it is always a good idea to give that job process some
time to terminate properly.  For example, it may have opened a file, and even if
you tell it to shutdown, the process may need a few seconds to flush buffers and
avoid losing data.

On the other hand, there are times when programs are buggy and simply get stuck,
refusing to terminate nicely no matter what.  For this reason, cronstable always
checks if a process exited some time after being asked to do so. If it hasn't,
it tries to forcefully kill the process.  The option `killTimeout` option
indicates how many seconds to wait for the process to gracefully terminate
before killing it more forcefully.  In Unix systems, we first send a SIGTERM,
but if the process doesn't exit after `killTimeout` seconds (30 by default)
then we send SIGKILL.  For example, this cron job ignores SIGTERM, and so cronstable
will send it a SIGKILL after half a second:

```yaml
- name: test-03
  command: |
    trap "echo '(ignoring SIGTERM)'" TERM
    echo "starting..."
    sleep 10
    echo "all done."
  schedule:
    minute: "*"
  captureStderr: true
  executionTimeout: 1
  killTimeout: 0.5
```

### Change to another user/group

You can request that Cronstable change to another user and/or group for a specific
cron job.  The field `user` indicates the user (uid or userame) under which
the subprocess must be executed.  The field `group` (gid or group name)
indicates the group id.  If only `user` is given, the group defaults to the
main group of that user.  Example:

```yaml
- name: test-03
  command: id
  schedule:
    minute: "*"
  captureStderr: true
  user: www-data
```

Naturally, cronstable must be running as root in order to have permissions to
change to another user.

This feature is POSIX-only (it relies on `setuid`/`setgid`). On Windows, a job
with `user` or `group` set is rejected with a configuration error; see
[Running on Windows](#running-on-windows).

### Remote web/HTTP interface

If you wish to remotely control cronstable, you can optionally enable an HTTP REST
interface, with the following configuration (example):

```yaml
web:
  listen:
     - http://127.0.0.1:8080
     - unix:///tmp/cronstable.sock
```

With the web interface enabled, cronstable also serves the
[web dashboard](#web-dashboard) at the root path (`/`) of any `http://`
listener; set `ui: false` to expose only the REST API. With `web.authToken`
set, the dashboard page loads without a token, then prompts for one and
stores it only in that browser tab. See the
[full dashboard tour](https://github.com/ptweezy/cronstable/wiki/Web-Dashboard)
in the wiki.

The API covers the daemon (version, status, summary, metrics, job-set id),
jobs (start, cancel, pause and resume, run history, live SSE log tails,
resources), schedules (preview, pressure, duplicates, suggest, why), DAGs,
the durable state store, push-device pairing, the cluster and fleet views,
and an iCal feed of upcoming fires. For example, pausing a job for a
two-hour maintenance window (HTTPie shown):

```shell
$ http post http://127.0.0.1:8080/jobs/test-02/pause durationSeconds:=7200 note="db migration"
HTTP/1.1 200 OK

{"paused": {"since": "2026-07-19T14:00:00+00:00", "until": "2026-07-19T16:00:00+00:00", "note": "db migration", "by": "api", "channel": "api"}}
```

Every endpoint, with request and response shapes, is documented in the
[HTTP API](https://github.com/ptweezy/cronstable/wiki/HTTP-API) reference in
the wiki; the repo also ships a machine-readable
[OpenAPI specification](docs/openapi.yaml).

#### Serving the API over TLS

`web.listen` also accepts `https://` addresses, served from a `web.tls` block.
Each entry keeps its own transport, so one runner can serve the same API and
dashboard in plaintext on loopback and over TLS on a routable interface;
`unix://` listeners are always plaintext, where the socket's own permissions
(`socketMode`) are the access control.

```yaml
web:
  listen:
     - http://127.0.0.1:8080                      # loopback, plaintext
     - https://0.0.0.0:8443                       # served with the material below
  tls:
    cert: /etc/cronstable/web.pem
    key:  /etc/cronstable/web.key
    clientCa: /etc/cronstable/callers-ca.pem      # optional: require client certs
```

`clientCa` turns the listener into mutual TLS, web certificates rotate in
place without a daemon restart, and the clients (`cronstable tui`,
`cronstable mcp`) take matching `--cacert` / `--client-cert` /
`--client-key` / `--insecure` flags. This is a teaser: issuing the
certificates, the mTLS trust model and how it interacts with
`web.authToken`, the rotation mechanics and what they do not cover, the job
state API's trust anchor, and the full client flag surface are covered in
depth in the
[Listener TLS](https://github.com/ptweezy/cronstable/wiki/Listener-TLS)
guide in the wiki.

### Job-set id

The **job-set id** is an order-independent fingerprint of the set of jobs a
cronstable instance is running. Two instances produce the *same* id if and only if
they hold the same set of jobs, which lets several replicas deployed from the
same configuration confirm they are running the same thing, or detect that one
has drifted from the others.

The id is taken over the *effective* (post-merge) configuration of every job,
which gives it some useful properties:

* it is **independent of job order**, and of whether a setting was written
  inline on each job or hoisted into a `defaults` block;
* **equivalent schedule spellings match**: the `minute:`/`hour:` object form
  fingerprints the same as the equivalent five-field crontab string;
* it covers **every behavior-affecting field** (command, schedule, shell, the
  *names* of `environment` variables, capture flags, `failsWhen`,
  retry/reporting policy, timezone, `enabled`, and so on), so any meaningful
  change to a job changes the id;
* `user`/`group` are fingerprinted **as configured** (e.g. `www-data`), not as
  the resolved numeric uid/gid, which can differ host to host;
* **secret/value material is never embedded**: inline reporting secrets
  (Sentry DSN, mail password, webhook URL and header values) are redacted,
  and only the *names* of
  `environment` variables are hashed, not their values (env commonly holds
  secrets, and a per-host value, e.g. from `env_file`, would otherwise make
  identical configs differ across hosts). The id is safe to log and serve, and
  rotating a secret or changing an env value does not change it.

Because it reflects *effective* config, it also reflects platform-dependent
defaults (the default `shell` is `/bin/sh` on POSIX, `cmd.exe` on Windows), so
compare instances running on the same platform, which replicas are. The scheme
is versioned with a `v1:` prefix; ids are only comparable within a scheme
version.

It is available three ways:

* **CLI**: print it and exit (handy in scripts / health checks):

  ```shell
  $ cronstable -c /etc/cronstable.d --job-set-id
  v1:b834d7565aee0da50cd017f666651a5ba3b2e6b161daf0cb6e430f23f51ce90b
  ```

* **HTTP**: `GET /job-set-id` on the [web interface](#remote-webhttp-interface)
  (also `application/json`), and shown in the dashboard header:

  ```shell
  $ http get http://127.0.0.1:8080/job-set-id
  v1:b834d7565aee0da50cd017f666651a5ba3b2e6b161daf0cb6e430f23f51ce90b

  $ http get http://127.0.0.1:8080/job-set-id Accept:application/json
  {"job_set_id": "v1:b834d7…51ce90b", "jobs": 3}
  ```

* **Logs**: it is logged once at startup, and again whenever a config reload
  changes it.

### Clustering and leader election

By default cronstable runs as a single instance and every replica runs every job.
An optional `cluster` section lets several replicas coordinate: each node serves
a small `GET /peer` endpoint over **mutual TLS** and periodically polls its
configured peers, comparing [job-set ids](#job-set-id) so they can confirm they
are running the *same* set of jobs (cluster peer attestation). Turning on
`electLeader` promotes that same attestation into a **quorum-gated leader
election**, so you can run more than one replica from one config without
double-running scheduled jobs:

```yaml
cluster:
  listen: "0.0.0.0:8443"          # the mTLS listener for this node
  tls:
    ca:   /etc/cronstable/cluster-ca.pem   # trust anchor for peer certificates
    cert: /etc/cronstable/this-node.pem    # this node's certificate
    key:  /etc/cronstable/this-node.key
  peers:
    - host: cronstable-b.internal:8443
    - host: cronstable-c.internal:8443
  nodeName: cronstable-a              # optional; defaults to the system hostname
  interval: 30                    # optional; seconds per round (default 30)
  connectTimeout: 10              # optional; per-peer connect timeout (default 10)
  driftAfter: 3                   # optional; rounds before "drifted" (default 3)
  electLeader: true               # observe-only if false (the default)
```

Each node independently elects, as leader, the lowest `nodeName` among the
members it currently sees agreeing on the job-set id, but only if that set is a
**quorum** (a strict majority) of the cluster, so under a clean partition at
most one side leads. This is best-effort (the default `gossip` backend keeps no
shared state); for a fenced, exactly-once guarantee set
`cluster.backend: kubernetes` or `cluster.backend: etcd`
to elect through a `coordination.k8s.io/v1` `Lease` or a lease-bound etcd key
instead.

Each job can override the cluster-wide default with a per-job `clusterPolicy`
(`Leader`, the default, may skip under a partition; `PreferLeader` never
skips but may double-run; `EveryNode` runs everywhere), picking its own
point on the liveness-vs-duplication trade-off.

The current view (members, elected leader, quorum, and any conflicts) is
available at `GET /cluster` and shown as a panel in the dashboard. This is a
teaser: the full trust model, per-peer status table, quorum math, sizing
guidance, `distribution: spread` load-balancing, and the fenced lease backends
are all covered in depth in the
[Clustering and Leader Election](https://github.com/ptweezy/cronstable/wiki/Clustering-and-Leader-Election)
guide in the wiki. To watch it live, see [Try it](#web-dashboard) below.

### Includes

You may have a use case where it's convenient to have multiple config files,
and choose at runtime which one to use.  In that case, it might be useful if
you can put common definitions (such as defaults for reporting, shell, etc.)
in a separate file, that is included by the other files.

To support this use case, it is possible to ask one config file to include
another one, via the `include` directive.  It takes a list of file names:
those files will be parsed as configuration and merged in with this file.

Example, your main config file could be:

```yaml
include:
  - _inc.yaml

jobs:

  - name: my job
    ...
```

And your included `_inc.yaml` file could contain some useful defaults:

```yaml
defaults:
  shell: /bin/bash
  onPermanentFailure:
    report:
      sentry:
        ...
```

### Environment variable interpolation

Any string value in the config can pull from cronstable's environment with
`${VAR}`, or `${VAR:-default}` for a fallback, so one config file serves many
environments without a wrapper script templating it. Write `$$` for a literal
`$`. Interpolation runs after the file is validated, so it reaches any
string-typed field (a listen address, a state path, a timezone, a webhook URL),
and a `${VAR}` that is unset and has no default is a hard configuration error
that names the variable, caught by `cronstable --validate-config`.

```yaml
web:
  listen:
    - "0.0.0.0:${WEB_PORT:-8080}"   # port from the environment, default 8080
state:
  path: ${STATE_DIR}               # required: unset fails --validate-config
jobs:
  - name: rollup-${REGION}
    command: run-rollup             # ${VAR} in a command is left for the shell
    schedule:
      minute: "0"
    timezone: ${TZ:-UTC}
```

A job's (and reporter's) `command` and `shell` are deliberately left untouched,
so their `${VAR}` is expanded by the runtime shell against the job's own
environment, not the daemon's; the `logging` section is likewise left for
Python's `logging.config`. See
[Environment-Variable Interpolation](https://github.com/ptweezy/cronstable/wiki/Environment-Variable-Interpolation)
for the full rules, including how it affects the [job-set id](#job-set-id).

### Custom logging

It's possible to provide a custom logging configuration, via the `logging`
configuration section.  For example, the following configuration displays log lines with
an embedded timestamp for each message.

```yaml
logging:
  # In the format of:
  # https://docs.python.org/3/library/logging.config.html#dictionary-schema-details
  version: 1
  disable_existing_loggers: false
  formatters:
    simple:
      format: '%(asctime)s [%(processName)s/%(threadName)s] %(levelname)s (%(name)s): %(message)s'
      datefmt: '%Y-%m-%d %H:%M:%S'
  handlers:
    console:
      class: logging.StreamHandler
      level: DEBUG
      formatter: simple
      stream: ext://sys.stdout
  root:
    level: INFO
    handlers:
      - console
```

### Obscure configuration options

#### enabled: true|false (default true)

It is possible to disable a specific cron job by adding a `enabled: false` option.  Jobs
with `enabled: false` will simply be skipped, as if they aren't there, apart from
validating the configuration.

```yaml
jobs:
  - name: test-01
    enabled: false  # this cron job will not run until you change this to `true`
    command: echo "foobar"
    shell: /bin/bash
    schedule: "* * * * *"
```

## Performance

cronstable is built to run on small and old machines, and CI holds it to
that: every commit runs an exhaustive benchmark suite (startup time, schedule
computation for 100,000 jobs, config parsing, DAG planning, durable-state
I/O, memory footprint, about 37 metrics in all) paired against the latest
release on the same runner. A release that regresses a metric past its
declared limit does not ship, and every release page carries a chart and a
full table of the change against the previous release.

Run the suite yourself with `python benchmarks/bench.py --quick`; see
[Performance Benchmarks](https://github.com/ptweezy/cronstable/wiki/Performance-Benchmarks)
for how the comparison and the gate work.

## Documentation map

Every feature has its own page in the
[wiki](https://github.com/ptweezy/cronstable/wiki); the sidebar there is the
full index. Good starting points:
[Installation](https://github.com/ptweezy/cronstable/wiki/Installation),
the [Configuration Reference](https://github.com/ptweezy/cronstable/wiki/Configuration-Reference),
the [Web Dashboard tour](https://github.com/ptweezy/cronstable/wiki/Web-Dashboard),
and [Troubleshooting](https://github.com/ptweezy/cronstable/wiki/Troubleshooting).

## Contributing and license

Bug reports, feature ideas, and pull requests are welcome; see
[CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, how to sign off
your commits (DCO), and
[Contributing and Releasing](https://github.com/ptweezy/cronstable/wiki/Contributing-and-Releasing)
for how releases work. cronstable is [MIT-licensed](LICENSE); see
[LICENSING.md](LICENSING.md) for how the repository's licensing is organized.

**Security.** Please report vulnerabilities privately rather than in a public
issue; [SECURITY.md](SECURITY.md) has the disclosure process, what is in scope
(including the hosted relay and the public demo), and what to expect.

**Trademarks.** The MIT License covers the code, not the brand. cronstable™ and
the cronstable logo are trademarks of Parker Loflin; see
[TRADEMARKS.md](TRADEMARKS.md). The rendered logo artwork is also reserved
rather than MIT-granted, while the code that draws it stays MIT; see
[Brand assets](LICENSING.md#brand-assets).

cronstable is a fork of [yacron](https://github.com/gjcarneiro/yacron) (by Gustavo Carneiro), continuing development from version 0.19.
