# Metrics with Prometheus

cronstable natively exposes Prometheus metrics at `GET /metrics` on the [HTTP web API](HTTP-API). This is the pull-side sibling of the per-job [statsd push metrics](Metrics-with-Statsd): a Prometheus server scrapes cumulative counters (run outcomes, durations, retries), live gauges (running instances, next scheduled run, last-run outcome), and the full [cluster health surface](Clustering-and-Leader-Election) on demand. The exposition is generated by cronstable itself -- no exporter sidecar and no extra dependency, so it works on every architecture and platform cronstable ships for, including Windows.

Both exposition formats are supported: the classic Prometheus text format (`text/plain; version=0.0.4`) by default, and [OpenMetrics 1.0](https://openmetrics.io/) (`application/openmetrics-text`) when the scraper advertises it in its `Accept` header, as modern Prometheus servers do.

## Enabling the endpoint

The endpoint is on by default whenever the [web API](HTTP-API) is enabled; a `web.listen` block is all it takes:

```yaml
web:
  listen:
    - http://127.0.0.1:8080
```

```console
$ curl http://127.0.0.1:8080/metrics
# HELP cronstable_info cronstable build information.
# TYPE cronstable_info gauge
cronstable_info{version="1.0.13"} 1
...
```

The `web.metrics` option tunes or disables it. It accepts a boolean shorthand or a map:

```yaml
web:
  listen:
    - http://127.0.0.1:8080
  # metrics: false            # shorthand: disable the endpoint entirely
  metrics:
    enabled: true             # default
    public: false             # default; see Authentication below
    durationBuckets:          # histogram bounds override (seconds)
      - 1
      - 10
      - 60
      - 600
```

### Options

The schema for the block is `Bool() | Map({Opt("enabled"): Bool(), Opt("public"): Bool(), Opt("durationBuckets"): Seq(Float())})` -- every key is optional.

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Serve `GET /metrics`. `metrics: false` / `metrics: true` are shorthands for `{enabled: ...}`. |
| `public` | bool | `false` | Exempt `/metrics` (and only `/metrics`) from `web.authToken` bearer-token authentication, for scrapers that cannot send credentials. |
| `durationBuckets` | list of float | `0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600` | Upper bounds (seconds) of the `cronstable_job_duration_seconds` histogram. Must be finite, positive and strictly increasing; a `+Inf` bucket is always appended automatically. Changing the bounds on a reload restarts the histograms from zero (an ordinary counter reset to Prometheus). |

## Scraping and authentication

When [`web.authToken`](HTTP-API#authentication) is configured, `/metrics` requires the bearer token like every other data endpoint. Prometheus supports this directly in its scrape configuration:

```yaml
scrape_configs:
  - job_name: cronstable
    static_configs:
      - targets: ["cron-host:8080"]
    authorization:
      type: Bearer
      credentials: "the-secret-token"
```

If your scraper cannot send credentials, set `web.metrics.public: true` to exempt `/metrics` from authentication while everything else stays gated. The metrics reveal job names, schedules and outcomes (but never command lines, output, or secrets); treat `public` accordingly.

Two response details:

- `web.headers` are stamped onto the response as usual, **except** `Content-Type`: the exposition format's contract always wins.
- Content negotiation is on the `Accept` header: any request advertising `application/openmetrics-text` gets OpenMetrics (terminated by `# EOF`); everything else gets the text format.

## Metric reference

All metrics are prefixed `cronstable_`. Per-job metrics carry the job name in a `job_name` label (not `job`, which would collide with Prometheus's own target label). Counters are cumulative since process start; they reset when the daemon restarts, which Prometheus's `rate()`/`increase()` handle natively (with a [durable state store](Durable-State) configured, the per-job counters are re-seeded across restarts; see [Semantics and guarantees](#semantics-and-guarantees)).

### Daemon

| Metric | Type | Description |
| --- | --- | --- |
| `cronstable_info{version}` | gauge (info) | Build information; always `1`. |
| `cronstable_start_time_seconds` | gauge | Unix time this cronstable process started. |
| `cronstable_job_set_info{job_set_id}` | gauge (info) | The [job-set fingerprint](Clustering-and-Leader-Election#the-job-set-id-foundation) of the currently loaded configuration; always `1`. Compare across a fleet to spot config drift. |
| `cronstable_jobs{state}` | gauge | Number of configured jobs, `state="enabled"` / `"disabled"`. |
| `cronstable_config_last_reload_successful` | gauge | `1` if the most recent configuration parse succeeded, `0` if the config on disk is currently broken (the daemon keeps running the previous good config). Absent until the first parse. |
| `cronstable_config_last_reload_success_timestamp_seconds` | gauge | Unix time of the last successful configuration parse. |

### Per job

| Metric | Type | Description |
| --- | --- | --- |
| `cronstable_job_runs_total{job_name, status}` | counter | Finished runs by outcome: `status="success"` / `"failure"` / `"cancelled"` / `"skipped"`, exactly as recorded in the [run history](HTTP-API#get-jobsnameruns). `skipped` counts the synthetic ledger rows a [paused](Pausing-Jobs) job writes in place of a launch; those stamp neither last-outcome timestamp. Zero-filled for every configured job from the first scrape. Runs replaced under `concurrencyPolicy: Replace` are not runs and are not counted, matching the dashboard. |
| `cronstable_job_duration_seconds` | histogram | Duration of finished runs (bucket bounds from `durationBuckets`). Cancelled-without-starting records carry no duration and are not observed. |
| `cronstable_job_retries_total{job_name}` | counter | Retry attempts actually launched (`onFailure.retry`); deferred or abandoned retries are not counted. |
| `cronstable_job_permanent_failures_total{job_name}` | counter | Failures with no retry remaining -- exactly when [`onPermanentFailure`](Failure-Detection-and-Retries) reporting fires. |
| `cronstable_job_start_failures_total{job_name}` | counter | Runs whose command could not be launched at all (also recorded as failures with the conventional exit code `127`). |
| `cronstable_job_last_success_timestamp_seconds{job_name}` | gauge | Unix time of the last successful finish. Absent until the first success. |
| `cronstable_job_last_failure_timestamp_seconds{job_name}` | gauge | Unix time of the last failed finish. Absent until the first failure. |
| `cronstable_job_info{job_name, schedule, cluster_policy}` | gauge (info) | Static configuration facts; always `1`. |
| `cronstable_job_enabled{job_name}` | gauge | `1` if the job is enabled in the loaded config. |
| `cronstable_job_running{job_name}` | gauge | Number of currently running instances (can exceed `1` under `concurrencyPolicy: Allow`). |
| `cronstable_job_next_run_timestamp_seconds{job_name}` | gauge | Unix time of the next scheduled run. Absent for disabled jobs and `@reboot` schedules. |
| `cronstable_job_last_run_timestamp_seconds{job_name}` | gauge | Unix time the most recent run finished. |
| `cronstable_job_last_run_duration_seconds{job_name}` | gauge | Duration of the most recent finished run. |
| `cronstable_job_last_run_exit_code{job_name}` | gauge | Exit code of the most recent finished run. |
| `cronstable_job_last_run_success{job_name}` | gauge | `1` if the most recent finished run succeeded, else `0` (a cancelled run counts as `0`). |
| `cronstable_job_cpu_seconds_total{job_name, mode}` | counter | CPU time consumed by [`monitorResources`](Configuration-Reference#metrics) runs, `mode="user"` / `"system"`, summed over the job's monitored runs. Emitted only once the job has a monitored run (an unmonitored job exports no CPU series). |
| `cronstable_job_peak_rss_bytes{job_name}` | gauge | Highest resident-set size (bytes) observed across all of the job's monitored runs. Absent until the first monitored run. |
| `cronstable_job_last_run_cpu_seconds{job_name}` | gauge | Total CPU time of the most recent monitored run. Absent unless that run had `monitorResources` on. |
| `cronstable_job_last_run_max_rss_bytes{job_name}` | gauge | Peak resident-set size (bytes) of the most recent monitored run. Absent unless that run had `monitorResources` on. |
| `cronstable_job_paused{job_name}` | gauge | `1` while the job is [paused at runtime](Pausing-Jobs), else `0`. |
| `cronstable_job_late{job_name, check}` | gauge | `1` while the job's [SLA check](Late-Run-Detection) is breached, per `check` (`maxTimeSinceSuccess` / `lateAfter` / `maxRuntime`). Emitted only for checks the job configures in its `sla:` block, from the monitor's first evaluation; paused jobs are exempt. |
| `cronstable_job_sla_breaches_total{job_name, check}` | counter | [SLA breaches](Late-Run-Detection) latched for the job, per `check`; increments once per breach, not per evaluation. Emitted alongside `cronstable_job_late`, zero-filled from the first evaluation. |

The four resource families above require the job to opt into
[`monitorResources`](Configuration-Reference#metrics); they are sampled and
best-effort -- see [resource monitoring](Resource-Monitoring) for the sampling
model and every other surface the numbers reach. Like the run counters,
`cronstable_job_cpu_seconds_total` and
`cronstable_job_peak_rss_bytes` are made restart-durable by a `state:` store (see
[Semantics and guarantees](#semantics-and-guarantees)).

### State backend

Emitted when a [durable state store](Durable-State) is configured (a `state:` section); without one cronstable is stateless and these families are absent. They observe the store itself -- the job families above are unchanged, except that the store makes the per-job counters restart-durable (see [Semantics and guarantees](#semantics-and-guarantees)). Families and label values appear as they first become relevant, so a store that never contends a lock or is never throttled does not export frozen zero series forever.

| Metric | Type | Description |
| --- | --- | --- |
| `cronstable_state_info{backend, topology}` | gauge (info) | Static store facts: the backend implementation (`filesystem`) and the effective topology (`single-node` / `shared`, after `topology: auto` resolves); always `1`. |
| `cronstable_state_ops_total{op}` | counter | Store operations by kind: `op="append"` / `"list"` / `"derive-max"` / `"prune"` / `"lease-acquire"` / `"lease-renew"` / `"lease-release"` / `"lease-read"` / `"start"` / `"gc"`. Each `op` value appears once that operation has first run. |
| `cronstable_state_op_errors_total{op}` | counter | Store operations that raised, same `op` label. |
| `cronstable_state_op_seconds_total{op}` | counter | Seconds spent inside store operations; divide by `cronstable_state_ops_total` (e.g. `rate()` over `rate()`) for mean in-store latency per operation. |
| `cronstable_state_lock_acquisitions_total` | counter | Advisory-lock acquisitions by the store. Emitted once nonzero. |
| `cronstable_state_lock_wait_seconds_total` | counter | Seconds spent waiting to acquire the store's advisory locks -- the lock-contention signal. Emitted alongside `cronstable_state_lock_acquisitions_total`. |
| `cronstable_state_throttled_ops_total` | counter | Store operations delayed by the `state.maxOpsPerSecond` token bucket. Emitted once nonzero. |
| `cronstable_state_throttle_wait_seconds_total` | counter | Seconds those operations spent queued behind the rate limit. Emitted alongside `cronstable_state_throttled_ops_total`. |
| `cronstable_state_dropped_writes_total{kind}` | counter | Durable writes that failed and were dropped (with a logged warning) -- writes are fire-and-forget under either `onStoreUnavailable` policy -- by `kind="run-record"` / `"checkpoint"` / `"retry"` / `"reboot-marker"` / `"counters"` / `"manifest"`. Emitted once nonzero -- the single most alertable state signal. |

If reading the store's stats fails during a scrape, the state families are omitted from that scrape (with the error logged) rather than failing it -- the same degradation contract as the cluster block below. `cronstable_state_dropped_writes_total` is the exception: it is accumulated by the scheduler as it drops writes, not read from the store, so it keeps being served even when the store itself cannot be read.

### Cluster

Emitted when a [cluster](Clustering-and-Leader-Election) is configured; the values mirror the pre-derived fields of [`GET /cluster`](HTTP-API#get-cluster), so the same alert rules documented there apply.

| Metric | Type | Description |
| --- | --- | --- |
| `cronstable_cluster_enabled` | gauge | `1` when a leadership backend is running on this node (`0` also covers a configured backend that failed to start). |
| `cronstable_cluster_info{backend, node_name, distribution}` | gauge (info) | Backend (`gossip`/`kubernetes`/`etcd`), this node's name, and the ownership distribution; always `1`. |
| `cronstable_cluster_size` / `cronstable_cluster_quorum` | gauge | Effective cluster size N and the quorum threshold (lease backends report `1`/`1`). |
| `cronstable_cluster_quorate` | gauge | Whether this node is part of a quorum -- the single most alertable cluster signal. |
| `cronstable_cluster_is_leader` | gauge | Whether this node holds scheduled-job leadership. Always `0` under `distribution: spread` (ownership is per job there), mirroring `GET /cluster`. |
| `cronstable_cluster_leader_info{leader}` | gauge (info) | The observed leader's name; absent when there is none. |
| `cronstable_cluster_conflict{kind}` | gauge | The three fail-closed conflict gates, `kind="nodename"` / `"size"` / `"policy"`. Any `1` means Leader jobs are standing down cluster-wide. |
| `cronstable_cluster_peers{status}` | gauge | Gossip backend only: configured peers by observed status (`agreed`, `syncing`, `drifted`, `unreachable`, `untrusted`, `conflict`, `self`, `unknown`), zero-filled so alert series never vanish. |
| `cronstable_cluster_leader_transitions_total` | counter | Times this node acquired or lost leadership. Observed at scheduler cadence (once per minute), so a flap shorter than a scheduler tick may be missed. Emitted only when `electLeader` is on (an observe-only cluster has no leadership to transition). |
| `cronstable_cluster_quorum_transitions_total` | counter | Times this node joined or left quorum (same cadence and `electLeader` caveats). |

If a backend read fails during a scrape, the cluster block degrades to `cronstable_cluster_enabled` alone (with the error logged by the `prometheus` logger) rather than failing the whole scrape.

## Semantics and guarantees

- **Metrics never affect scheduling.** Every hook is a synchronous in-memory increment on the scheduler's event loop; there is nothing to time out, retry, or crash on. This is the same contract the statsd writer has, without the UDP.
- **Counters agree with the API.** `cronstable_job_runs_total` increments at the exact point a run enters the run history (`GET /jobs/{name}/runs`), so the two surfaces can never disagree on outcomes.
- **Counters survive reloads -- and, with durable state, restarts.** The accumulators live on the daemon (not the web app), so a config reload -- even one that restarts the web server or the cluster manager -- keeps them. Without a `state:` section a process restart resets them, which is normal for Prometheus counters. With a [durable state store](Durable-State) configured, the per-job counters (runs by outcome, retries, permanent and start failures, the duration histogram, and the last success/failure timestamps) are snapshotted durably -- at most one write per 15 seconds while running, plus a final snapshot on clean shutdown -- and seeded back on boot for the jobs still in the config (histogram state only if the `durationBuckets` bounds are unchanged). A restart then reads as, at worst, a small ordinary counter reset covering the events since the last snapshot; a crash forfeits at most those last few seconds. `cronstable_start_time_seconds` still resets per process by design -- that is how you see the restart itself.
- **Removed jobs disappear.** On every successful reload, series for jobs no longer in the config are dropped; a renamed job starts over at zero.
- **`@reboot` keep-alives look "always running".** A long-running `@reboot` job shows `cronstable_job_running 1` with zero finished runs -- that is the correct reading, not missing data.

## Example alerts

```yaml
groups:
  - name: cronstable
    rules:
      # a job's most recent run failed
      - alert: YacronJobFailed
        expr: cronstable_job_last_run_success == 0
        for: 5m
      # a job that should run hourly has not succeeded for 2 hours
      - alert: YacronJobStale
        expr: time() - cronstable_job_last_success_timestamp_seconds > 7200
      # the config file on disk is broken; the daemon is running stale config
      - alert: YacronConfigBroken
        expr: cronstable_config_last_reload_successful == 0
        for: 10m
      # the cluster cannot elect a leader: Leader jobs are not running anywhere
      - alert: YacronClusterNotQuorate
        expr: cronstable_cluster_quorate == 0
        for: 5m
      # more than one node believes it is the leader (split-brain)
      - alert: YacronMultipleLeaders
        expr: sum(cronstable_cluster_is_leader) > 1
```

The [Clustering monitoring guide](Clustering-and-Leader-Election#monitoring-and-alerting) discusses the cluster signals in depth.

The staleness rule above (`YacronJobStale`) is the external backstop to pair with per-job [late-run detection](Late-Run-Detection): the in-process `sla:` monitor gives per-job thresholds and rich `onLate` notifications, but it dies with the daemon, while a Prometheus rule on `time() - cronstable_job_last_success_timestamp_seconds` (plus `up == 0` on the scrape itself) still fires when the daemon, its host, or its network is gone.

## See also

- [HTTP Control API](HTTP-API): the web server `/metrics` is part of -- listeners, `authToken`, headers, reload behavior.
- [Metrics with statsd](Metrics-with-Statsd): the push-side alternative; both can be enabled at once.
- [Resource Monitoring](Resource-Monitoring): the `monitorResources` sampling behind the per-job CPU/RSS families.
- [Clustering and Leader Election](Clustering-and-Leader-Election): what the cluster gauges mean and when they change.
- [Failure Detection and Retries](Failure-Detection-and-Retries): how run outcomes, retries, and permanent failures are determined.
- [Late-Run Detection](Late-Run-Detection): the `sla:` monitor behind `cronstable_job_late` and the breach counter.
- [Pausing Jobs](Pausing-Jobs): the runtime pause behind `cronstable_job_paused` and the `skipped` run outcome.
- [Configuration Reference](Configuration-Reference): the full `web` section.
