Metadata-Version: 2.5
Name: celery_redis_prometheus
Version: 2.0.0
Summary: Exports task execution metrics in Prometheus format
Project-URL: repository, https://github.com/ZeitOnline/celery_redis_prometheus
Author-email: DIE ZEIT <engineering@zeit.de>
License: BSD-3-Clause
License-File: LICENSE
Requires-Python: >=3.7
Requires-Dist: celery>=5.0.0.dev0
Requires-Dist: click
Requires-Dist: prometheus-client>=0.14.0
Description-Content-Type: text/markdown

# celery_redis_prometheus

Exports task execution metrics in Prometheus format: how many tasks were started
and have completed successfully or with failure, and how many tasks are still in
the queues (supported only for broker redis).

Inspired by <https://gitlab.com/kalibrr/celery-prometheus>


## Usage

### Start HTTP service

Start the HTTP server like this:

```
$ bin/celery prometheus --host=127.0.0.1 --port=9691
```


### Configure Prometheus

```
scrape_configs:
  - job_name: 'celery'
    static_configs:
      - targets: ['localhost:9691']
```

We export the following metrics:

* `celery_tasks_total{state="started|succeeded|failed|retried|retries-exceeded", queue="..."}`, counter
* `celery_task_queuetime_seconds{queue}`, histogram (only if `task_send_sent_event` is enabled in Celery)
* `celery_task_runtime_seconds{queue}`, histogram

If you pass `--queue-length`, the queue lengths are read from the broker on each scrape (NOTE: this only works with redis as the broker), resulting in this additional metric:

* `celery_queue_length{queue="..."}`, gauge

While the broker cannot be reached, `celery_queue_length` is left out.
So an alert like `celery_queue_length > 100` silently stops firing;
add one that fires when the metric is missing.
With a single exporter per alert selector, `absent()` does that:

```yaml
- alert: CeleryQueueLengthMissing
  expr: absent(celery_queue_length{namespace="myapp"})
  for: 5m
```

`absent()` only fires if *no* series matches the selector at all.
If the selector covers several exporters (e.g. one per broker), 
one of them failing goes unnoticed as long as another still reports.
Compare against the `up` series Prometheus records for each scrape target instead, 
which yields one alert per exporter that is down or cannot read its broker:

```yaml
- alert: CeleryQueueLengthMissing
  expr: up{job="myapp/celery-metrics"} unless on(namespace, pod) celery_queue_length
  for: 5m
```

The `up` selector must match only exporters started with `--queue-length`, and `on(...)`
must name labels that identify the exporter on both sides.

## Run tests

With `bin/test` wrapper script (or `uv run pytest`).


## celery_redis_prometheus changes

<!-- towncrier release notes start -->

## 2.0.0 (2026-10-12)

### 

- Give up reading queue lengths on a hanging broker after 3 seconds per connect, handshake and query (so about 9 seconds at worst), to still answer within the default 10 second scrape timeout. Configurable via `socket_timeout`/`socket_connect_timeout` in `broker_transport_options`
- Read queue lengths from the broker on each scrape instead of polling them in a background thread. While the broker cannot be reached, `celery_queue_length` is left out instead of keeping stale values; see the README on how to alert on that
- Replace `--queuelength-interval=x` with the flag `--queue-length`. The old option is rejected, so update the exporter arguments when upgrading


## 2.0.0 (unreleased)

- Nothing changed yet.


## 1.5.2 (2026-07-27)

- Remove unused `setuptools` dependency


## 1.5.1 (2023-10-27)

- Fix brown-bag release


## 1.5.0 (2023-10-27)

- Be explicit about unregistering default collectors


## 1.4.0 (2022-10-28)

- Add `queue` label to task_queuetime and task_runtime metrics


## 1.3.0 (2022-09-14)

- Add `queue` label to `celery_tasks_total` metric
- Add separate state `retries-exceeded` (previously used `failed`)


## 1.2.0 (2022-06-23)

- Update to celery-5.x


## 1.1.1 (2020-03-20)

- Fix py2 class syntax issues


## 1.1.0 (2020-03-20)

- Make py2 compatible.


## 1.0.0 (2020-03-20)

- Initial release
