Metadata-Version: 2.4
Name: prometheus-dcache-exporter
Version: 0.2.0
Summary: A Prometheus exporter for dCache.
Author-email: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
Maintainer-email: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
Project-URL: website, https://gitlab.com/calestyo/prometheus-dcache-exporter
Keywords: monitoring,prometheus,prometheus exporter,dcache
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: No Input/Output (Daemon)
Classifier: Environment :: Plugins
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking :: Monitoring
Requires-Python: >=3.13
Description-Content-Type: text/x-rst
License-File: LICENCE
Requires-Dist: certifi
Requires-Dist: httpx
Requires-Dist: humanfriendly
Requires-Dist: paramiko
Requires-Dist: prometheus-client
Requires-Dist: python-jsonpath
Requires-Dist: python-module-utils
Provides-Extra: extended-logging-formatter
Requires-Dist: python-logging-extras; extra == "extended-logging-formatter"
Provides-Extra: rich-text
Requires-Dist: rich; extra == "rich-text"
Requires-Dist: rich-argparse; extra == "rich-text"
Dynamic: license-file

==============================
``prometheus-dcache-exporter``
==============================
This Prometheus_ exporter collects metrics from dCache_ and provides them via a
HTTP server, writes them to standard output or writes them atomically to a file.




Building And Installation
=========================


Dependencies
------------
- Python 3.13
- The packages/modules |certifi|_, |httpx|_, |humanfriendly|_, |jsonpath|_,
  |module_utils|_, |paramiko|_ and |prometheus_client|_.
- Optionally, the packages/modules |logging_extras|_, |rich|_ and
  |rich_argparse|_.


Getting The Source Code
-----------------------
Clone the |prometheus-dcache-exporter git repository|_ via either of:

.. code-block:: shell

        $ git clone ssh://git@gitlab.com/calestyo/prometheus-dcache-exporter.git

.. code-block:: shell

        $ git clone https://gitlab.com/calestyo/prometheus-dcache-exporter.git


Building A Debian Package
-------------------------
The Debian_ packaging is maintained in the
|debian-branch of the prometheus-dcache-exporter git repository|_.

Preparations:

.. code-block:: shell

        $ cd prometheus-dcache-exporter
        $ git switch -c temporary-build-branch master
        $ temporary_version="$(python3 -m setuptools_scm --format plain)"
        $ [ -z "$( git tag --list "v${temporary_version}" )" ]  &&  git tag "v${temporary_version}"
        $ git merge --allow-unrelated-histories -m "Merge branch 'debian' into temporary-build-branch" debian
        $ debian_changelog_version="$(dpkg-parsechangelog --show-field Version)"
        $ [ "${temporary_version}"   !=   "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E 's/^[0-9]+://; s/-[^-]+$//;' )" ]    &&    debchange --release-heuristic log --newversion "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E -n 's/^([0-9]+:).*/\1/p' )${temporary_version}-1" 'Local version.'
        $ git deborig

Any changes since the tip of the ``master``-branch must be committed right after
switching to the ``temporary-build-branch``-branch.

Build via either of:

.. code-block:: shell

        $ dpkg-buildpackage

.. code-block:: shell

        $ debuild

optionally adding the ``--no-sign``-option if signing is not desired.

Beware that the above creates files in the parent directory.

|git-deborig|_ is part of the |git-debpush|_-package, |dpkg-buildpackage|_ and
|dpkg-parsechangelog|_ are part of the |dpkg-dev|_-package and |debuild|_ is
part of the |devscripts|_-package.


Installation From The Python Package Index (PyPI)
-------------------------------------------------
.. code-block:: shell

        $ pip install prometheus-dcache-exporter




Usage
=====
There are two main operation modes:

- | Executing ``prometheus-dcache-exporter`` in the ``http``-export-mode via a
    `systemd service`_, et cetera, with the former providing the metrics via an
    internal HTTP server.
  | The metrics are (freshly) collected on every scrape. The scraping is to be
    done directly (by Prometheus).
- | Executing ``prometheus-dcache-exporter`` in the ``file``-export-mode or in
    the ``stdout``-export-mode together with a program like |sponge|_ (in both
    cases) via a `systemd timer`_, ``cron`` job, et cetera, which writes the
    metrics atomically (via moving a newly created temporary file) to a
    ``.prom``-file for `Prometheus node exporter’s textfile collector`_.
  | The metrics are (periodically) collected (once) on every execution. The
    scraping is to be done indirectly via the `Prometheus node exporter`_ (by
    Prometheus).

Invoke ``prometheus-dcache-exporter`` with the ``--help``- or ``-h``-option for
information about the command arguments.

| It’s not necessary that dCache_ respectively the REST interface of its
  ``frontend``-service and/or the SSH interface of its ``admin``-service are
  executed on the same host as that on which ``prometheus-dcache-exporter`` is
  executed.
| For security reasons and to allow for alleviation of ``frontend``’s rate
  limiting without affecting other users, it’s suggested to run dedicated
  instances of (at least) ``frontend`` and (possibly also) ``admin`` that are
  only accessible by trusted users (see the example below, which assumes however
  that all users on the host are trusted).

| While sensible defaults have been chosen for all options, in particular the
  connection and authentication options for the REST and SSH interfaces may need
  to be adapted.
| Amongst others, these are part of the ``--dcache-frontend-rest.*``-
  respectively the ``--dcache-admin-ssh.*``-family of options.

For example, assuming ``frontend`` and ``admin`` being executed with their
default configurations except for:

.. code-block:: jproperties

        frontend.net.listen=localhost
        frontend.authn.protocol=http
        frontend.authz.readonly=true
        frontend.authz.anonymous-operations=READONLY
        frontend.authz.unlimited-operation-visibility=true
        frontend.limits.rate.overall=10000
        frontend.limits.rate.per-client.fractions=100
        admin.net.listen=localhost

and login to the SSH interface being authorised for the user ``admin`` with the
SSH private key at ``~/.ssh/id_rsa``, ``prometheus-dcache-exporter`` might be
invoked like:

.. code-block:: shell

        $ prometheus-dcache-exporter --dcache-frontend-rest.base-url http://localhost:3880/api/ --dcache-admin-ssh.user admin --dcache-admin-ssh.private-key ~/.ssh/id_rsa

| When executed via either of ``prometheus-dcache-exporter.service``- or
  ``prometheus-node-exporter-dcache.service``, the above options are usually set
  in the ``ARGS``-variable of their respective “configuration file” (see the
  ``EnvironmentFile=``-option of the respective systemd service).
| It shall be noted, that files read by ``prometheus-dcache-exporter`` (like
  ``~/.ssh/id_rsa`` in the example above) need to be accessible by the user
  executing ``prometheus-dcache-exporter``, which is usually
  ``_prometheus-dcache-exporter``.




Metrics
=======


Collection
----------
``prometheus-dcache-exporter`` uses the REST interface of dCache’s
``frontend``-service and the SSH interface of dCache’s ``admin``-service for
collecting the metrics.
The latter is only used for raw data that isn’t available via the former, yet.

As mentioned previously, the metrics are (freshly) collected on every execution
respectively scrape.
Likewise, dCache (freshly) retrieves the raw data on every request to the REST
interface respectively command execution via the SSH interface.
*This should be considered when choosing Prometheus’ scrape interval.*

Connections to the REST and SSH interfaces are tried to be kept open and
re-used, as configured via the
``--dcache-frontend-rest.connection-pool-*``-family of options respectively the
``--dcache-admin-ssh.keepalive-interval``-option.

Per default, the requests to the REST interface respectively the command
executions via the SSH interface are performed concurrently (using
multithreading), as configured via the
``--dcache-frontend-rest.max-concurrent-requests``- and
``--dcache-admin-ssh.max-concurrent-commands``-options.
This is done, amongst other reasons, so that the raw data is as closely
synchronised as possible.


Errors
~~~~~~
The metrics of the ``<metrics-name-prefix>__errors_*``-family indicate errors
during metric collection, with the
``<metrics-name-prefix>__errors_occurred``-metric being a boolean that indicates
whether any (metric collection) errors occurred or not.


Extra Labels
------------
In general, labels describe the entity to which a time series of a metric
corresponds.
For example, in:

.. code-block:: promql

        # HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
        # TYPE dcache_space_reservation_space_total_bytes gauge
        dcache_space_reservation_space_total_bytes{space_reservation_id="110000"} 4748869032235006
        dcache_space_reservation_space_total_bytes{space_reservation_id="380987"} 292706065309706

the label ``space_reservation_id`` uniquely identifies the entity (here, a space
reservation), thereby effectively being a “primary key”.

There may also be labels that are not necessary for unique identification, which
is particularly (but not only) the case for “info metrics” (whose names
typically end in ``_info``).
For example, in:

.. code-block:: promql

        # HELP dcache_space_reservation_info Information about a space reservation.
        # TYPE dcache_space_reservation_info gauge
        dcache_space_reservation_info{description="ATLASDATADISK",space_reservation_id="110000"} 1.0
        dcache_space_reservation_info{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 1.0

only the label ``space_reservation_id`` identifies the corresponding space
reservation uniquely, while the label ``description`` – albeit often uniquely
used by dCache operators – is strictly speaking not ensured to be unique and is
not necessary for identifying the corresponding space reservation.

Per default and except for info metrics, ``prometheus-dcache-exporter``
generally includes only the minimal set of labels that are necessary to uniquely
identify the corresponding entity.

Sometimes it may however be convenient to have additional labels included.
For example, so that the above metric looks like:

.. code-block:: promql

        # HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
        # TYPE dcache_space_reservation_space_total_bytes gauge
        dcache_space_reservation_space_total_bytes{description="ATLASDATADISK",space_reservation_id="110000"} 4748869032235006
        dcache_space_reservation_space_total_bytes{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 292706065309706

This can be achieved – for certain metrics and labels where this seemed
reasonable – using the ``--metrics.extra-label``-option.

In Prometheus, the change of a label’s value (as well as the addition/removal of
a label) results in a new/different time series.
It’s therefore suggested to choose which extra labels to include, if any,
already before starting to use ``prometheus-dcache-exporter`` and to avoid
changing this later.




Issues
======


Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use
---------------------------------------------------------------------------------------------------------------------
Currently, quite a number of concurrent requests respectively command executions
are made, including one request per dCache pool and one command execution per
dCache domain.

These particular two cases can easily cause high numbers of concurrent requests
respectively command executions and while they should eventually be reduced to
only two requests (see
`dCache issues #8126 <https://github.com/dCache/dcache/issues/8126>`_ and
`#8127 <https://github.com/dCache/dcache/issues/8127>`_), this may cause various
failures, which are unfortunately not yet understood (in particular, whether
they’re caused by ``prometheus-dcache-exporter`` itself, the libraries it uses
or dCache respectively the libraries it uses in the ``frontend``- and/or
``admin``-services).

Because of the concurrency, such failures usually cause log message flooding.
Further, because threads cannot really be forcibly terminated in Python, the
already running (but all failing) threads continue their execution despite
exiting signals like ``SIGINT`` or ``SIGTERM``.

Tests seemed to have indicated that it even makes a difference when the
connections to the REST and/or SSH interfaces are tunnelled via OpenSSH’s port
forwarding (using ``ssh``’s ``-L``-option).




Troubleshooting
===============


REST Interface: HTTP Status Code ``429`` (``Too Many Requests``)
----------------------------------------------------------------
In case of log messages like::

        ERROR: prometheus_dcache_exporter._dcache.frontend: 1: Failed to request the resource `/pools/<pool-name>/usage` from the REST interface of the dCache `frontend` service: Client error '429 Too Many Requests' for url 'http://localhost:3880/api/v1/pools/<pool-name>/usage'
        ERROR: prometheus_dcache_exporter._dcache.frontend: 2: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

``prometheus-dcache-exporter`` likely exceeded the thresholds of the
``frontend``-service’s rate limiting.

This can be configured via the ``frontend.limits.rate.overall``- and
``frontend.limits.rate.per-client.fractions``-options of the `frontend``-service
(see the example in the chapter “Usage_” above).

See also
`Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use`_
above.


SSH Interface: ``Timeout opening channel.``, ``SSHException`` And Other Paramiko Errors
---------------------------------------------------------------------------------------
In case of log messages like::

        ERROR: prometheus_dcache_exporter._dcache.admin: Failed to execute the command `\s <cell-name>@<domain-name> get hostname` via the SSH interface of the dCache `admin` service (try 2 of 2): Timeout opening channel.

despite the ``Timeout opening channel.``, the timeout that usually causes this
is that configured via the
``--dcache-admin-ssh.command-channel-timeout``-option.
Increasing that or decreasing the concurrency via the
``--dcache-admin-ssh.max-concurrent-commands``-option might help.

Ironically and inexplicably, tests have shown that, instead, it might help best
to disable connection re-use of the REST interface of dCache’s
``frontend``-service.
Similarly, decreasing the concurrency of that via the
``--dcache-frontend-rest.max-concurrent-requests``-options might help.

See also
`Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use`_
above.




References
==========
- `Presentation at the 20th International dCache Workshop <https://indico.nikhef.nl/event/7562/contributions/29945/>`_




Licence_
========








.. _LICENCE: LICENCE

.. _prometheus-dcache-exporter git repository: https://gitlab.com/calestyo/prometheus-dcache-exporter
.. _debian-branch of the prometheus-dcache-exporter git repository: https://gitlab.com/calestyo/prometheus-dcache-exporter/-/tree/debian

.. _dCache: https://dcache.org/
.. _Prometheus: https://prometheus.io/
.. _Prometheus node exporter: https://github.com/prometheus/node_exporter
.. _Prometheus node exporter’s textfile collector: https://github.com/prometheus/node_exporter#textfile-collector
.. _Debian: https://debian.org/

.. _certifi: https://github.com/certifi/python-certifi
.. _httpx: https://github.com/encode/httpx/
.. _humanfriendly: https://github.com/xolox/python-humanfriendly
.. _jsonpath: https://github.com/jg-rp/python-jsonpath
.. _logging_extras: https://gitlab.com/calestyo/python-logging-extras
.. _module_utils: https://gitlab.com/calestyo/python-module-utils
.. _paramiko: https://github.com/paramiko/paramiko/
.. _prometheus_client: https://github.com/prometheus/client_python
.. _rich: https://github.com/Textualize/rich
.. _rich_argparse: https://github.com/hamdanal/rich-argparse

.. _debuild: https://manpages.debian.org/stable/devscripts/debuild.1.html
.. _dpkg-buildpackage: https://manpages.debian.org/stable/dpkg-dev/dpkg-buildpackage.1.html
.. _dpkg-parsechangelog: https://manpages.debian.org/stable/dpkg-dev/dpkg-parsechangelog.1.html
.. _git-deborig: https://manpages.debian.org/stable/devscripts/git-deborig.1.html
.. _sponge: https://manpages.debian.org/stable/moreutils/sponge.1.html
.. _systemd service: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html
.. _systemd timer: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html

.. _devscripts: https://packages.debian.org/stable/devscripts
.. _dpkg-dev: https://packages.debian.org/stable/dpkg-dev
.. _git-debpush: https://packages.debian.org/stable/git-debpush




.. |certifi| replace:: ``certifi``
.. |debian-branch of the prometheus-dcache-exporter git repository| replace:: ``debian-branch`` of the ``prometheus-dcache-exporter`` git repository
.. |debuild| replace:: ``debuild``
.. |devscripts| replace:: ``devscripts``
.. |dpkg-buildpackage| replace:: ``dpkg-buildpackage``
.. |dpkg-parsechangelog| replace:: ``dpkg-parsechangelog``
.. |dpkg-dev| replace:: ``dpkg-dev``
.. |git-deborig| replace:: ``git-deborig``
.. |git-debpush| replace:: ``git-debpush``
.. |httpx| replace:: ``httpx``
.. |humanfriendly| replace:: ``humanfriendly``
.. |jsonpath| replace:: ``jsonpath``
.. |logging_extras| replace:: ``logging_extras``
.. |module_utils| replace:: ``module_utils``
.. |paramiko| replace:: ``paramiko``
.. |prometheus_client| replace:: ``prometheus_client``
.. |prometheus-dcache-exporter git repository| replace:: ``prometheus-dcache-exporter`` git repository
.. |rich| replace:: ``rich``
.. |rich_argparse| replace:: ``rich_argparse``
.. |sponge| replace:: ``sponge``
