========================
Customizing miteʼs stats
========================

There are two components of the mite pipeline that are concerned with
the reporting of test data: the stats component and the prometheus
exporter.  The prometheus exporter works in a fully automatic way, based
on the behavior of the stats component.  Therefore, when customizing the
statistics that mite exports, it is sufficient to work on the stats
component; the prometheus exporter will do the right thing.

Mite stats are registered as `entry points`_ in python.  Therefore, any
modules that you install that extend miteʼs stats will automatically be
picked up when the stats process starts.  It logs a message for each
moduleʼs stats that it finds, so you will know exactly what is loaded.

.. _entry points: https://amir.rachum.com/blog/2017/07/28/python-entry-points/

Writing custom stats
--------------------

Prometheus, miteʼs chosen data backend, supports three different kinds
of metrics: a Gauge, a Counter, and a Histogram.

.. note::

   Prometheus also supports a Summary metric type, but this is just (for
   our purposes) a less performant version of a Histogram, so it will not
   be further discussed.

Each of these is exposed as a class in the ``mite.stats`` module.  When
instantiating one of these classes to create a custom metric, you will
need to supply a name, a matcher, and an extractor.  The name is
directly translated into the prometheus name for the metric, so it is
useful for it to start with ``mite_`` so that related metrics will be
grouped together (especially if the prometheus instance that collects
your mite metrics also collects other data, like CPU and memory usage of
application processes).

The stats component acts by processing a stream of messages generated by
the runner and controller.  The matcher is a function that filters these
messages, determining which are of interest to a particular stat and
which it can ignore.  Because each message has a ``type`` field, it is
often useful to match on the value of that field, which is the purpose
of the included ``mite.stats.matcher_by_type`` function.  (However, the
matcher receives the entire message dict as an argument, and can perform
arbitrary computation on its contents if it wishes.)

The extractor function operates on the messages that have been selected
by the matcher.  The extractor pulls out some labels from the message,
which will be added as labels to the time series in prometheus.  It is
also, in most cases, responsible for extracting a numerical value from
the message.  (The exception is stats of the Counter type, which count
the occurrences of a particular message type.  There, no value dependent
on the message is needed, as the value of the counter is always
incremented by one).  The built-in ``mite.stats.extractor`` function
covers most use-cases.  It takes two arguments.  The first is a sequence
of strings, which will be extracted as keys from the message
dictionary.  The values of these keys will be used as the labels on the
prometheus metric.  The second argument is a single string, which names
the dictionary key containing the numeric value to be accumulated.  For
more advanced usages, it will be necessary to construct a
``mite.stats.Extractor`` class directly.  See
``mite.stats.controller_report_extractor`` for an example of this.

Here is an example of a custom stat from our work on AMQP testing:

.. code-block:: python

   def _amqp_extract(msg):
       for key, value in msg["total_received"].items():
           yield (key,), value


   _MITE_STATS = [
       Gauge(
           "mite_amqp_tx_stats",
           matcher_by_type("amqp_tx_stats"),
           extractor(["message_name"], "total_sent")
       ),
       Gauge(
           "mite_amqp_rx_stats",
           matcher_by_type("amqp_rx_stats"),
           Extractor(labels=["message_name"], extract=_amqp_extract)
       )
   ]

In our AMQP injection functions, we send two messages.  The first,
``amqp_tx_stats``, names a ``message_name`` and contains a
``total_sent`` value.  That is extracted into a Gauge by the first
stat.  Because each ``message_name`` is represented by a separate
message, this is a simple stat.  The second is the ``mite_rx_stats``.
This message is send by the worker coroutine that drains the AMQP
queues.  Once a second, it reports on the total number of messages it
has received – of all types.  Therefore, we need to write a custom
``_amqp_extract`` function, which will yield a sequence of ``(key,
value)`` tuples.

We also need to inform mite about our stats, using pythonʼs entry points
mechanism.  To do so, we add a section to the ``setup.cfg`` file for our
package:

.. code-block:: cfg

   [options.entry_points]
   mite_stats =
       mite_amqp = id_mite_nft.amqp:_MITE_STATS


.. note::

   It is also possible to specify the equivalent information in
   ``setup.py`` if that is the configuration file your project uses.


Configuring custom stats
------------------------

By default, all stats processors that are registered as entrypoints will
be used by the mite stats component.  However, because each processor
has to execute code for each message (even if only to determine that it
is not interested in that message), there can be a large cumulative
performance impact if you have many stats processors registered.  The
``--stats-include-processors`` and ``--stats-exclude-processors`` can be
used to fine-tune this behavior.  If ``--stats-include-processors`` is
passed, then only processors whose name is on the list will be included.
If ``--stats-exclude-processors`` is passed, then all processors will be
included except those on the list.  (You can pass both arguments, which
will result in the processors from the include list but not on the
exclude list being used – but it doesnʼt make much sense to do this,
just pass the difference of the two sets as the include argument, and
omit the exclude one).

The format of both argumentsʼ value is a comma-separated list without
spaces: ``--stats-include-processors=processor1,processor2``

Available processors
--------------------

The following processors are provided by mite:

- ``mite`` – statistics about the controller and runner workloads
- ``mite_http`` – statistics about miteʼs http calls
- ``mite_selenium`` – statistics about miteʼs selenium-integrated
  browser testing

  .. In the future, we can add here any processors provided by
     third-party libraries (our own, or other teamsʼ)
