Metadata-Version: 2.4
Name: dit
Version: 2.1
Summary: Python package for information theory on discrete random variables.
Project-URL: Homepage, http://dit.io
Project-URL: Repository, https://github.com/dit/dit
Project-URL: Documentation, http://docs.dit.io
Author-email: chebee7i <admin@dit.io>, "Ryan G. James" <dit@autoplectic.com>
License-Expression: BSD-3-Clause
License-File: LICENSE.txt
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.11
Requires-Dist: boltons
Requires-Dist: debtcollector
Requires-Dist: lattices>=0.5.0
Requires-Dist: loguru
Requires-Dist: networkx>=2.6
Requires-Dist: numpy>=1.22
Requires-Dist: pltable
Requires-Dist: scipy>=1.7
Requires-Dist: xarray
Provides-Extra: broja
Requires-Dist: ecos; extra == 'broja'
Provides-Extra: dev
Requires-Dist: colorama; extra == 'dev'
Requires-Dist: hypothesis; extra == 'dev'
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: hypothesis[numpy]; extra == 'dev'
Requires-Dist: ipython; extra == 'dev'
Requires-Dist: jax; extra == 'dev'
Requires-Dist: jaxlib; extra == 'dev'
Requires-Dist: matplotlib; extra == 'dev'
Requires-Dist: nbconvert; extra == 'dev'
Requires-Dist: numdifftools; extra == 'dev'
Requires-Dist: optimistix; extra == 'dev'
Requires-Dist: pint; extra == 'dev'
Requires-Dist: pycddlib-standalone; extra == 'dev'
Requires-Dist: pypoman; extra == 'dev'
Requires-Dist: pytensor; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-rerunfailures; extra == 'dev'
Requires-Dist: pytest-sugar; extra == 'dev'
Requires-Dist: pytest-xdist; extra == 'dev'
Requires-Dist: pytest>=9.0.3; extra == 'dev'
Requires-Dist: python-ternary; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: scikit-learn; extra == 'dev'
Requires-Dist: slsqp-jax; extra == 'dev'
Requires-Dist: sphinx; extra == 'dev'
Requires-Dist: sphinx-rtd-theme; extra == 'dev'
Requires-Dist: sphinxcontrib-bibtex; extra == 'dev'
Requires-Dist: torch; extra == 'dev'
Requires-Dist: ty; extra == 'dev'
Provides-Extra: docs
Requires-Dist: colorama; extra == 'docs'
Requires-Dist: hypothesis; extra == 'docs'
Requires-Dist: ipython; extra == 'docs'
Requires-Dist: matplotlib; extra == 'docs'
Requires-Dist: nbconvert; extra == 'docs'
Requires-Dist: numdifftools; extra == 'docs'
Requires-Dist: pint; extra == 'docs'
Requires-Dist: pycddlib-standalone; extra == 'docs'
Requires-Dist: pypoman; extra == 'docs'
Requires-Dist: python-ternary; extra == 'docs'
Requires-Dist: scikit-learn; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-rtd-theme; extra == 'docs'
Requires-Dist: sphinxcontrib-bibtex; extra == 'docs'
Provides-Extra: jax
Requires-Dist: jax; extra == 'jax'
Requires-Dist: jaxlib; extra == 'jax'
Requires-Dist: optimistix; extra == 'jax'
Requires-Dist: slsqp-jax; extra == 'jax'
Provides-Extra: optional
Requires-Dist: colorama; extra == 'optional'
Requires-Dist: matplotlib; extra == 'optional'
Requires-Dist: numdifftools; extra == 'optional'
Requires-Dist: pint; extra == 'optional'
Requires-Dist: pycddlib-standalone; extra == 'optional'
Requires-Dist: pypoman; extra == 'optional'
Requires-Dist: scikit-learn; extra == 'optional'
Provides-Extra: pytensor
Requires-Dist: pytensor; extra == 'pytensor'
Provides-Extra: test
Requires-Dist: hypothesis>=6.0; extra == 'test'
Requires-Dist: hypothesis[numpy]; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-rerunfailures; extra == 'test'
Requires-Dist: pytest-sugar; extra == 'test'
Requires-Dist: pytest-xdist; extra == 'test'
Requires-Dist: pytest>=9.0.3; extra == 'test'
Provides-Extra: torch
Requires-Dist: torch; extra == 'torch'
Description-Content-Type: text/x-rst

``dit`` is a Python package for information theory.

|build| |codecov| |docs| |conda|

|joss| |zenodo| |slack|

Introduction
------------

Information theory is a powerful extension to probability and statistics, quantifying dependencies
among arbitrary random variables in a way that is consistent and comparable across systems and
scales. Information theory was originally developed to quantify how quickly and reliably information
could be transmitted across an arbitrary channel. The demands of modern, data-driven science have
been coopting and extending these quantities and methods into unknown, multivariate settings where
the interpretation and best practices are not known. For example, there are at least four reasonable
multivariate generalizations of the mutual information, none of which inherit all the
interpretations of the standard bivariate case. Which is best to use is context-dependent. ``dit``
implements a vast range of multivariate information measures in an effort to allow information
practitioners to study how these various measures behave and interact in a variety of contexts. We
hope that having all these measures and techniques implemented in one place will allow the
development of robust techniques for the automated quantification of dependencies within a system
and concrete interpretation of what those dependencies mean.

Citing
------

If you use `dit` in your research, please cite it as::

   @article{dit,
     Author = {James, R. G. and Ellison, C. J. and Crutchfield, J. P.},
     Title = {{dit}: a {P}ython package for discrete information theory},
     Journal = {The Journal of Open Source Software},
     Volume = {3},
     Number = {25},
     Pages = {738},
     Year = {2018},
     Doi = {https://doi.org/10.21105/joss.00738}
   }

Basic Information
-----------------

Documentation
*************

http://docs.dit.io

Downloads
*********

https://pypi.org/project/dit/

https://anaconda.org/conda-forge/dit

+-------------------------------------------------------------------+
| Dependencies                                                      |
+===================================================================+
| * Python 3.11+                                                    |
| * `boltons <https://boltons.readthedocs.io>`_                     |
| * `debtcollector <https://docs.openstack.org/debtcollector/>`_    |
| * `lattices <https://github.com/dit/lattices>`_                   |
| * `loguru <https://loguru.readthedocs.io>`_                       |
| * `networkx <https://networkx.github.io/>`_                       |
| * `numpy <http://www.numpy.org/>`_                                |
| * `PLTable <https://github.com/platomav/PLTable>`_                |
| * `scipy <https://www.scipy.org/>`_                               |
| * `xarray <https://docs.xarray.dev/>`_                            |
+-------------------------------------------------------------------+

Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~
* colorama: colored column heads in PID indicating failure modes
* cython: faster sampling from distributions
* hypothesis: random sampling of distributions
* jax, jaxlib: JAX-based optimization backend with autodiff support
* matplotlib, python-ternary: plotting of various information-theoretic expansions
* numdifftools: numerical evaluation of gradients and hessians during optimization
* pint: add units to informational values
* pycddlib-standalone, pypoman: polytope vertex enumeration for convex-maximization-based measures
* pytensor: PyTensor-based optimization backend with autodiff support
* scikit-learn: faster nearest-neighbor lookups during entropy/mutual information estimation from samples
* torch: PyTorch-based optimization backend with autodiff and GPU support

Install
*******

The easiest way to install is:

.. code-block:: bash

  pip install dit

If you want to install ``dit`` within a conda environment, you can simply do:

.. code-block:: bash

  conda install -c conda-forge dit

For development, we recommend `uv <https://docs.astral.sh/uv/>`_:

.. code-block:: bash

  git clone https://github.com/dit/dit.git
  cd dit
  uv sync --extra dev

This installs ``dit`` in editable mode with all development dependencies
(tests, docs, linting, type checking, and optional backends).

Testing
*******

.. code-block:: bash

  # Using uv (recommended)
  uv run pytest

  # Or with pip
  pip install -e ".[test]"
  pytest

Code and bug tracker
********************

https://github.com/dit/dit

License
*******

BSD 3-Clause, see LICENSE.txt for details.

Implemented Measures
--------------------

``dit`` implements the following information measures. Most of these are implemented in multivariate & conditional
generality, where such generalizations either exist in the literature or are relatively obvious --- for example,
though it is not in the literature, the multivariate conditional exact common information is implemented here.

+------------------------------------------+-----------------------------------------+-----------------------------------+
| Entropies                                | Mutual Informations                     | Divergences                       |
|                                          |                                         |                                   |
| * Shannon Entropy                        | * Co-Information                        | * Variational Distance            |
| * Renyi Entropy                          | * Interaction Information               | * Kullback-Leibler Divergence /   |
| * Tsallis Entropy                        | * Total Correlation / Multi-Information |   Relative Entropy                |
| * Necessary Conditional Entropy          | * Dual Total Correlation / Binding      | * Cross Entropy                   |
| * Residual Entropy / Independent         |   Information                           | * Jensen-Shannon Divergence       |
|   Information / Variation of Information | * CAEKL Multivariate Mutual Information | * Earth Mover's Distance          |
+------------------------------------------+ * O-Information                         +-----------------------------------+
| Common Informations                      | * Cohesion                              | Other Measures                    |
|                                          | * Transmission                          |                                   |
| * Gacs-Korner Common Information         | * Union Information                     | * Channel Capacity                |
| * Wyner Common Information               | * Logarithmic Decomposition             | * Complexity Profile              |
| * Exact Common Information               | * Delta^k / Gamma^k                     | * Connected Informations          |
| * Functional Common Information          | * DeWeese Mutual Information            | * Copy Mutual Information         |
| * MSS Common Information                 +-----------------------------------------+ * Cumulative Residual Entropy     |
| * Kamath-Anantharam Dual Common          | Partial Information Decomposition       | * Extropy                         |
|   Information                            |                                         | * Hypercontractivity Coefficient  |
| * Beta Common Information                | * :math:`I_{min}`                       | * Information Bottleneck          |
| * Salamatian-Cohen-Medard Maximum Entropy| * :math:`I_{\wedge}`                    | * Information Diagrams            |
|   Function                               | * :math:`I_{RR}`                        | * Information Trimming            |
+------------------------------------------+ * :math:`I_{\downarrow}`                | * Lautum Information              |
| Secret Key Agreement Bounds              | * :math:`I_{proj}`                      | * LMPR Complexity                 |
|                                          | * :math:`I_{BROJA}`                     | * Marginal Utility of Information |
| * Secrecy Capacity                       | * :math:`I_{ccs}`                       | * Maximum Correlation             |
| * Intrinsic Mutual Information           | * :math:`I_{\pm}`                       | * Maximum Entropy Distributions   |
| * Reduced Intrinsic Mutual Information   | * :math:`I_{sx}`                        | * Perplexity                      |
| * Minimal Intrinsic Mutual Information   | * :math:`I_{dep}`                       | * Rate-Distortion Theory          |
| * Necessary Intrinsic Mutual Information | * :math:`I_{RAV}`                       | * TSE Complexity                  |
| * Two-Part Intrinsic Mutual Information  | * :math:`I_{mmi}`                       | * Gray-Wyner Network              |
|                                          | * :math:`I_{\prec}`                     |                                   |
|                                          | * :math:`I_{RA}`                        |                                   |
|                                          | * :math:`I_{SKAR}`                      |                                   |
|                                          | * :math:`I_{IG}`                        |                                   |
|                                          | * :math:`I_{RDR}`                       |                                   |
|                                          | * :math:`I_{do}`                        |                                   |
+------------------------------------------+-----------------------------------------+-----------------------------------+

Quickstart
----------

The basic usage of ``dit`` corresponds to creating distributions, modifying them
if need be, and then computing properties of those distributions. First, we
import:

.. code:: python

   >>> import dit

Suppose we have a really thick coin, one so thick that there is a reasonable
chance of it landing on its edge. Here is how we might represent the coin in
``dit``.

.. code:: python

   >>> d = dit.Distribution(['H', 'T', 'E'], [.4, .4, .2])
   >>> print(d)
   Class:          Distribution
   Alphabet:       ('E', 'H', 'T') for all rvs
   Base:           linear
   Outcome Class:  str
   Outcome Length: 1
   RV Names:       None

   x   p(x)
   E   0.2
   H   0.4
   T   0.4

Calculate the probability of ``H`` and also of the combination ``H or T``.

.. code:: python

   >>> d['H']
   0.4
   >>> d.event_probability(['H','T'])
   0.8

Calculate the Shannon entropy and extropy of the joint distribution.

.. code:: python

   >>> dit.shannon.entropy(d)
   1.5219280948873621
   >>> dit.other.extropy(d)
   1.1419011889093373

Create a distribution where ``Z = xor(X, Y)``.

.. code:: python

   >>> import dit.example_dists
   >>> d = dit.example_dists.Xor()
   >>> d.set_rv_names(['X', 'Y', 'Z'])
   >>> print(d)
   Class:          Distribution
   Alphabet:       ('0', '1') for all rvs
   Base:           linear
   Outcome Class:  str
   Outcome Length: 3
   RV Names:       ('X', 'Y', 'Z')

   x     p(x)
   000   0.25
   011   0.25
   101   0.25
   110   0.25

Calculate the Shannon mutual informations ``I[X:Z]``, ``I[Y:Z]``, and
``I[X,Y:Z]``.

.. code:: python

   >>> dit.shannon.mutual_information(d, ['X'], ['Z'])
   0.0
   >>> dit.shannon.mutual_information(d, ['Y'], ['Z'])
   0.0
   >>> dit.shannon.mutual_information(d, ['X', 'Y'], ['Z'])
   1.0

Calculate the marginal distribution ``P(X,Z)``.
Then print its probabilities as fractions, showing the mask.

.. code:: python

   >>> d2 = d.marginal(['X', 'Z'])
   >>> print(d2.to_string(show_mask=True, exact=True))
   Class:          Distribution
   Alphabet:       ('0', '1') for all rvs
   Base:           linear
   Outcome Class:  str
   Outcome Length: 2 (mask: 3)
   RV Names:       ('X', 'Z')

   x     p(x)
   0*0   1/4
   0*1   1/4
   1*0   1/4
   1*1   1/4

Convert the distribution probabilities to log (base 3.5) probabilities, and
access its probability mass function.

.. code:: python

   >>> d2.set_base(3.5)
   >>> d2.pmf
   array([-1.10658951, -1.10658951, -1.10658951, -1.10658951])

Draw 5 random samples from this distribution.

.. code:: python

   >>> dit.math.prng.seed(1)
   >>> d2.rand(5)
   ['01', '10', '00', '01', '00']

Source and Channel Coding
-------------------------

Beyond measures, ``dit`` builds explicit codes and ships a catalog of channels.

The ``dit.coding`` module constructs lossless *source codes* (Shannon, Fano,
Shannon-Fano-Elias, Huffman, length-limited Huffman, Golomb/Rice, Tunstall, and
the universal integer codes) and reports their code-theoretic properties (rate,
redundancy, efficiency, the Kraft sum, and whether the code is prefix-free /
uniquely decodable / optimal).

.. code:: python

   >>> from dit.coding import huffman
   >>> d = dit.Distribution(['a', 'b', 'c', 'd', 'e'], [0.4, 0.2, 0.2, 0.1, 0.1])
   >>> code = huffman(d)
   >>> code.average_length()
   2.2
   >>> float(code.source_entropy())
   2.1219280948873624
   >>> code.is_optimal(), code.is_prefix_free()
   (True, True)

It also builds binary (GF(2)) *channel codes* --- linear block codes
(repetition, parity-check, Hamming, Reed-Muller, Golay) as well as LDPC, polar,
and convolutional codes --- and evaluates them against a noisy channel supplied
as a conditional ``Distribution`` ``p(Y|X)``.

.. code:: python

   >>> from dit.coding import hamming
   >>> code = hamming(3)
   >>> code.length, code.dimension, code.minimum_distance()
   (7, 4, 3)
   >>> bsc = dit.example_channels.binary_symmetric_channel(0.05)
   >>> float(code.probability_of_error(bsc, method='exact'))
   0.04438054218749993

The ``dit.example_channels`` module is a catalog of canonical discrete
memoryless channels (binary symmetric, binary erasure, Z-channel, q-ary
symmetric/erasure, noisy typewriter, ...). Each constructor returns a
conditional ``Distribution`` ``p(Y|X)`` ready for
``dit.algorithms.channel_capacity`` or the coding layer above.

.. code:: python

   >>> from dit.algorithms import channel_capacity
   >>> bec = dit.example_channels.binary_erasure_channel(0.25)
   >>> float(channel_capacity(bec)[0])
   0.75

Contributions & Help
--------------------

If you'd like to feature added to ``dit``, please file an issue. Or, better yet, open a pull request. Ideally, all code should be tested and documented, but please don't let this be a barrier to contributing. We'll work with you to ensure that all pull requests are in a mergable state.

If you'd like to get in contact about anything, you can reach us through our `slack channel <https://dit-python.slack.com/>`_.


.. badges:

.. |build| image:: https://github.com/dit/dit/actions/workflows/build.yml/badge.svg
   :target: https://github.com/dit/dit/actions/workflows/build.yml
   :alt: Continuous Integration Status

.. |codecov| image:: https://codecov.io/gh/dit/dit/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/dit/dit
   :alt: Test Coverage Status

.. |docs| image:: https://readthedocs.org/projects/dit/badge/?version=latest
   :target: http://dit.readthedocs.org/en/latest/?badge=latest
   :alt: Documentation Status

.. |conda| image:: https://anaconda.org/conda-forge/dit/badges/installer/conda.svg
   :target: https://anaconda.org/conda-forge/dit
   :alt: Conda installation

.. |zenodo| image:: https://zenodo.org/badge/13201610.svg
   :target: https://zenodo.org/badge/latestdoi/13201610
   :alt: DOI

.. |slack| image:: https://img.shields.io/badge/Slack-dit--python-lightgrey.svg
   :target: https://dit-python.slack.com/
   :alt: dit chat

.. |joss| image:: http://joss.theoj.org/papers/10.21105/joss.00738/status.svg
   :target: https://doi.org/10.21105/joss.00738
   :alt: JOSS Status
