Metadata-Version: 2.4
Name: entropath
Version: 1.0.0
Summary: ENTRO-PATH: Irreducible Path Entropy in Neural Networks - A Quantitative Information-Theoretic Framework
Author-email: Samir Baladi <gitdeeper@gmail.com>
License: MIT
Project-URL: Homepage, https://entropath.netlify.app
Project-URL: Documentation, https://entropath.netlify.app/docs
Project-URL: Repository, https://github.com/gitdeeper10/entropath
Project-URL: DOI, https://doi.org/10.5281/zenodo.20222840
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Theory
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
License-File: AUTHORS.md
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: torch>=2.0.0
Requires-Dist: matplotlib>=3.7.0
Dynamic: license-file


# 📊 Irreducible Path Entropy in Neural Networks

> **A Quantitative Information-Theoretic Framework for Entropy Propagation
> Across Computational Decision Trajectories**

---

<div align="center">

[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.20222840-crimson?style=flat-square)](https://doi.org/10.5281/zenodo.20222840)
[![License: MIT](https://img.shields.io/badge/License-MIT-darkred?style=flat-square)](LICENSE)
[![ORCID](https://img.shields.io/badge/ORCID-0009--0003--8903--0029-green?style=flat-square)](https://orcid.org/0009-0003-8903-0029)
[![PyPI](https://img.shields.io/badge/PyPI-entropath-blue?style=flat-square)](https://pypi.org/project/entropath)
[![Status](https://img.shields.io/badge/Status-Preprint%20Ready-blue?style=flat-square)]()
[![Target Journal](https://img.shields.io/badge/Target-Entropy%20MDPI-orange?style=flat-square)](https://www.mdpi.com/journal/entropy)

[![OSF Registration](https://img.shields.io/badge/OSF-Registration-purple?style=flat-square)](https://doi.org/10.17605/OSF.IO/7WP9H)
[![Internet Archive](https://img.shields.io/badge/Internet%20Archive-Snapshot-blue?style=flat-square)](https://archive.org/details/osf-registrations-7wp9h-v1)
[![CC-BY-4.0](https://img.shields.io/badge/License-CC--BY--4.0-lightgrey?style=flat-square)](https://creativecommons.org/licenses/by/4.0/)
[![GitHub Stars](https://img.shields.io/github/stars/gitdeeper12/ENTRO-PATH?style=flat-square&color=gold)](https://github.com/gitdeeper12/ENTRO-PATH)
[![Python 3.8+](https://img.shields.io/badge/Python-3.8+-blue?style=flat-square)](https://www.python.org/)

</div>

---

## 📌 Overview

This repository contains the full research output for the paper:

**"Irreducible Path Entropy in Neural Networks"**
*Samir Baladi — EntropyLab Independent Research Series, May 2026*

The paper introduces **Irreducible Path Entropy** (`H_path`) — a formally defined,
layer-integrated information-theoretic metric that quantifies how much uncertainty
accumulates, transforms, and becomes unrecoverable along the inference trajectory
of a neural network.

The framework is grounded exclusively in **information theory** and **systems-level
analysis**, without semantic, cognitive, or anthropomorphic assumptions.

---

## 📋 OSF Preregistration

| Field | Value |
|-------|-------|
| **Registration Type** | OSF Preregistration |
| **Registry** | OSF Registries |
| **Associated Project** | [https://osf.io/yaevt](https://osf.io/yaevt) |
| **Date Registered** | May 20, 2026 · 6:17 AM UTC |
| **License** | CC-By Attribution 4.0 International |
| **Internet Archive** | [osf-registrations-7wp9h-v1](https://archive.org/details/osf-registrations-7wp9h-v1) |
| **Registration DOI** | **10.17605/OSF.IO/7WP9H** |

---

## 🧭 Motivation

Modern neural networks achieve high performance while remaining structurally opaque.
As noted by Hinton (2023), the learning algorithm is designed — but the precise
inference dynamics remain inaccessible even to the architects who built them.

Existing interpretability tools address specific aspects of this opacity.
**This work addresses a gap**: no unified, layer-integrated metric existed for
characterising entropy accumulation along the full computational decision path.

`H_path` fills this gap.

---

## 🔬 Core Constructs

### Local Path Entropy
```

H_path(l) = − Σ_k p_{l,k} · log p_{l,k}

```
Shannon entropy of the conditional activation distribution at layer `l`.

### Cumulative Path Entropy
```

H_path^(L) = Σ_{l=1}^{L} H(P_l)

```
Total informational uncertainty accumulated across all `L` layers.

### Irreducible Path Entropy
```

H_irr^(L) = H_path^(L) − H_red^(L)

```
The component of path entropy that cannot be recovered from external observations.

### Observability Index
```

Ω(N) = 1 − H_irr^(L) / H_path^(L)  ∈ [0, 1]

```
- `Ω = 1` → fully observable network
- `Ω = 0` → completely irreducible inference dynamics

### Reducibility Condition
A layer `l` is **reducible** if there exists a measurement operator `M_l` such that:
```

I(h_l ; M_l(y)) ≥ H_path(l) − δ*

```
where `δ*` is the reducibility tolerance threshold.

### Entropic Leakage
```

Δ(L) = H_path^(L) − I(x ; h_L)

```
Uncertainty introduced across computation not explained by retained input information.

---

## 📐 Scope and Interpretive Closure

> This framework is restricted to **formal quantitative analysis** of entropy
> propagation in artificial neural networks.

**Not within scope:**
- General theories of intelligence, cognition, or consciousness
- Claims regarding intentionality, agency, or phenomenology
- Semantic or anthropomorphic interpretation of results

**Within scope:**
- Reproducible computational analysis
- Information-theoretic formalisation
- Systems-level characterisation of inference behaviour
- Experimentally observable entropy dynamics

---

## 📂 Repository Structure

```

irreducible-path-entropy/
│
├── 📄 README.md                          # This file
├── 📄 LICENSE                            # MIT License
├── 📄 CHANGELOG.md                       # Version history
├── 📄 AUTHORS.md                         # Author and contributor metadata
│
├── 📁 paper/
│   ├── Irreducible_Path_Entropy_Baladi_2026.pdf   # Publication-ready paper
│   └── preprint_metadata.json                     # Zenodo/OSF submission metadata
│
├── 📁 formalism/
│   ├── definitions.md                    # All formal definitions (1–5)
│   ├── reducibility_conditions.md        # Reducibility threshold derivations
│   ├── observability_index.md            # Ω construction and properties
│   └── entropic_leakage.md               # Δ(L) derivation and interpretation
│
├── 📁 figures/
│   ├── fig1_path_entropy_accumulation.png   # Layer-wise H_path vs H_red
│   ├── fig2_reducibility_phase_diagram.png  # Phase diagram (ρ vs I)
│   └── fig3_observability_architectures.png # Ω across architecture types
│
├── 📁 numerical/
│   ├── entropy_estimator.py              # k-NN entropy estimation module
│   ├── mutual_information.py             # MI estimator for H_red
│   ├── observability_compute.py          # Ω computation pipeline
│   ├── architecture_comparison.py        # MLP / CNN / Transformer benchmarks
│   └── requirements.txt                  # Python dependencies
│
├── 📁 experiments/
│   ├── protocol.md                       # Full reproducibility protocol
│   ├── config_feedforward.yaml           # MLP experiment configuration
│   ├── config_cnn.yaml                   # CNN experiment configuration
│   └── config_transformer.yaml           # Transformer experiment configuration
│
└── 📁 references/
└── bibliography.bib                  # BibTeX reference file

```

---

## ⚙️ Reproducibility Protocol

All results are reproducible under the following conditions:

1. **Fixed weights** — no stochastic inference-time modifications
2. **Consistent discretisation** — activation binning scheme fixed across layers
3. **Fixed estimator parameters** — bandwidth / neighbourhood `k` held constant
4. **Fixed dataset** — `D = {x_i}` held constant across comparative measurements
5. **Fixed random seed** — `seed=42` for deterministic behaviour

### Estimation Pipeline

```

Step 1  →  Record activations {h_l(x_i)} at each layer l
Step 2  →  Apply k-NN entropy estimator → H_path(l)
Step 3  →  Estimate I(h_l ; y) → H_red^(L)
Step 4  →  Compute Ω = 1 − H_irr / H_path

```

---

## 🏗️ Architecture Findings (Illustrative)

| Architecture     | Depth | `H_path` (nats) | `Ω` Index | Regime       |
|-----------------|-------|-----------------|-----------|-------------|
| MLP (2L)        | 2     | 0.31            | 0.91      | Reducible   |
| MLP (6L)        | 6     | 0.68            | 0.74      | Reducible   |
| MLP (12L)       | 12    | 1.14            | 0.61      | Reducible   |
| CNN (8L)        | 8     | 0.87            | 0.68      | Reducible   |
| Transformer (12L)| 12   | 1.42            | 0.52      | Borderline  |
| Transformer (24L)| 24   | 2.05            | 0.39      | Irreducible |

> Values are illustrative. Empirical calibration required for specific architectures.

---

## 🧪 Test Results

```

$ pytest tests/
============================= test session starts =============================
collected 19 items

tests/test_entropy_estimator.py ......... [47%]
tests/test_mutual_information.py ..... [73%]
tests/test_observability.py ...... [100%]

============================= 19 passed in 0.435s =============================

```

---

## 👤 Author

**Samir Baladi**
Independent Interdisciplinary Researcher
Ronin Institute / Rite of Renaissance

- 📧 [gitdeeper@gmail.com](mailto:gitdeeper@gmail.com)
- 🆔 [ORCID: 0009-0003-8903-0029](https://orcid.org/0009-0003-8903-0029)
- 🐙 [GitHub: gitdeeper12](https://github.com/gitdeeper12)
- 🦊 [GitLab: gitdeeper12](https://gitlab.com/gitdeeper12)
- 🏕 [Codeberg: gitdeeper12](https://codeberg.org/gitdeeper12)

---

## 📚 Key References

| # | Reference |
|---|-----------|
| 1 | Sundararajan et al. (2017). Axiomatic attribution for deep networks. *ICML*. |
| 2 | Alain & Bengio (2016). Understanding intermediate layers via linear probes. *arXiv*:1610.01644. |
| 3 | Elhage et al. (2021). A mathematical framework for transformer circuits. *Anthropic*. |
| 4 | Tishby & Schwartz-Ziv (2017). Opening the black box via information. *arXiv*:1703.00810. |
| 5 | Kozachenko & Leonenko (1987). Sample estimate of entropy of a random vector. *PIT*. |
| 6 | Cover & Thomas (2006). *Elements of Information Theory* (2nd ed.). Wiley. |
| 7 | Hinton, G. (2023). Interview. *60 Minutes*, CBS News. |
| 8 | Baladi, S. (2026). ENTRO-OMEGA: Unified Adaptive Stabiliser. DOI: [10.5281/zenodo.19562999](https://doi.org/10.5281/zenodo.19562999). |

---

## 🔗 Links

| Resource | Link |
|----------|------|
| 📄 Zenodo Preprint | [doi.org/10.5281/zenodo.20222840](https://doi.org/10.5281/zenodo.20222840) |
| 📝 OSF Registration | [doi.org/10.17605/OSF.IO/7WP9H](https://doi.org/10.17605/OSF.IO/7WP9H) |
| 📦 PyPI Package | [pypi.org/project/entropath](https://pypi.org/project/entropath) |
| 🐙 GitHub Repository | [github.com/gitdeeper12/ENTRO-PATH](https://github.com/gitdeeper12/ENTRO-PATH) |
| 🦊 GitLab Mirror | [gitlab.com/gitdeeper12/ENTRO-PATH](https://gitlab.com/gitdeeper12/ENTRO-PATH) |
| 🪣 Bitbucket Mirror | [bitbucket.org/gitdeeper-12/ENTRO-PATH](https://bitbucket.org/gitdeeper-12/ENTRO-PATH) |
| 🏕 Codeberg Mirror | [codeberg.org/gitdeeper12/ENTRO-PATH](https://codeberg.org/gitdeeper12/ENTRO-PATH) |
| 🏛️ ENTRO-OMEGA (E-LAB-10) | [doi.org/10.5281/zenodo.19562999](https://doi.org/10.5281/zenodo.19562999) |
| 🔬 OSF Project | [osf.io/yaevt](https://osf.io/yaevt) |
| 📚 Internet Archive | [archive.org/details/osf-registrations-7wp9h-v1](https://archive.org/details/osf-registrations-7wp9h-v1) |
| 🆔 ORCID Profile | [orcid.org/0009-0003-8903-0029](https://orcid.org/0009-0003-8903-0029) |

---

## 📜 License

This project is released under the **MIT License**.
See [LICENSE](LICENSE) for full terms.

The OSF registration is released under **CC-By Attribution 4.0 International**.

---

<div align="center">

*EntropyLab Independent Research Series · May 2026*
*Information Theory · Neural Network Interpretability · Entropy Dynamics*

**Registration DOI: 10.17605/OSF.IO/7WP9H** · **Preprint DOI: 10.5281/zenodo.20222840**

</div>
