Metadata-Version: 2.4
Name: egora
Version: 0.5.2
Summary: EgoRA — Entropy-Governed Orthogonality Regularization for Adaptation. Dynamic information-theoretic regularization for fine-tuning neural networks.
Project-URL: Homepage, https://github.com/ArsSocratica/EgoRA
Project-URL: Documentation, https://github.com/ArsSocratica/EgoRA#readme
Project-URL: Repository, https://github.com/ArsSocratica/EgoRA
Project-URL: Issues, https://github.com/ArsSocratica/EgoRA/issues
Project-URL: Dataset, https://huggingface.co/datasets/ArsSocratica/egora-benchmarks
Project-URL: DOI, https://doi.org/10.5281/zenodo.19410504
Author-email: Mark Dillerop <mark@dillerop.com>
License: AGPL-3.0-or-later
License-File: LICENSE
License-File: LICENSE-ACADEMIC
License-File: NOTICE
Keywords: deep-learning,egora,entropy,fine-tuning,knowledge-preservation,lora,orthogonality,pytorch,regularization,transformer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21
Requires-Dist: torch>=2.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: diagnostics
Requires-Dist: matplotlib>=3.5; extra == 'diagnostics'
Requires-Dist: scipy>=1.7; extra == 'diagnostics'
Description-Content-Type: text/markdown

# EgoRA — Entropy-Governed Orthogonality Regularization for Adaptation

[![PyPI version](https://img.shields.io/pypi/v/egora)](https://pypi.org/project/egora/)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![Dataset on HF](https://img.shields.io/badge/🤗_Dataset-egora--benchmarks-yellow)](https://huggingface.co/datasets/ArsSocratica/egora-benchmarks)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19410504.svg)](https://doi.org/10.5281/zenodo.19410504)

**EgoRA** is a dynamic, information-theoretic regularization method for fine-tuning neural networks. It uses the model's own output entropy to modulate an orthogonality penalty on LoRA adapter weights, preventing knowledge destruction and rank collapse during adaptation.

The method is based on the **Rotation-Retention Law**: knowledge loss in fine-tuned language models is proportional to representational rotation ($\Delta M \propto \bar{\theta}$).

## Installation

```bash
pip install egora
```

With diagnostics support (matplotlib, scipy):

```bash
pip install egora[diagnostics]
```

For development:

```bash
pip install egora[dev]
```

## Quick Start

### Training with EgoRA

```python
from transformers import AutoModelForCausalLM
from egora import apply_lora, get_total_egora_penalty, refresh_all_shadows
from egora import EntropyGovernor

# Load model and apply EgoRA-LoRA adapters
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
lora_modules = apply_lora(model, rank=16, use_egora=True)
gov = EntropyGovernor(alpha=1.359, lam_floor=0.6931)

# Training loop
for batch in dataloader:
    outputs = model(**batch)
    task_loss = outputs.loss

    # Dynamic entropy-governed penalty
    lam = gov.compute_lambda(outputs.logits)
    penalty = get_total_egora_penalty(lora_modules)
    total_loss = task_loss + lam * penalty

    total_loss.backward()
    optimizer.step()
    optimizer.zero_grad()

# Periodically refresh shadow matrices
refresh_all_shadows(lora_modules, momentum=0.9)
```

### Post-Training Diagnostics

```python
from egora import compute_head_geometry

results = compute_head_geometry(base_model, tuned_model)
print(f"Mean rotation: {results['theta_bar_deg']:.2f} deg")
print(f"Damaged heads: {results['damaged_fraction']*100:.1f}%")
```

## Adapter Types

| Adapter | Class | Description |
|---------|-------|-------------|
| **EgoRA-LoRA** | `EgoRALoRALinear` | Standard LoRA + entropy-governed orthogonality penalty |
| **DoRA** | `DoRALinear` | Weight-Decomposed LoRA (Liu et al., 2024) |
| **rsLoRA** | `rsLoRALinear` | Rank-Stabilized LoRA (Kalajdzievski, 2023) |

## Command-Line Interface

EgoRA includes a CLI for training, diagnostics, and visualization — no Python code needed:

```bash
# Fine-tune a model with EgoRA (one command!)
egora train meta-llama/Llama-3.2-1B tatsu-lab/alpaca --epochs 1 --rank 16 --save

# Fine-tune with all options
egora train meta-llama/Llama-3.2-1B my-dataset \
    --rank 16 --lora-alpha 32 --epochs 3 --batch-size 4 --lr 2e-4 \
    --max-length 512 --max-samples 1000 --save --merge

# Compare base vs fine-tuned model
egora diagnose meta-llama/Llama-3.2-1B ./my-finetuned-model -o results.json --plot

# Show model architecture info (layers, d_head, θ_crit, LoRA param estimates)
egora info meta-llama/Llama-3.2-1B

# Launch interactive web demo (requires: pip install gradio)
egora demo
egora demo --share  # public link

# Version
egora version
```

## Visualization

Generate publication-quality rotation geometry plots (requires `pip install egora[diagnostics]`):

```python
from egora.plotting import plot_rotation_report, plot_rotation_heatmap

# Combined 4-panel report: heatmap, layer profile, modes, projections
fig = plot_rotation_report(results)
fig.savefig("rotation_report.png", dpi=150)

# Individual plots
fig = plot_rotation_heatmap(results)     # layer × projection heatmap
fig = plot_layer_profile(results)        # rotation across depth
fig = plot_mode_distribution(results)    # preserved/additive/substitutive/damaged
fig = plot_projection_comparison(results) # Q vs K vs V vs O
```

## HuggingFace Trainer Integration (Recommended)

The easiest way to fine-tune with EgoRA — a drop-in replacement for `Trainer`:

```python
from egora import EgoRATrainer, EgoRATrainingArguments, EgoRALoraConfig

config = EgoRALoraConfig(r=16, lora_alpha=32, use_egora=True)
args = EgoRATrainingArguments(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-4,
)

trainer = EgoRATrainer(
    model_name="meta-llama/Llama-3.2-1B",
    egora_config=config,
    args=args,
    train_dataset=dataset,
)
trainer.train()

# Post-training diagnostics
report = trainer.diagnose()
print(f"Mean rotation: {report['theta_bar_deg']:.2f}°")
print(f"Damaged heads: {report['damaged_fraction']*100:.1f}%")
```

> **Try it now:** [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ArsSocratica/EgoRA/blob/main/notebooks/quickstart_colab.ipynb)

## PEFT-Compatible API

For users familiar with HuggingFace PEFT, EgoRA provides a compatible wrapper:

```python
from egora import EgoRAPeftModel, EgoRALoraConfig

config = EgoRALoraConfig(r=16, lora_alpha=32, use_egora=True)
model = EgoRAPeftModel.from_pretrained("meta-llama/Llama-3.2-1B", config)

model.print_trainable_parameters()
# trainable params: 6,553,600 || all params: 1,235,814,400 || trainable%: 0.53

# One-liner training loss
total_loss = model.compute_total_loss(task_loss, logits)

# Save / load adapters
model.save_pretrained("./my-egora-adapter")
model.load_adapter("./my-egora-adapter")

# Merge into base weights for deployment
base_model = model.merge_and_unload()
```

## Key Concepts

- **Entropy Governor**: Dynamically scales the regularization penalty based on model uncertainty — high entropy (uncertain) → strong penalty, low entropy (confident) → relaxed penalty.
- **Shadow Matrix**: A pseudo-inverse reference that tracks the adapter's structural orientation, enabling measurement of spectral conditioning.
- **Rotation-Retention Law**: $\Delta M \propto \bar{\theta}$ — the empirical law linking representational rotation to benchmark performance change. Critical threshold: $\theta_{\text{crit}} \approx 5°$.
- **Learning Modes**: Per-head classification into additive, substitutive, preserved, or damaged based on rotation angle and magnitude ratio.

## API Reference

### Core Functions

- `apply_lora(model, rank, use_egora, ...)` — Replace attention projections with LoRA adapters
- `get_total_egora_penalty(lora_modules)` — Sum penalty across all adapter modules
- `refresh_all_shadows(lora_modules, momentum)` — Update shadow matrices
- `merge_lora(model, lora_modules)` / `unmerge_lora(replacements)` — Merge/restore adapters

### Governor

- `EntropyGovernor(alpha, lam_floor, ...)` — Dynamic scaling controller
- `GovernorConfig(...)` — Configuration dataclass with adaptive alpha support

### Diagnostics

- `compute_head_geometry(base_model, tuned_model)` — Per-head rotation analysis
- `compute_interhead_diversity(model)` — Head diversity matrix
- `save_geometry(results, output_path)` — Export results to JSON + numpy

## Examples

See the [`examples/`](https://github.com/ArsSocratica/EgoRA/tree/main/examples) directory:

- **`quickstart_training.py`** — End-to-end fine-tuning with EgoRA + post-training diagnostics
- **`diagnostics_only.py`** — Standalone rotation analysis between any two checkpoints
- **`peft_compat_example.py`** — PEFT-style API demo with save/load/merge

## Resources

| | Link |
|---|---|
| 📦 **PyPI** | [`pip install egora`](https://pypi.org/project/egora/) |
| 💻 **GitHub** | [ArsSocratica/EgoRA](https://github.com/ArsSocratica/EgoRA) |
| 🤗 **Dataset** | [ArsSocratica/egora-benchmarks](https://huggingface.co/datasets/ArsSocratica/egora-benchmarks) — benchmark results, rotation geometry, training curves |
| 🔖 **DOI** | [10.5281/zenodo.19410504](https://doi.org/10.5281/zenodo.19410504) |

### Benchmark Dataset

The full experiment data is published on HuggingFace:

- **Llama 3.2 1B / 3B, Llama 3.1 8B** — Alpaca + Medical domain, 4 methods (Baseline LoRA, DoRA, EgoRA e², EgoRA adaptive v2)
- **Cross-modal** — Mistral-7B, Phi-3 Mini
- **Rotation geometry** — per-head θ, learning modes, knowledge maps, alignment landscapes
- **Threshold analysis** — Rotation-Retention Law validation, dimensionality threshold, phase transition

```python
# Load with HuggingFace datasets
from datasets import load_dataset
ds = load_dataset("ArsSocratica/egora-benchmarks")
```

## Citation

If you use EgoRA in your research, please cite:

```bibtex
@software{dillerop2026egora,
  title={The Rotation-Retention Law: Knowledge Loss Is Proportional to 
         Representational Rotation in Fine-Tuned Language Models.
         With EgoRA: Entropy-Governed Orthogonality Regularization 
         for Adaptation},
  author={Dillerop, Mark},
  year={2026},
  doi={10.5281/zenodo.19410504},
  url={https://zenodo.org/records/19410504}
}
```

## License

This software is licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0)**, with an **Additional Permission for Academic Use** pursuant to AGPL Section 7.

- **Academic use**: Free, no copyleft obligations, citation required. See [LICENSE-ACADEMIC](https://github.com/ArsSocratica/EgoRA/blob/main/LICENSE-ACADEMIC).
- **Commercial use**: Requires both a software license and a patent license.

### Patent Notice

The methods implemented in this software are covered by U.S. Provisional Patent Application No. 64/024,742 ("Entropy-Governed Orthogonality Regularization for Knowledge-Preserving Neural Network Adaptation and Rotation-Retention Diagnostic Framework"), filed April 1, 2026, by Mark Dillerop.

Academic use is permitted without a separate patent license. Commercial use requires both a software license and a patent license.

**Contact**: mark@dillerop.com
