Metadata-Version: 2.4
Name: luminav
Version: 1.2.1
Summary: A master-free, memory-efficient adaptive optimizer for low-precision deep learning
Author: Lumina Moon
Maintainer: Lumina Moon
License: Apache-2.0
Project-URL: Homepage, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
Project-URL: Organization, https://huggingface.co/cloverx-id
Project-URL: Software DOI, https://doi.org/10.57967/hf/10365
Project-URL: Paper DOI, https://doi.org/10.57967/hf/10270
Project-URL: Bug Tracker, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/discussions
Project-URL: Changelog, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/CHANGELOG.md
Keywords: pytorch,optimizer,low-vram,fp16,bf16,triton,deep-learning,llm,transformer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Provides-Extra: triton
Requires-Dist: triton; extra == "triton"
Dynamic: license-file

---
license: apache-2.0
language:
- en
tags:
- pytorch
- optimizer
- custom-optimizer
- triton
- memory-efficient
- stochastic-rounding
- low-vram
- bf16
- fp16
- adamw
- cuda
- llm
- transformer
- deep-learning
- research
- 3am-engineering
---

<div align="center">
  <h2>LuminaV Optimizer</h2>
  <p><em>We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them</em></p>
  <p><strong>Official Upstream & Standalone Codebase | Current Version: v1.2.1 | Check `Files and Versions`</strong></p>

  <p>
    <a href="https://pypi.org/project/luminav/"><img src="https://img.shields.io/pypi/v/luminav.svg?style=flat-square" alt="PyPI Version" /></a>
    <a href="https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/CHANGELOG.md"><img src="https://img.shields.io/badge/Changelog-v1.2.1-blue?style=flat-square" alt="Changelog" /></a>
    <a href="https://doi.org/10.57967/hf/10365"><img src="https://img.shields.io/badge/Software_DOI-10.57967%2Fhf%2F10365-purple?style=flat-square" alt="Software DOI" /></a>
    <a href="https://doi.org/10.57967/hf/10270"><img src="https://img.shields.io/badge/Paper_DOI-10.57967%2Fhf%2F10270-darkblue?style=flat-square" alt="Paper DOI" /></a>
    <a href="https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/config.json"><img src="https://img.shields.io/badge/Config-JSON-green?style=flat-square" alt="Config JSON" /></a>
    <a href="https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-yellow?style=flat-square" alt="License" /></a>
  </p>

  <table>
    <tr>
      <td align="center" width="100%">
        <h3>Official Research Paper</h3>
        <a href="https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/LuminaV.pdf">
          <img src="https://cdn-uploads.huggingface.co/production/uploads/6835b520186e98712b0386a6/GfVA7g5xUHwopzOSRR5QD.png" alt="LuminaV Paper Preview" width="380" style="border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.15);" />
        </a>
        <br /><br />
        <strong>LuminaV Optimizer Theory & Mechanics</strong><br />
        <a href="https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/LuminaV.pdf">Read LuminaV.pdf (Local Mirror)</a>
        &nbsp;|&nbsp;
        <a href="https://huggingface.co/cloverx-id/XoneLM-1.0-Paper/blob/main/LuminaV.pdf" target="_blank">Primary Paper Archive</a>
      </td>
    </tr>
  </table>
  <p><em>Click the preview above to read or download the official paper PDF.</em></p>
</div>

---

## Notice: Official Upstream Repository

This repository (`cloverx-id/LuminaV-Optimizer-Paper`) is the **official standalone and living development repository** for the LuminaV optimizer family. 

While LuminaV was originally conceived and validated as the core engine for the **[XoneLM-1.0](https://huggingface.co/cloverx-id/XoneLM-1.0-Paper)** language model series, all subsequent optimizer upgrades, low-precision Triton kernels, PyTorch standards compliance, and bug fixes are actively maintained and released directly in this repository.

---

## What's New in v1.2.1 (Latest Release)

The **v1.2.1** release delivers critical GPU pipeline acceleration, eliminates host-device blocking synchronization, and ensures 100% compliance with modern PyTorch compilation and strict determinism:

- **Zero-Sync In-Kernel Parameter Norm Reduction:** Fused the parameter Euclidean norm squared `∑(p_i²)` calculation directly into Pass 1 Triton reduction kernels (`_lumina_v2_pass1_kernel` and `_lumina_v1_pass1_kernel`). Pass 2 and Pass 3 load and resolve `norm_p` entirely within GPU SRAM/registers. Completely eradicated `p_contig.float().norm().item()`, saving **200–400 synchronous PCIe roundtrips and pipeline stalls per training step** on deep Transformer models.
- **CUDA Graphs & `torch.compile` Compatibility:** Eradicated all host-blocking `.item()` calls, resolving fatal TorchDynamo graph breaks and making the optimizer step fully captureable via CUDA Graphs (`torch.accelerator.Graph`).
- **Stateless CPU Golden-Ratio PRNG Hashing:** Replaced dynamic GPU RNG tensor allocation with a pure CPU integer multiplicative hash using Knuth's 32-bit golden ratio constant: `(step * 0x9E3779B9 + i * 10007) & 0x7FFFFFFF`. Generates uncorrelated 31-bit integer seeds per layer in nanoseconds without launching GPU kernels.
- **Strict Deterministic Compliance (`torch.use_deterministic_algorithms`):** Injected dedicated per-device generator caches (`self._generators`) into fallback stochastic rounding (`_sr_update`). Prevents stochastic rounding from mutating PyTorch's default CUDA RNG state, ensuring bit-for-bit reproducibility for RLHF, Dropout, and DataLoader shuffling.
- **Consolidated Batch Scratch Buffer Reset:** Replaced hundreds of per-slice `memset` micro-calls with a single consolidated `scratch.zero_()` reset per step, removing up to 400–800 driver launch overheads per iteration.

  For the full version history and detailed patch notes, see [CHANGELOG.md](https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/CHANGELOG.md).)

---

## Overview

**LuminaV** is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (`FP16` / `BF16`) without maintaining redundant 4-byte FP32 master weights.

By combining **Centered Innovation Variance**, **Hyperbolic Tangent (tanh) Coordinate Bounding**, a **Directional Traffic-Cop Mask**, and **On-Chip Bitwise Stochastic Rounding**, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while avoiding weight freezing and gradient shocks.

---

## Key Features

1. **Zero Master-Weight Copies:** Directly mutates parameter weights in native `FP16` or `BF16`, eliminating the 4-byte FP32 master weight allocation.
2. **On-Chip Bitwise Stochastic Rounding (SR):** Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
3. **Hyperbolic tanh Bounding Envelope:** Maps normalized momentum through a `(-1.0, 1.0)` transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
4. **The Traffic-Cop Directional Gate:** Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (`u_t · g_t ≤ 0`).
5. **Centered Innovation Variance:** Tracks centered innovation dispersion `(g_t - m_t)²` rather than uncentered raw second moments, suppressing variance inflation during confident descent.
6. **Automatic FP16 Cliff Governor:** Built-in asymptotic boundary governor that dampens steps near the IEEE-754 FP16 overflow limit (> 65,504), enabling stable pure FP16 training without external schedulers or clipping.
7. **Dual Execution Engine:** Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ `torch._foreach` multi-tensor fallbacks.

---

## Installation

### From PyPI (Recommended)

```bash
pip install luminav
```

For GPU acceleration via OpenAI Triton:

```bash
pip install luminav[triton]
```

### From Source (Editable Mode)

```bash
git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper
pip install -e .
```

### Direct File Drop-in

Alternatively, you can copy `luminav.py` directly into your working project directory without packaging overhead:

```bash
wget https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/raw/main/luminav.py
```

---

## Quickstart

### Standard Instantiation

```python
import torch
from luminav import LuminaV

# Instantiate your model in native low precision (e.g. BF16 or FP16)
model = YourModel().to(device="cuda", dtype=torch.float16)

# Initialize LuminaV v1.2.1
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,                    # or 8e-5 / 8e-6 for fine-tuning
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    alpha_ss=0.5,
    cautious=True,
    cautious_clamp_min=0.5,     # Exact power-of-two ceiling (2.00x)
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Extreme Low VRAM)
    stochastic_rounding=True,
    bound=True,                 # Smooth asymptotic step bounding
    bound_type="radial",        # "radial" (preserves 100% angular direction) or "coordinate"
    bound_ratio=0.03,
    execution="auto"
)

# Standard training step
optimizer.zero_grad(set_to_none=True)
loss = model(inputs, targets)
loss.backward()
optimizer.step()
```

### Loading from `config.json`

```python
import json
import torch
from luminav import LuminaV

with open("config.json", "r") as f:
    config = json.load(f)

# Initialize with verified default configuration
optimizer = LuminaV(model.parameters(), **config["default_params"])
```

---

## Parameter Reference

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `params` | `iterable` | *Required* | Iterable of parameters to optimize or dicts defining parameter groups. |
| `lr` | `float` | `8e-4` | Learning rate (η). |
| `betas` | `Tuple[float, float]` | `(0.9, 0.999)` | Coefficients (β₁, β₂) for running momentum and centered innovation variance. |
| `eps` | `float` | `1e-8` | Numerical stability term (ε). Automatically floored to `1e-4` in FP16 to prevent subnormal underflow. |
| `weight_decay` | `float` | `8e-2` | Decoupled weight decay coefficient (λ). |
| `tau` | `float` | `0.8` | Analytical bias correction temperature parameter (τ). |
| `alpha_ss` | `float` | `0.5` | Softsign dampening factor (`α_ss`) used in single-buffer mode (`buffer=1`). |
| `cautious` | `bool` | `True` | If `True`, enables Traffic-Cop directional verification masking. |
| `cautious_clamp_min` | `float` | `0.5` | Safety floor density clamp (`γ_min`) enforcing a power-of-two maximum energy scaling ceiling (2.00x, 2¹) and preventing division by zero. |
| `buffer` | `int` | `2` | Buffer mode: `2` (Dual-buffer tracking `m_t` and `v_t`) or `1` (Single-buffer scalar RMS tracking). |
| `stochastic_rounding` | `bool` | `True` | Enables bitwise stochastic rounding on native FP16/BF16 weights. |
| `bound` | `bool` | `True` | If `True`, enables smooth asymptotic parameter bounding to prevent divergence in deep networks. |
| `bound_type` | `str` | `"radial"` | Asymptotic bounding formulation: `"radial"` (direction-preserving squashing using `tanh(r)/r`) or `"coordinate"` (elementwise squashing). |
| `bound_ratio` | `float` | `0.03` | Maximum allowed step displacement ratio relative to parameter norm or magnitude (`R = bound_ratio * ‖p‖`). |
| `execution` | `str` | `"auto"` | Execution engine: `"auto"`, `"triton"`, `"foreach"`, or `"single"`. |

---

## Operational Modes

### LuminaV-2 (Dual-Buffer Default: `buffer=2`)
Maintains first moment m_t and centered innovation variance v_t:

$$
m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t
$$

$$
v_t = \beta_2 v_{t-1} + (1 - \beta_2)(g_t - m_t)^2
$$

Updates are bounded through the hyperbolic tangent envelope:

$$
u_t = \tanh\left(\frac{\tilde{m}_t}{\sigma_t}\right)
$$

### LuminaV-1 (Single-Buffer Extreme-Poverty Mode: `buffer=1`)
Collapses variance tracking into a scalar Root-Mean-Square (RMS) across the entire tensor, saving 50% optimizer state memory by maintaining only a single state buffer (m_t):

$$
\text{RMS}(\tilde{m}_t) = \sqrt{\frac{1}{N} \sum_{i=1}^N \tilde{m}_{t,i}^2 + \epsilon}
$$

$$
u_t = \tanh\left(\frac{z}{1 + \alpha_{ss}|z|}\right), \quad z = \frac{\tilde{m}_t}{\tau \cdot \text{RMS}(\tilde{m}_t) + \epsilon(1 - \beta_1^t)\tau}
$$

---

## Contributors & Acknowledgements

LuminaV is developed and maintained by **Silver Moon ([@cloverxion](https://huggingface.co/cloverxion))** and the **Lumina Moon community contributors**. 

For the complete list of individuals who have contributed code, mathematical analyses, and experimental validation, please refer to [CONTRIBUTORS.md](https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/CONTRIBUTORS.md).

---

## Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

```bibtex
# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion) and Lumina Moon Contributors}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face / PyPI},
  version      = {1.2.1},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}
```

---

## License

Apache License 2.0. See [LICENSE](https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/LICENSE) for full terms.
