Metadata-Version: 2.4
Name: luminav
Version: 1.1.5.post1
Summary: A master-free, memory-efficient adaptive optimizer for low-precision deep learning
Author: Lumina Moon
Maintainer: Lumina Moon
License: Apache-2.0
Project-URL: Homepage, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
Project-URL: Organization, https://huggingface.co/cloverx-id
Project-URL: Software DOI, https://doi.org/10.57967/hf/10365
Project-URL: Paper DOI, https://doi.org/10.57967/hf/10270
Project-URL: Bug Tracker, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/discussions
Project-URL: Changelog, https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/blob/main/CHANGELOG.md
Keywords: pytorch,optimizer,low-vram,fp16,bf16,triton,deep-learning,llm,transformer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Provides-Extra: triton
Requires-Dist: triton; extra == "triton"
Dynamic: license-file

# LuminaV Optimizer

> **We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them**

[![PyPI - Version](https://img.shields.io/pypi/v/luminav.svg?style=flat-square)](https://pypi.org/project/luminav/)
[![Software DOI](https://img.shields.io/badge/Software_DOI-10.57967%2Fhf%2F10365-purple?style=flat-square)](https://doi.org/10.57967/hf/10365)
[![Paper DOI](https://img.shields.io/badge/Paper_DOI-10.57967%2Fhf%2F10270-darkblue?style=flat-square)](https://doi.org/10.57967/hf/10270)
[![Python Version](https://img.shields.io/badge/python-3.10+-blue?style=flat-square)](https://pypi.org/project/luminav/)
[![License](https://img.shields.io/badge/License-Apache%202.0-yellow?style=flat-square)](LICENSE)

---

## Official Repository & Research Paper

- **Official Repository & Issue Tracker:** [https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper](https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper)
- **Primary Research Paper Archive:** [https://huggingface.co/cloverx-id/XoneLM-1.0-Paper](https://huggingface.co/cloverx-id/XoneLM-1.0-Paper)
- **Software DOI:** [10.57967/hf/10365](https://doi.org/10.57967/hf/10365)
- **Paper DOI:** [10.57967/hf/10270](https://doi.org/10.57967/hf/10270)

---

## What's New in v1.1.5 (Latest Release)

The **v1.1.5** release introduces the **Zero-VRAM FP16 Numerical Safety Shield**, resolving early pretraining instability and eliminating `NaN` collapses in pure half-precision training without allocating 4-byte master weights:

- **Dynamic Epsilon Floor for FP16:** Automatically floors effective epsilon to `max(eps, 1e-4)` for `torch.float16` parameters, preventing the bias-corrected scale factor `c2` from underflowing into IEEE-754 subnormal/zero limits (~5.96e-8) and eliminating step-1 `0.0 / 0.0 = NaN` errors on zero-gradient parameters (e.g., unselected vocabulary tokens, padding, or dropout paths).
- **On-Chip Second-Moment Clamping:** Added an upper-bound register ceiling (`65,000.0`) to the centered innovation variance in Triton kernels and PyTorch loops before storing into FP16 pointers. This prevents unclipped initial gradient spikes (`|g| > 256`) from exceeding the FP16 maximum dynamic range (65,504) and permanently saturating state buffers to `+inf`.
- **Zero-Sigma Division Guard:** Embedded an explicit `sigma + 1e-6` protection across all Triton kernel passes and PyTorch fallbacks to ensure division safety during cold-start iterations.
- **Native Hardware Capability Detection:** Integrated `_supports_native_bf16()` to automatically detect Nvidia Ampere SM80+ architectures for optimal native execution.
- **Empirically Proven on Tesla T4:** Empirically verified by pretraining `SmolLM-135M` from scratch on Wikimedia Wikipedia in pure FP16 on an Nvidia Tesla T4 GPU for 100 consecutive steps with zero `NaN` occurrences—even with external gradient clipping completely disabled.

---

## Overview

**LuminaV** is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (`FP16` / `BF16`) without maintaining redundant 4-byte FP32 master weights.

By combining **Centered Innovation Variance**, **Hyperbolic Tangent (tanh) Coordinate Bounding**, a **Directional Traffic-Cop Mask**, and **On-Chip Bitwise Stochastic Rounding**, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while preventing weight stagnation and numerical explosions.

---

## Key Features

1. **Zero Master-Weight Copies:** Directly mutates parameter weights in native `FP16` or `BF16`, eliminating the redundant 4-byte FP32 master weight allocation.
2. **On-Chip Bitwise Stochastic Rounding (SR):** Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
3. **Hyperbolic tanh Bounding Envelope:** Maps normalized momentum through a `(-1.0, 1.0)` transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
4. **The Traffic-Cop Directional Gate:** Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (`u_t · g_t <= 0`).
5. **Centered Innovation Variance:** Tracks centered innovation dispersion `(g_t - m_t)^2` rather than uncentered raw second moments, suppressing variance inflation during confident descent.
6. **Dual Execution Engine:** Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ `torch._foreach` multi-tensor fallbacks.

---

## Installation

Install directly via PyPI:

```bash
pip install luminav
```

For GPU acceleration via OpenAI Triton:

```bash
pip install luminav[triton]
```

Or install in editable mode from source:

```bash
git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper
pip install -e .
```

---

## Quickstart

```python
import torch
import torch.nn as nn
from luminav import LuminaV

# 1. Instantiate model directly in native half precision (e.g. BF16 or FP16)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = nn.Linear(1024, 1024).to(device=device, dtype=torch.float16)

# 2. Initialize LuminaV
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Low VRAM)
    stochastic_rounding=True,
    execution="auto"
)

# 3. Standard training loop
data = torch.randn(32, 1024, device=device, dtype=torch.float16)
target = torch.randn(32, 1024, device=device, dtype=torch.float16)
criterion = nn.MSELoss()

optimizer.zero_grad(set_to_none=True)
output = model(data)
loss = criterion(output, target)
loss.backward()

optimizer.step()
```

---

## Parameter Reference

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `params` | `iterable` | *Required* | Iterable of parameters to optimize or dicts defining parameter groups. |
| `lr` | `float` | `8e-4` | Learning rate (eta). |
| `betas` | `Tuple[float, float]` | `(0.9, 0.999)` | Coefficients (beta1, beta2) for running momentum and centered innovation variance. |
| `eps` | `float` | `1e-8` | Numerical stability term (epsilon). Automatically floored to `1e-4` in FP16 to prevent subnormal underflow. |
| `weight_decay` | `float` | `8e-2` | Decoupled weight decay coefficient (lambda). |
| `tau` | `float` | `0.8` | Analytical bias correction temperature parameter (tau). |
| `alpha_ss` | `float` | `0.5` | Softsign dampening factor (alpha_ss) used in single-buffer mode (`buffer=1`). |
| `cautious` | `bool` | `True` | If `True`, enables Traffic-Cop directional verification masking. |
| `cautious_clamp_min` | `float` | `0.2` | Safety floor density clamp (gamma_min) preventing division by zero in masked normalization. |
| `buffer` | `int` | `2` | Buffer mode: `2` (Dual-buffer tracking m_t and v_t) or `1` (Single-buffer scalar RMS tracking). |
| `stochastic_rounding` | `bool` | `True` | Enables bitwise stochastic rounding on native FP16/BF16 weights. |
| `execution` | `str` | `"auto"` | Execution engine: `"auto"`, `"triton"`, `"foreach"`, or `"single"`. |

---

## Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

```bibtex
# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face},
  version      = {1.1.5},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}
```

---

## License

Apache License 2.0. See [LICENSE](LICENSE) for full terms.
