Metadata-Version: 2.4
Name: steklov-activations
Version: 0.1.0
Summary: Compact-support piecewise-polynomial activation functions with tunable sparsity
Author-email: Aleksandr Masalskikh <alex@masalskikh.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/masalskikh/steklov-activations
Project-URL: Paper, https://arxiv.org/abs/XXXX.XXXXX
Keywords: deep-learning,activation-functions,sparsity,pruning,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.0
Provides-Extra: triton
Requires-Dist: triton>=2.1; extra == "triton"
Provides-Extra: all
Requires-Dist: triton>=2.1; extra == "all"

# Steklov Activations

Compact-support piecewise-polynomial activation functions with tunable sparsity.

**Paper:** [*Steklov Activations: Piecewise-Polynomial Gates with Compact Support and Tunable Sparsity*](https://zenodo.org/records/19454642)


## What is this?

SteklovSiLU is a drop-in replacement for GELU/SiLU/ReLU that produces **exact zeros** outside a finite support region. One parameter `α` controls the support width:

- `α = 2.0` → approximates GELU (sup error < 0.0091)
- `α = 0.8` → 28–64% of activations are exactly zero per token
- `α = 0.1` → 87% zeros, 98.4% natural NVIDIA 2:4 compliance
- `α = 0.005` → 90% zeros, 99.2% compliance, quality matches SiLU

The exact zeros enable structural pruning (permanently remove dead neurons) and hardware-accelerated N:M sparse inference.

## Install

```bash
pip install steklov-activations

# With Triton kernel support
pip install steklov-activations[triton]
```

## Quick Start

### Drop-in activation replacement

```python
from steklov_activations import SteklovSiLU

# GELU-like (α=2.0, sup error < 0.0091)
act = SteklovSiLU(alpha=2.0)

# High sparsity
act = SteklovSiLU(alpha=0.1)

# Learnable α (network finds its own sparsity level)
act = SteklovSiLU(alpha=2.0, learnable=True)
```

### Swap activations in an existing architecture

```python
from transformers import AutoConfig, AutoModelForCausalLM
from steklov_activations import swap_activation

# Initialize model with random weights
config = AutoConfig.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_config(config)

# Replace all GELU activations with SteklovSiLU
swap_activation(model, alpha=0.8)
# Replaced 12 activation(s) with SteklovSiLU(alpha=0.8, order=3)

# Train from scratch (post-hoc swap on pretrained models
# does NOT produce meaningful sparsity)
```

### Profile inactivity

```python
from steklov_activations import InactivityProfiler

profiler = InactivityProfiler(model)

# Run calibration data through the model
for batch in calibration_loader:
    with torch.no_grad():
        model(**batch)

# Get report
report = profiler.report(threshold=0.999)
print(report.summary())
# Inactivity Report (50,000 tokens, threshold=99.9%)
#   Dynamic sparsity (per-token zeros): 64.4%
#   Structural sparsity (permanently dead): 9.0%
#   ...

profiler.remove_hooks()
```

### Prune dead neurons

```python
from steklov_activations import prune_inactive_neurons

# Remove permanently dead neurons from weight matrices
pruned = prune_inactive_neurons(model, report)
# Total: pruned 2,800 neurons across 12 layers

# Model is now physically smaller and faster
# Save the pruned model
model.save_pretrained("my-model-pruned")
```

## Key Results (from paper)

| Setting | Result |
|---------|--------|
| Vision (CIFAR-10/100, 5 seeds) | Steklov α=0.8 wins all 4 benchmarks |
| GPT-2 Small (124M, 5 seeds) | 22.39 PPL vs GELU 22.97 (p=0.30) |
| GPT-2 Medium (354M, 3-4 seeds) | Matches GELU within seed variability |
| LLaMA 105M (3 seeds) | All variants beat SiLU |
| LLaMA 105M α=0.005 (1 seed) | 90% zeros, PPL better than dense SiLU |
| Pruning 7% of 354M neurons | +0.05 PPL (magnitude pruning: model collapse) |
| N:M compliance at α≤0.1 | 98–99% natural 2:4; SiLU: 0% |
| Inference after pruning | 3–5% faster than unpruned GELU |

## Important Notes

- **Train from scratch.** Post-hoc activation swap on pretrained models does not produce meaningful sparsity. The network must train with compact support to develop the inactivity patterns.
- **Single α is sufficient.** A single global α shared across all layers works as well as per-layer α.
- **Architecture matters.** GPT-2-style models produce permanently dead neurons (good for pruning). LLaMA-style models produce per-token zeros with no permanent death (good for N:M hardware sparsity).

## Citation

```bibtex
@article{masalskikh2026steklov,
  author  = {Masalskikh, A.},
  title   = {Steklov Activations: Piecewise-Polynomial Gates with Compact Support and Tunable Sparsity},
  journal = {Zenodo},
  year    = {2026},
  doi     = {10.5281/zenodo.19454642},
  url     = {https://doi.org/10.5281/zenodo.19454642}
}
```

## License

Apache 2.0
