Metadata-Version: 2.4
Name: zeta-scaling-toolkit
Version: 0.5.0
Summary: Experimental PyTorch scaling utilities
Author: Zeta Scaling Toolkit Contributors
License: MIT
Project-URL: Homepage, https://github.com/Slangy0-AFK/zeta-scaling-toolkit
Project-URL: Documentation, https://github.com/Slangy0-AFK/zeta-scaling-toolkit#readme
Project-URL: Repository, https://github.com/Slangy0-AFK/zeta-scaling-toolkit.git
Project-URL: Bug Tracker, https://github.com/Slangy0-AFK/zeta-scaling-toolkit/issues
Keywords: machine-learning,deep-learning,transformers,attention,scaling,pytorch,numerical-stability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: hypothesis>=6.100.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: hypothesis>=6.100.0; extra == "test"
Dynamic: license-file

# Zeta Scaling Toolkit

Experimental PyTorch utilities for attention, learning-rate schedules,
coordinates, and linear initialization. Numerical behavior is covered by the
linked tests; none of these components is validated as a training improvement
on large-scale tasks.

## Components

- **`NonCollapsingAttention`** computes `softmax(tau * cos(q, k)) @ v`, where
  `tau` is learnable, starts at `tau0` (default 10), and is bounded to `[tau_min, tau_max]`
  (defaults 1 and 50) using a logit-sigmoid parameterization. This parameterization
  ensures non-zero gradients throughout the bounded range, allowing the temperature
  parameter to recover if perturbed beyond the bounds. Its normalization and softmax
  calculations use at least float32, and the result is cast to `v.dtype`. Boolean
  masks use `True` to retain a position; floating masks are added to the logits.
  Fully masked query rows return zeros. Shapes, dtypes, devices, and optional
  finite-value checks are validated; an empty key sequence raises `ValueError`,
  while empty query sequences and batches are supported. See [attention property tests](./tests/test_properties.py),
  [edge-case tests](./tests/test_edge_cases.py), and
  [regression tests](./tests/test_hardening_regressions.py).
- **`AsymptoticLRScheduler`** is the retained public class name for a schedule
  that linearly warms up, follows cosine decay to `floor_ratio * base_lr`, then
  stays at that floor. `floor_ratio` must be in `[0, 1]`, `total_steps >= 1`,
  and `0 <= warmup_steps < total_steps`. Without warmup it is mathematically
  equivalent to `CosineAnnealingLR` with the same floor inside the configured
  horizon, up to floating-point rounding; unlike stock cosine it stays clamped
  to the floor beyond that horizon. See
  [scheduler property and resume tests](./tests/test_properties.py).
- **`get_isotropic_coords`** returns pixel-centred `(x, y)` coordinates
  normalized by `max(height, width)`. **`get_scale_invariant_vit_coords`**
  returns `(radius, angle)` or, with `smooth_angle=True`,
  `(radius, cos(angle), sin(angle))`. See the
  [coordinate tests](./tests/test_properties.py) and
  [compatibility tests](./tests/test_zeta_toolkit.py).
- **`make_corrected_linear`** applies orthogonal initialization scaled by a
  gain supported by `torch.nn.init.calculate_gain`; its default is ReLU gain.
  GELU and SiLU are not supported as fixed gains. Use **`calibrate_gain`** with
  representative sample data for sequential GELU/SiLU stacks. See
  [initialization properties](./tests/test_properties.py) and
  [calibration regressions](./tests/test_hardening_regressions.py).

Deprecated `A_attn`, `T`, and `A_coords` arguments remain accepted, warn when
non-default values are supplied, and have no effect. Legacy attention
checkpoints containing `q_scale` and `k_scale` load with a warning and retain
the new module's initialized temperature; see
[compatibility tests](./tests/test_zeta_toolkit.py).

## Installation

```bash
pip install -e .
```

For development and tests:

```bash
pip install -e ".[dev]"
```

## Quick start

```python
import torch
from zeta_scaling_toolkit import (
    AsymptoticLRScheduler,
    NonCollapsingAttention,
    get_isotropic_coords,
    make_corrected_linear,
)

attention = NonCollapsingAttention(d_k=64)
q, k, v = (torch.randn(2, 128, 64) for _ in range(3))
output = attention(q, k, v)

linear = make_corrected_linear(1024, 4096, nonlinearity="relu")
optimizer = torch.optim.Adam(linear.parameters(), lr=1e-3)
scheduler = AsymptoticLRScheduler(optimizer, total_steps=300, warmup_steps=10)

coords = get_isotropic_coords(height=224, width=448)
```

## Testing and quality checks

`pytest` runs the full deterministic property and regression suite. Coverage can
be checked with:

```bash
pytest --cov=zeta_scaling_toolkit --cov-report=term-missing --cov-fail-under=95
ruff check zeta_scaling_toolkit tests
mypy --strict zeta_scaling_toolkit
```

Run the optional mutation check with `python tools/mutation_check.py`. It
temporarily injects targeted source changes and runs the full test suite for
each one, restoring the source afterward. `KILLED` means a test detected the
injected bug; `SURVIVED` means the suite missed it. This expensive check is not
part of the default CI matrix.

CI runs the CPU test matrix and a nightly Hypothesis profile. CUDA coverage is
skipped when no CUDA device is available. See [TESTING.md](./TESTING.md).
Reviewers can see the passing full GitHub Actions matrix at commit
[719bfb7](https://github.com/Slangy0-AFK/zeta-scaling-toolkit/commit/719bfb73f97e350087cfefa9617f94571a9d02d1).

## Real-task evaluation

The matched-seed char-level language-model benchmark and its results are
documented in [benchmarks/RESULTS.md](./benchmarks/RESULTS.md). The 100-step
table is early-optimisation only. At 500 steps, only the scheduler-only variant
improved loss on all ten matched seeds; attention-only, initialization-only,
and the combined variant had higher loss. At 1,000 steps over 20 seeds, the
scheduler-only variant improved loss on all 20 matched seeds; attention-only
was worse, while initialization-only and all-components paired intervals
included zero. These results apply to the measured model and dataset only.

## What this can and cannot do

- Opt-in log-length temperature scaling compensates for softmax dilution as
  unmasked key count grows. The synthetic measurement and approximation are
  recorded in [benchmarks/RESULTS.md](./benchmarks/RESULTS.md) and tested in
  [test_properties.py](./tests/test_properties.py). This does not establish a
  language-model training benefit.
- RoPE and ALiBi do not require a learned absolute-position table. Their
  relative-position and numerical properties are covered in
  [test_properties.py](./tests/test_properties.py); extrapolation quality is
  task-dependent and must be measured.
- Context is not infinite. The optional streaming mask exposes a fixed window
  plus sink tokens and cannot recall information outside that window. Its
  visibility set is tested in [test_properties.py](./tests/test_properties.py).
  The mask alone is dense. `StreamingKVCache` supplies a bounded inference-only
  KV cache when used one token at a time; it detaches cached values and retains
  only sinks plus a recent window. Cache retention and agreement with the dense
  mask are tested in [test_properties.py](./tests/test_properties.py).
- Attention uses SDPA when one scale applies to every row and chunks query rows
  for row-specific scales or fully masked rows. Dense masks and ALiBi biases
  still use quadratic memory when constructed for a full sequence. None of
  these mechanisms provide unbounded recall.
- RoPE angles are computed in float64 from int64 positions before casting back
  to the input type. Large positions should not be converted directly to
  float16 or bfloat16; the precision behavior is tested in
  [test_properties.py](./tests/test_properties.py).
- Language-model measurements use one 32-wide model and one small dataset; the
  synthetic vision task used 500 steps and is deliberately simple. Validated
  on a 1,000-step character-level LM and a separate 500-step synthetic vision
  task, but not on large-scale foundation models. These results do not
  generalize to large-scale training; see
  [benchmarks/RESULTS.md](./benchmarks/RESULTS.md).

## Known limitations

- The epsilon in normalization makes scale invariance approximate near zero;
  its relative effect becomes small for vector norms around `1e-2` and larger.
- Attention temperature is bounded to `[tau_min, tau_max]` via a logit-sigmoid
  parameterization; at extreme raw_tau values, sigmoid saturation may produce zero
  gradients, but the parameter can recover for finite perturbations.
- Fixed-temperature configurations (where tau_min == tau_max) are not supported;
  tau bounds must have tau_min < tau_max.
- GELU/SiLU require data-dependent `calibrate_gain`; fixed `calculate_gain`
  does not support them.
- An all-masked attention row returns zeros, not a probability distribution.
- The `(cos(angle), sin(angle))` representation removes the branch cut away
  from the origin; angle remains undefined at the origin.
- For rectangular orthogonal matrices with fewer output than input features,
  norm shrinkage of roughly `sqrt(out_features / in_features)` is structurally
  expected. See [initialization properties](./tests/test_properties.py).
- `calibrate_gain` equalises the whole-batch norm, not each sample's norm. For
  GELU/SiLU in deep stacks without normalisation layers the batch norm can be
  preserved while the median sample's signal collapses (the fixed-seed
  30-layer SiLU characterization test checks for a median per-row ratio below
  0.5). See `test_known_limitation_silu_calibration_*`.
- `calibrate_gain` emits a `RuntimeWarning` when its target ratio is not reached
  within 2% (for example, bounded activations such as tanh cannot reach 1.0).
- `calibrate_gain` is now transactional: if validation fails on any layer, all
  parameters revert to their original values.
- Nothing here has been validated on large-scale training runs.

## License and support

Python >= 3.9 and PyTorch >= 2.0. MIT License; see [LICENSE](./LICENSE).
Experimental results are not a substitute for evaluating these components on
the intended model, data, and training budget.
