Metadata-Version: 2.4
Name: eltq
Version: 1.12.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Dist: numpy>=2.1
Requires-Dist: scipy>=1.14 ; extra == 'reference'
Provides-Extra: reference
License-File: LICENSE
Summary: Ferociously fast ELT secondary-uncertainty quantization
Keywords: catastrophe modeling,event loss table,secondary uncertainty,aggregate distribution,reinsurance,actuarial
Author-email: "Stephen J. Mildenhall" <mynl@me.com>
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/mynl/eltq/blob/master/CHANGELOG.md
Project-URL: Repository, https://github.com/mynl/eltq

# eltq — ferociously fast ELT secondary-uncertainty quantization

Converts a portfolio event loss table (ELT, ~1M rows) into two discrete, evenly
spaced, rate-weighted loss-grid distributions:

- **raw** — each event contributes an atom of rate λ at its conditional mean μ;
- **secondary** — each event's conditional loss is modeled as a beta on
  [0, M] matching (μ, σ = σ_I + σ_C) and approximated by K equally weighted
  midpoint quantiles x_k = M·Q(α, β; (k+½)/K).

Both are mapped to the grid g_i = i·h by a linear deposit rule that preserves
mass and first moment exactly. The mathematical contract is
`secondary-uncertainty-mathematical-specification.md`; the implementation plans
are `dev/done/plan-implementation.md` (1.0.0-1.5.0) and
`dev/done/plan-multiunit.md` (1.6.0-1.11.0: joint multi-unit secondary
uncertainty, unit terms, view totals).

## Install

```
pip install eltq                # or: uv add eltq
pip install "eltq[reference]"   # adds SciPy for the reference backend
```

Wheels are built for Python 3.12 and later on Windows x86_64, Linux x86_64
(manylinux) and macOS arm64; no Rust toolchain is needed. The only runtime
dependency is NumPy. The reference backend (`eltq.reference`,
`eltq.reference_multiunit`) is the SciPy oracle the test suite checks the Rust
engine against; it also serves as the fallback when the compiled extension is
missing.

The x86_64 wheels target the `x86-64-v3` instruction set (AVX2: Intel Haswell,
2013, and AMD Excavator, 2015, onward). On an older CPU the import fails with an
illegal-instruction error; build from source instead, which needs a Rust
toolchain:

```
pip install --no-binary eltq eltq
```

Setting `RUSTFLAGS="-C target-cpu=native"` for that build tunes it to the
machine. On the multi-unit route that is about 8% faster than the wheel on a
Zen 5 CPU; elsewhere the two are equally fast, and every output is bitwise
identical.

## Status

| Milestone | Version | State |
| --- | --- | --- |
| [reference] SciPy oracle, golden fixtures, acceptance suite | 1.0.0 | done |
| [rust-core] eltq-core scalar path | 1.1.0 | done |
| [python] PyO3 bindings + CSV route | 1.2.0 | done |
| [memmap] quantize_into / quantize_to_npy | 1.3.0 | done |
| [ferocious] parallelism + batched kernel + benchmarks | 1.4.0 | done |
| [fallback] SciPy backend in-package + demo notebook | 1.5.0 | done |
| [sigma-rule] total sd = σ_I + σ_C (Burke 2016), outright | 1.6.0 | done |
| [reference-n-unit] n-unit reference, terms grammar, views | 1.7.0 | done |
| [joint-two-unit] Rust bivariate joint pmf | 1.8.0 | done |
| [terms-grammar] Rust parser + view-map compiler | 1.9.0 | done |
| [multiunit-views] per-event engine, `multiunit_views` | 1.10.0 | done |
| [ferocious-multiunit] own erf, real FFTs, cross-view sharing, 1M unit-row report | 1.11.0 | done |
| [pypi-release] wheels on PyPI: Windows, Linux, macOS; CI against the oracle | 1.12.0 | done |

## Layout

```
crates/eltq-core/     pure Rust core (no PyO3/NumPy/CLI types)
crates/eltq-python/   PyO3 + numpy bindings
python/eltq/          pure-Python layer: API wrappers, SciPy reference/fallback
                      (one-unit: reference.py; multi-unit: reference_multiunit.py),
                      DecL-style unit terms (terms.py)
tests/                pytest acceptance suite (runs both backends)
fixtures/             synthetic ELTs + golden outputs (never real data)
benchmarks/           bench harness + machine-readable results history
notebooks/            demo notebook (eltq_demo.ipynb)
dev/                  plans (moved to dev/done when complete)
.github/workflows/    CI: build and test wheels, publish to PyPI (wheels.yml)
```

## Environment (uv)

`uv sync` creates `.venv` in the checkout and builds the extension into
`./target`; both are gitignored. The repository sets no compiler flags, so a
source build targets the baseline instruction set of the Rust target. To tune
local builds to the machine, set the flag in a cargo config outside the
repository, which cargo reads for every project below it (the author's is
`V:\dev\.cargo\config.toml`), or in `RUSTFLAGS`, which overrides it:

```toml
[build]
rustflags = ["-C", "target-cpu=native"]
```

uv does not rebuild when that file changes; force it with
`uv sync --reinstall-package eltq`.

## Commands

```powershell
uv sync                                        # build wheel (maturin) + deps;
                                               # rebuilds when Rust sources change
uv run pytest                                  # acceptance suite (reference + eltq)
uv run python fixtures/generate_fixtures.py    # regenerate fixtures + goldens
cargo test -p eltq-core                        # Rust unit + property tests
cargo bench -p eltq-core                       # criterion kernel benchmarks
uv run python benchmarks/run_end_to_end.py     # 1M-row end-to-end benchmark;
                                               # appends benchmarks/results.json
uv run python benchmarks/run_multiunit.py      # 1M unit-row multi-unit benchmark
                                               # (100k events x 10 units); appends
uv run python benchmarks/z_convergence.py      # shared-factor quadrature study
uv run --with jupyter --with matplotlib jupyter lab
                                               # demo: notebooks/eltq_demo.ipynb
```

## Performance

Headline (i7-7820HQ 4C/8T, Windows 11, see `benchmarks/report-1.4.0.md`):
one million mixed-regime rows at K = 32 quantized in ~4.3 s — ~230k rows/s,
~6.6M beta quantiles/s, ~44x the SciPy reference — with bitwise-identical
results for any thread count. Non-kernel overhead (validation + deposits +
reduction) is ~0.02 s; the inverse-beta kernel is everything. Grid length is
irrelevant to throughput (each deposit touches two nodes whatever n is):
n = 1,001, n = 65,536 (2^16) and n = 100,000 all run at the same speed.

Determinism contract: for a fixed input, config (including `chunk_rows`,
default 65536) and build, output arrays are bitwise reproducible regardless
of `threads` and work stealing. Reordering input rows changes results only
within floating-point summation tolerance.

Multi-unit route (`benchmarks/report-1.11.0.md`, same laptop): ~0.91M unit
rows (100k events x 10 units, typical term sparsity) with all four views at
Q = 32 in 22.5 s, ground_up only in 15.2 s, Q = 8 in 7.9 s; bitwise identical
across thread counts (whole-event chunks, default 2048 rows); the two-unit
joint at reference accuracy (Q = 1001) in 10 ms. These figures are for local
builds with `target-cpu=native` (see Environment), so rebuild after moving
machines. On an AMD Ryzen 9 9950X3D2 (Zen 5, 16C/32T) the 1M-row headline takes
0.92 s, and the `x86-64-v3` wheel matches a native build except on the
multi-unit route, where it is 8% slower (`dev/done/plan-pypi-release.md`).

## Python API

```python
import eltq

raw, secondary, meta = eltq.quantize_arrays(
    rate, max_loss, mean_loss, sd_indep, sd_corr,
    *, bucket_size, grid_length=None, quadrature_points=32,
    normalize=False, mean_is_loss_rate=False,
    variance_policy="clamp", clamp_factor=0.999, event_ids=None)

raw, secondary, meta = eltq.quantize_csv(path, *, bucket_size,
    columns={"rate": "annual_rate"}, ...)      # optional column-name mapping

raw, secondary, meta = eltq.quantize_dataframe(df, *, bucket_size, ...)

meta = eltq.quantize_into(source, *, raw_out, secondary_out, ...)
    # writable float64 arrays incl. np.memmap; their length defines the grid

meta = eltq.quantize_to_npy(source, "grids.npy", *, bucket_size, ...)
    # transactional (2, n) .npy + grids.meta.json; np.load(..., mmap_mode="r")
```

`source` for the last two: a CSV path, a DataFrame with canonical columns, or
a `(rate, max_loss, mean_loss, sd_indep, sd_corr)` tuple.

Arrays are borrowed zero-copy (contiguous float64; the wrapper converts
explicitly otherwise), compute releases the interpreter, and results hand
Rust-owned buffers to NumPy without a copy. Errors raise `eltq.EltqError` or
its subclasses `VarianceError` / `GridOverflowError`. `meta` is a `Meta`
dataclass with the fields listed under Output below.

**Backends.** `eltq.BACKEND` reports which implementation serves the API:
`"rust"` (the compiled extension) or `"reference"` — the pure-Python SciPy
implementation in `eltq.reference`, used automatically when the extension is
not importable (requires scipy; ~40x slower; ignores `threads`/`chunk_rows`).
`eltq.reference.quantize_reference` is also importable directly as the
independent oracle the acceptance suite tests both backends against.

## Multi-unit route (1.7.0)

For ELTs at unit level (event x unit rows) the units of one event share a
common factor (Burke 2016, section III.E): per unit the conditional loss is a
beta on [0, M] with total sd σ_I + σ_C; one shared standard normal Z_c per
event occurrence and an independent Z_i per unit give a one-factor Gaussian
copula, and the units are independent given Z_c. `dev/secondary-uncertainty-
additivity.md` proves why this cannot be reduced to rolled rows, so unit-level
rows are kept and per-unit terms applied before summation.

```python
views, meta = eltq.multiunit_views(
    event_id, unit_id, rate, mean_loss, sd_indep, sd_corr, max_loss,
    *, terms, bucket_size, grid_length=None,
    views=("ground_up", "gross", "ceded", "net"),
    z_nodes=32, z_rule="midpoint", normalize=False,
    variance_policy="clamp", clamp_factor=0.999, roll_up_diagnostic=False)
# views: dict view -> rate-weighted array on the grid i * h; meta: MultiMeta

pmf, x1, x2, meta = eltq.joint_two_units(
    (mu1, sd_i1, sd_c1, M1), (mu2, sd_i2, sd_c2, M2),
    *, bucket_size, n1, n2, z_nodes=1001, z_rule="midpoint")
```

Rows of one event must be contiguous with one row per unit and a constant
rate (`MultiUnitError` otherwise). Terms are DecL-style, one line per unit
(`eltq.parse_terms` / `eltq.terms`):

```
unit plant_a 750 xs 250 occurrence net of 50% po 400 xs 100 and 500 xs 500
unit plant_b 1000 xs 0 occurrence net of 100 po 400 xs 100
unit fleet
```

`ground_up` = X; `gross` = min(max(X − ded, 0), limit); `ceded` = Σ share ·
min(max(gross − attach, 0), layer) over the clauses (each applied to gross;
overlaps allowed as long as the combined share stays ≤ 100% in every band, so
ceded never exceeds gross); `net` = gross − ceded. A percent share is the fraction of
the layer; a plain number is an amount of the layer limit (`100 po 400 xs
100` cedes 25%). Breakpoints must be multiples of `bucket_size`. Units
without a line get no terms. The shared factor is integrated by midpoint
quantile nodes (default) or Gauss-Hermite; `benchmarks/z-convergence-1.7.0.md`
records the convergence study behind the default. Both functions run in Rust
on the compiled backend (`joint_two_units`: conditional matrices plus one
GEMM, 1.8.0; `multiunit_views`: the per-event engine of 1.10.0, whole-event
chunks folded bitwise-deterministically like `quantize`, direct or FFT
convolution across units chosen by cost or forced with `convolution=`) and
on the SciPy reference `eltq.reference_multiunit` otherwise.
`multiunit_views_into(..., out={view: array})` writes into caller-provided
(memmap-capable) arrays, the keys selecting the views and the length the grid.

## Input schema

Canonical columns (CSV route allows a column-name mapping from 1.2.0):

| column | symbol | meaning |
| --- | --- | --- |
| `event_id` | e | optional identifier (row index fallback), audit only |
| `rate` | λ | annual event rate / nonnegative measure weight |
| `max_loss` | M | beta upper bound (maximum possible loss) |
| `mean_loss` | μ | conditional mean loss (or a rate on [0, 1] with `mean_is_loss_rate`) |
| `sd_indep` | σ_I | independent std-dev component |
| `sd_corr` | σ_C | correlated std-dev component |

Admissibility: λ, M, σ_I, σ_C ≥ 0 and 0 ≤ μ ≤ M. Total conditional sd is the
arithmetic sum σ = σ_I + σ_C (Burke, Rau-Chaplin, Varghese 2016, section
III.E; the root-sum-of-squares rule used before 1.6.0 was incorrect). The
model behind the two components, and why unit-level rows must be kept when
unit-level terms exist, is in `dev/secondary-uncertainty-additivity.md`.

## Row handling and policies

Structurally invalid rows are dropped and reported (never silently): nonfinite
fields, negative fields, μ > M. Zero-rate rows carry no mass and are skipped.
Degenerate rows (σ = 0, μ = 0, μ = M, or M = 0) contribute a single atom at μ.

Rows with inadmissible beta variance (σ² ≥ μ(M − μ)) follow `variance_policy`:

- `clamp` (default) — σ clamped to 0.999·√(μ(M − μ)), reported;
- `error` — typed error;
- `drop` — row dropped from both outputs, reported;
- `atom` — collapsed to an atom at μ, reported.

Diagnostics report per-category counts plus up to 10 representative event ids:
`nonfinite`, `negative_field`, `mean_exceeds_max`, `zero_rate`,
`degenerate_atom`, `sigma_clamped`, `variance_dropped`, `variance_atom`.

Grid coverage requires (n − 1)·h ≥ max contributing M; with `grid_length=None`
the length is auto-derived as ceil(max M / h) + 1. An insufficient explicit
grid raises `GridOverflowError` — no silent truncation, no overflow bin.

## Output

Two length-n `float64` arrays (weights at losses i·h; rate-weighted by
default, `normalize=True` divides both by the total contributing rate Λ) plus
metadata: h, n, K, normalized flag, Λ, analytic AAL, both output masses and
first moments, the quadrature secondary mean and its finite-K error vs the
AAL, row counts, and the diagnostics above.

## Open questions (spec §11)

- Confirm the real file's exposure field is exactly the beta upper bound M.
- `mean_is_loss_rate` scales only μ by M (per spec). If real ELTs give the
  standard deviations as damage ratios too, that convention needs a decision.

