Metadata-Version: 2.4
Name: sadcompressor
Version: 0.1.5
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: File Formats
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Archiving :: Compression
Requires-Dist: numpy
Requires-Dist: py-ubjson
Requires-Dist: pyyaml
Requires-Dist: packaging
Requires-Dist: rich
Requires-Dist: isal>=1.8 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'x86_64'
Requires-Dist: termcolor
Requires-Dist: pillow>=10.0 ; extra == 'demo'
Requires-Dist: h5py>=3.16.0 ; extra == 'export'
Requires-Dist: hdf5plugin>=6.0.0 ; extra == 'export'
Requires-Dist: numcodecs>=0.13.1 ; extra == 'export'
Requires-Dist: zarr>=2.18.3 ; extra == 'export'
Requires-Dist: h5py>=3.16.0 ; extra == 'export-lossy'
Requires-Dist: hdf5plugin>=6.0.0 ; extra == 'export-lossy'
Requires-Dist: h5py>=3.10 ; extra == 'hdf5'
Requires-Dist: h5py>=3.16.0 ; extra == 'hdf5-codecs'
Requires-Dist: hdf5plugin>=6.0.0 ; extra == 'hdf5-codecs'
Requires-Dist: pytest ; extra == 'tests'
Requires-Dist: numcodecs>=0.13.1 ; extra == 'zarr'
Requires-Dist: zarr>=2.18.3 ; extra == 'zarr'
Provides-Extra: demo
Provides-Extra: export
Provides-Extra: export-lossy
Provides-Extra: hdf5
Provides-Extra: hdf5-codecs
Provides-Extra: tests
Provides-Extra: zarr
License-File: LICENSE.md
Summary: Streamed Array Data compressor
Keywords: compression,archive,stream
Author-email: Igor Lobanov <lobanov.igor@gmail.com>
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://gitlab.com/alepoydes/sadcompressor.git

# sadcompressor

`sadcompressor` stores simulation time series as compact, time-indexed archives
of NumPy arrays and metadata dictionaries.

It is meant for scientific and engineering simulations where dense fields must
be saved at high time resolution. In many such runs, most of the field changes
slowly while small regions change quickly, for example near topological defects
or sharp localized events. That locality makes the history highly compressible,
but the raw `.npy`/`.npz` output can still become enormous.

SAD stores arrays on a configurable fixed-point grid. Floating-point inputs are
first rounded according to the selected quantization contract; bounded
contracts also clip to their declared integer range. After that, SAD
compression is lossless for the quantized values. Quantized archives can
therefore be copied, repacked, and recompressed without adding more error as
long as the same precision is preserved. Dictionary fields are stored as UBJSON
metadata.

## Why Use It For Simulations

Simulation data often has three useful properties: it arrives as a stream,
adjacent timesteps are related, and historical analysis can often use lower
precision than the simulation used internally. SAD is built around those facts.

- **Keep high time resolution.** When most cells barely change between saved
  steps and only localized regions move quickly, SAD deltas and residual codecs
  can exploit that structure instead of treating every snapshot as independent.
- **Write large runs as they happen.** Writers emit frames incrementally, so a
  simulation can save a long trajectory without holding the whole history in
  memory.
- **Read what you need.** Sequential readers stream full trajectories; random
  readers jump to selected time keys for plotting, diagnostics, or restart-like
  analysis.
- **Choose the analysis precision.** Store the historical archive at an
  explicit fixed-point precision, then preserve that quantized state exactly
  during later repacking.
- **Use Python or the shell.** Write archives from NumPy code, or pack, copy,
  extract, compare, export, and benchmark archives with the `sad` CLI.
- **Move data out when needed.** Export to HDF5 or Zarr for external tools, then
  rebuild SAD archives from exported SAD layouts.
- **Tune speed/size tradeoffs.** Presets such as `fast`, `balanced`, and
  `compact` let you choose between quick writing and smaller archives.

## Is It A Good Fit?

Use `sadcompressor` when:

- your data is a sequence of finite NumPy arrays over logical time;
- you can choose a fixed-point precision appropriate for later analysis;
- neighboring timesteps are correlated, especially with localized fast changes;
- you want to write and process large archives with memory bounded by the
  current frame or selected fields;
- you need both archive compression and later random access to selected frames;
- you want a scriptable CLI for packing snapshots, recompressing old runs, and
  exporting to HDF5/Zarr.

It is probably the wrong tool when:

- original floating-point values must round-trip bit-for-bit exactly;
- `NaN`/`Inf` values must be represented in array payloads;
- you need a general-purpose database or arbitrary HDF5/Zarr schema importer;
- your workload is mostly one-off, uncorrelated arrays where time-series deltas
  cannot help.

Quantized array writers reject `NaN` and `Inf` because the SAD payload format
does not store a validity mask for non-finite values.

## Performance Snapshot

Benchmarks are dataset- and machine-dependent. The real-data snapshot below
uses `tmp/CW_1600_consym_beta_1800.sad`, a 321-key field archive with array
fields `ph`, `x`, `y`, and `z`.

The archive comes from a simulation of a photoactive liquid-crystal film:
`x`, `y`, and `z` are components of the director field, while `ph` is the
concentration of the light isomer of a chiral dopant. This is a typical
smooth-field workload: the director varies coherently in space and time, and
most saved states differ by localized evolution rather than independent noise.
The source SAD file is `17.59 MiB`, while the decoded `float32` arrays exposed
by the public API are `3.13 GB` and the 8-bit quantized logical arrays are
`783.24 MB`.

Compression comparison on that dataset:

| Candidate | Time | Size | Decoded ratio | Quantized ratio |
| --- | ---: | ---: | ---: | ---: |
| Source SAD | - | 17.59 MiB | 169.90x | 42.48x |
| SAD balanced pack | 9.91 s | **16.60 MiB** | **180.00x** | **45.00x** |
| SAD fast pack | **9.42 s** | 21.56 MiB | 138.61x | 34.65x |
| HDF5 zstd15 shuffle 128MiB | 39.64 s | 25.31 MiB | 118.07x | 29.52x |
| Zarr zstd9 shuffle 128MiB | 18.68 s | 25.70 MiB | 116.27x | 29.07x |
| HDF5 zstd22 shuffle 128MiB | 720.18 s | 21.67 MiB | 137.89x | 34.47x |

Read performance on the same source SAD archive:

| Read mode | Payload | Time | Rate |
| --- | ---: | ---: | ---: |
| Sequential `ph` | 51.36 MB | 1.14 s | 282.7 frames/s |
| Sequential `x` | 1.03 GB | 1.45 s | 708.4 MB/s |
| Random all fields, 100-frame forward window | 976 MB | 0.65 s | **1.49 GB/s** |
| Random all fields, 16 keyframes | 156 MB | **0.15 s** | 1.07 GB/s |
| Random all fields, 100 random frames | 976 MB | 3.80 s | 257 MB/s |

See [benchmark results](benchmarks/RESULTS.md) and the
[external archive comparison](benchmarks/ARCHIVE_COMPARISON.md) for the full
tables, environment, commands, and dataset artifact notes. The real benchmark
archive is intentionally kept as an external artifact rather than committed to
git.

## Supported Platforms

`sadcompressor` contains a Rust extension and currently targets Python 3.10+.
Published PyPI wheels currently cover:

- Linux x86-64 glibc-based systems through `manylinux2014` /
  `manylinux_2_17`;
- Windows x86-64 through `win_amd64`.

Other platforms are not part of the published binary wheel set yet. They can be
built from a source checkout with a compatible Python, Rust, and `maturin`
toolchain; see the [Developer documentation](docs/devs/index.md).

## Install

```bash
pip install sadcompressor
```

Optional features are installed with extras:

```bash
pip install 'sadcompressor[hdf5]'
pip install 'sadcompressor[zarr]'
pip install 'sadcompressor[export]'
pip install 'sadcompressor[demo]'
```

## Python Quick Start

The example below simulates a scalar wave amplitude on a 2D periodic grid. A
Gaussian pulse starts at the center, then a finite-difference wave equation
spreads it across the domain. SAD stores the sequence as time keys, quantizes
each grid value to 8 bits, and lets the `compact` preset choose frame codecs for
the smooth spatial and temporal evolution.

![2D wave equation demo](https://gitlab.com/alepoydes/sadcompressor/-/raw/main/docs/assets/wave2d-demo.gif)

```python
import numpy as np
import sadcompressor as sad

# Build a small 2D wave field. This part represents your simulation state.
size = 96
frame_count = 120
axis = np.linspace(-1.0, 1.0, size, dtype=np.float32)
y, x = np.meshgrid(axis, axis, indexing="ij")
u = np.exp(-80.0 * (x**2 + y**2)).astype(np.float32)
u_prev = u.copy()

filename = "wave2d.sad"

# Open a streamed archive. Symmetric 8-bit quantization represents both
# endpoints of this [-1, 1] field.
with sad.SADWriter(
    filename,
    quantization=sad.QuantizationSpec.symmetric(nbits=8, absmax=1.0),
    compression_preset="compact",
) as writer:
    for frame in range(frame_count):
        writer["u"] = u

        # Advance the simulation after storing the current frame.
        neighbors = sum(
            np.roll(u, shift, axis)
            for shift, axis in ((1, 0), (-1, 0), (1, 1), (-1, 1))
        )
        u_prev, u = u, np.clip(
            2 * u - u_prev + 0.35**2 * (neighbors - 4 * u),
            -0.95,
            0.95,
        ).astype(np.float32)
        # Finish this archive key and advance stream time.
        writer.next_key(0.02)

# Stream the archive back. Values are dequantized to NumPy arrays on access.
with sad.SADReader(filename) as reader:
    for frame in reader:
        u = frame["u"]
        print(f"{frame.index:03d} t={frame.t:.2f} max={u.max():+.3f}")
```

Within one time key, each field can be assigned only once. Assign the same field
again after `writer.next_key(dt)`.

This example stores a wave amplitude field with one signed byte per grid node:
`QuantizationSpec.symmetric(nbits=8, absmax=1.0)` maps `[-1, 1]` to integer
codes `[-127, 127]`, so both endpoints and zero are exactly representable. A
full runnable version in [demo/wave2d.py](demo/wave2d.py) also writes the 8-bit
grayscale GIF shown above. By default, generated files go under
`tmp/wave2d-demo/` in the repository checkout; install the `demo` extra for GIF
output.

Running the demo with its defaults (`120` frames, `96x96` grid) gives:

| Representation | Size |
| --- | ---: |
| Raw `float32` arrays | 4.22 MiB |
| 8-bit quantized logical arrays | 1.05 MiB |
| SAD `compact` archive | **92.16 KiB** |
| 8-bit grayscale GIF | 208.31 KiB |

## Compression Choices

Most new workflows should start with one of the stable presets:

| Preset | Use it when |
| --- | --- |
| `fast` | write speed matters and you still want delta compression. |
| `balanced` | you want the default tradeoff for many simulation fields. |
| `compact` | smaller files matter more than compression time. |
| `legacy` | you need compatibility-style Q2/Q3 behavior. |

Experimental presets are available under the `experimental.*` prefix. Users can
optimize archive speed and size for their own datasets by creating custom
presets; `sad predictorstat --suggest-dir` can generate them from measurements
on real data. See [predictorstat](docs/cli-predictorstat.md) and
[compression presets](docs/devs/compression-presets.md).

## Large Archives And Random Access

SAD is designed for large arrays and long time histories. `SADWriter` streams
logical time keys as they are produced, `SADReader` replays the trajectory
sequentially, and `SADRandomReader` opens selected frames for interactive tools,
plotting, diagnostics, and sparse post-processing.

Random access works because built-in presets periodically write complete frames.
To reconstruct a selected state, the random reader starts from the nearest
complete frame and replays only the needed updates. New seekable archives also
include a compact tail index, so random readers can locate indexed frames
without scanning all payloads.

Sequential reading is best when an analysis consumes the whole trajectory. On
the benchmark archive above, reading one `x` field decoded `1.03 GB` in `1.45 s`
(`708.4 MB/s`). Random access is meant for different work: opening a viewer,
jumping to a frame, or reading a short time window without replaying the full
run. On the same archive, reading all fields for a 100-frame forward window took
`0.65 s` (`1.49 GB/s`), and reading 16 key frames took `0.15 s`. Jumping to 100
unrelated frames took `3.80 s` (`257 MB/s`) because more intermediate updates
must be replayed.

```python
with sad.SADRandomReader("wave2d.sad", decode_workers="auto") as reader:
    reader.seek(42)
    u42 = reader["u"]
    print(f"t={reader.t:.2f} max={u42.max():+.3f}")

    reader.seek(100)
    u100 = reader["u"]
    print(f"t={reader.t:.2f} max={u100.max():+.3f}")
```

## CLI Quick Start

Pack numbered NPZ snapshots into one SAD time-series archive:

```bash
sad pack -o trajectory.sad \
  --dt 0.1 \
  --nbits 20 \
  --preset balanced \
  'frames/frame_*.npz'
```

Extract selected SAD frames back to NPZ:

```bash
sad extract trajectory.sad --index-range 0:100:10 -o frames
```

Repack a SAD archive with another compression policy:

```bash
sad pack source.sad -o recompressed.sad \
  --global-raw never \
  --nbits 18 \
  --preset compact
```

Export SAD for external tools:

```bash
sad export trajectory.sad trajectory.h5
sad export trajectory.sad trajectory.zarr --representation quantized --codec zstd
```

Pack exported HDF5/Zarr data back to SAD. This path is still experimental and is
intended for SAD exports with preserved quantized metadata:

```bash
sad pack trajectory.zarr -o restored.sad
```

Full CLI documentation starts at [docs/cli.md](docs/cli.md).

## Documentation

Start with the [documentation index](docs/index.md) if you are not sure which
page you need.

- [Concepts](docs/concepts.md): archive model, precision, random access, and
  Python API boundaries.
- [CLI](docs/cli.md): command overview with links to every command page.
- [sad pack](docs/cli-pack.md): pack snapshots and recompress SAD archives.
- [sad extract](docs/cli-extract.md): export selected frames to NPZ.
- [sad export](docs/cli-export.md): write SAD data to HDF5 or Zarr.
- [Developer documentation](docs/devs/index.md): project internals, builds, and
  releases.

