Metadata-Version: 2.4
Name: sadcompressor
Version: 0.1.3
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: File Formats
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Archiving :: Compression
Requires-Dist: numpy
Requires-Dist: py-ubjson
Requires-Dist: packaging
Requires-Dist: rich
Requires-Dist: isal>=1.8 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'x86_64'
Requires-Dist: termcolor
Requires-Dist: pytest ; extra == 'tests'
Provides-Extra: tests
License-File: LICENSE.md
Summary: Streamed Array Data compressor
Keywords: compression,archive,stream
Author-email: Igor Lobanov <lobanov.igor@gmail.com>
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://gitlab.com/alepoydes/sadcompressor.git

# sadcompressor

`sadcompressor` is a compact archival format and Python library for streamed
time-series data, mainly NumPy arrays. It stores logical time keys with
quantized full frames, deltas, optional prediction, dictionaries, and a tail
index for fast random access.

Array compression is lossy by design. The reconstructed values are guaranteed
by the configured quantization precision, not by bitwise equality with source
arrays.

## Quick Start

```python
import numpy as np
import sadcompressor as sad

filename = "example.sad"

with sad.SADWriter(filename, prec_nbits=20, prec_maxexp=8) as writer:
    writer["x"] = np.array([1.0, 2.0, 3.0], dtype=np.float32)
    writer["meta"] = {"step": 0}

    writer.next_key(0.1)
    writer["x"] = np.array([1.1, 2.1, 3.1], dtype=np.float32)
    writer["meta"] = {"step": 1}

with sad.SADReader(filename) as reader:
    while not reader.next_key():
        print(f"t={reader.t:.3f}", reader["x"], reader["meta"])
```

Within one time key, each field can be assigned only once. Assign the same
field again after `writer.next_key(dt)`.

## Random Access

`SADWriter` writes a tail index into the final `EndFrame` by default. New
archives therefore open quickly through `SADRandomReader` without scanning all
frame descriptors.

```python
with sad.SADRandomReader(filename, decode_workers="auto") as reader:
    print(reader.nkeys)
    print(reader.timestamps)
    print(reader.list_arrays())

    reader.seek(1)
    print(reader.t, reader["x"])
```

For older archives without an index, `SADRandomReader` falls back to the old
descriptor scan while skipping compressed array payloads. If the file is
writable, it persists a tail index so the next random open is fast.

Use `persist_index="never"` to guarantee that opening for random access does
not modify the file:

```python
with sad.SADRandomReader(filename, persist_index="never") as reader:
    ...
```

Use `index_policy="require"` when tools should fail instead of scanning:

```python
with sad.SADRandomReader(filename, index_policy="require") as reader:
    ...
```

`SADWriter(..., write_index=False)` disables writing the tail index.

## Archive Inspection

`open_archive()` exposes the logical archive structure without reading
compressed array payloads:

```python
archive = sad.open_archive(filename, persist_index="never")

print(archive.header)
print(archive.index_status)
print(archive.structure.nkeys)
print(archive.structure.array_fields)
print(archive.structure.dict_fields)
```

It uses the tail index when available and falls back to descriptor scanning
according to `index_policy`.

## Compression Backends

The zlib-compatible backend is selected automatically. If `isal` is installed
on the current platform, it is used by default; otherwise `sadcompressor` falls
back to Python's stdlib `zlib`.

```python
from sadcompressor.codec import get_zlib_backend, set_zlib_backend

set_zlib_backend("isal")      # or "stdlib", or "auto"
print(get_zlib_backend())
```

The same choice can be made before import with:

```bash
SADCOMPRESSOR_ZLIB_BACKEND=stdlib python script.py
```

Compression levels are backend-specific:

- stdlib zlib: `-1..9`
- isal: `0..3`

```python
with sad.SADWriter(
    filename,
    prec_nbits=20,
    prec_maxexp=8,
    compression_level=2,
    encode_workers="auto",
) as writer:
    ...
```

`encode_workers` controls parallel compression of independent array payloads.
`decode_workers` controls parallel decoding of independent arrays in a frame.
Both accept `"auto"` or a positive integer.

## Command Line

Full user documentation: [docs/cli.md](docs/cli.md).

The package installs one main entry point:

```bash
sad info example.sad
sad dump example.sad
sad copy source.sad recompressed.sad --nbits 18 --fullframe 20
sad pack -o trajectory.sad --dt 0.1 'frames/frame_*.npz'
sad extract trajectory.sad --index-range 0:100:10 -o frames
```

Compatibility aliases are kept:

```bash
sadinfo example.sad
sadump example.sad
sadcopy source.sad recompressed.sad
```

Useful copy options:

```bash
sad copy source.sad out.sad \
  --zlib-backend isal \
  --compression-level 2 \
  --encode-workers auto \
  --nbits 20 \
  --fullframe 20 \
  --prediction \
  --packbits
```

## Benchmarks

Benchmark helpers are grouped under `sad bench`.

Generate a scalar 2D benchmark archive:

```bash
sad bench generate \
  --kind oscillating \
  --field u \
  --shape 512 512 \
  --frames 200 \
  --output tmp/bench_oscillating_2d.sad \
  --overwrite \
  --zlib-backend isal \
  --encode-workers auto
```

Generate a production-like normalized 3D vector field:

```bash
sad bench generate \
  --kind vector-normalized-3d \
  --field v \
  --shape 20 200 200 \
  --frames 200 \
  --null \
  --ui json
```

Read sequentially or by random access:

```bash
sad bench read-seq tmp/bench_oscillating_2d.sad \
  --field auto \
  --decode-workers auto \
  --ui json

sad bench read-random tmp/bench_oscillating_2d.sad \
  --pattern all \
  --read-fields all \
  --decode-workers auto \
  --ui json
```

`--ui` accepts `auto`, `interactive`, `plain`, `json`, or `quiet`.

