Metadata-Version: 2.4
Name: iatro-base-iac
Version: 0.1.4
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: pyarrow
Requires-Dist: numpy
Requires-Dist: brotli ; extra == 'dev'
Requires-Dist: imagecodecs ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Summary: IatroCache (.iac): a lightweight medical data cache format
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/iatrode/iatro-base-iac
Project-URL: Repository, https://github.com/iatrode/iatro-base-iac

# IatroCache

[![CI](https://github.com/iatrode/iatro-base-iac/actions/workflows/ci.yml/badge.svg)](https://github.com/iatrode/iatro-base-iac/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/iatro-base-iac.svg?cacheSeconds=300)](https://pypi.org/project/iatro-base-iac/)

IatroCache (`.iac`) is a lightweight, high-throughput binary cache format for
multimodal medical datasets. It combines Arrow metadata with directly
addressable payload bytes in one immutable file, supporting image tiles,
feature vectors, clinical text, DICOM instances, and project-defined schemas.

The repository contains two public packages:

- `iatro-base-iac`: the v2 container, native reader/writer, record packs, and
  codecs.
- `iatro-iac-adapters`: reusable schemas for tiles, WSI images, teacher
  features, clinical text, paired text, and patient-level DICOM.

## Why IatroCache

- Explicit record boundaries through metadata rather than stream scanning.
- Owned batch reads and an explicit mmap-backed zero-copy API.
- Variable-length records and fixed-width dense matrices in one format.
- Arrow tables for searchable, schema-rich metadata.
- Native checked addressing, I/O, and built-in codecs, with modality policy
  kept in Python adapters.
- Independent per-record compression for random access and data-loader
  concurrency.
- Native Raw, Brotli, Zstandard, JPEG XL, JPEG, JPEG 2000/HTJ2K, and JPEG-LS
  support for clinical text, radiology, and pathology payloads.

## Installation

Install the general container API:

```bash
pip install iatro-base-iac
```

Install the reusable medical-data schemas and matching core dependency:

```bash
pip install iatro-iac-adapters
```

## Quick start

```python
import pyarrow as pa

from iatro.iac import Codec, PackReader, build_pack

slides = pa.table({
    "slide_idx": pa.array([0], type=pa.uint8()),
    "slide_id": ["slide-001"],
    "patient_id": ["patient-001"],
})
items = pa.table({"item_id": ["a", "b"]})

build_pack(
    "example.iac",
    {"payload_type": "raw_bytes", "codec": Codec.NONE},
    slides,
    items,
    [b"first", b"second"],
)

reader = PackReader("example.iac")
try:
    payload = reader.read_payload(1)
    batch = reader.read_payloads([1, 0, 1])
    metadata = reader.index_table
finally:
    reader.close()
```

The writer adds `offset`, `length`, and `crc32` columns for variable records.
Batch reads preserve requested order and duplicates and return owned data.
`read_payload_views()` is the separate, explicit zero-copy interface.

## Parallel batch decoding

`VariableRecordPack` combines row gathering with the codec recorded in the
package header. Its compatibility API returns one Python object per row:

```python
from iatro.iac import VariableRecordPack

pack = VariableRecordPack("notes.iac")
try:
    records = pack.read_many([100, 4, 100], workers=4)
finally:
    pack.close()
```

Native Brotli and Zstandard decoding releases the GIL for the complete batch.
Raw has no decode work and uses an identity fast path. `workers=None` uses
bounded automatic parallelism, `workers=1` is strictly serial, and a larger
positive integer limits native work to that count, available logical CPUs, and
the number of selected records. Input order and duplicates are preserved.
Invalid worker values fail instead of being silently ignored.

Native image codecs keep batch decoding serial until their complete backend
paths have a verified parallel-safety contract. Explicit `workers>1` therefore
fails clearly for JPEG XL, JPEG, JPEG 2000/HTJ2K, and JPEG-LS rather than
pretending to provide concurrency.

## Contiguous decoded batches

Byte-record consumers can avoid constructing and joining thousands of Python
`bytes` objects:

```python
import numpy as np

pack = VariableRecordPack("notes.iac")
try:
    batch = pack.read_many_contiguous([100, 4, 100], workers=4)
finally:
    pack.close()

consume_buffer(batch.buffer)
first = batch[0]  # read-only memoryview, no record copy
offsets = np.frombuffer(batch.offsets_buffer, dtype=np.uint64)

assert batch.record_count == 3
assert tuple(offsets) == batch.offsets
assert batch.lengths[0] == len(first)
```

`ContiguousRecordBatch.buffer` is one owned read-only byte buffer.
`offsets_buffer` is a packed, read-only native-endian `uint64` buffer with
`record_count + 1` entries. The ergonomic `.offsets` tuple is materialized
only when requested, so large native batches do not eagerly allocate one
Python integer per boundary. Empty input, zero-length records, repeated rows,
and arbitrary row order retain exact boundaries. Batch data and views remain
valid after the reader closes.

## Performance

The 2026-08-04 native byte benchmark used macOS on ARM64, Python 3.11.15,
1,024 selected records of 64 KiB each (64 MiB decoded), two warmups, and five
timing rounds. Zstandard used level 3; Brotli used quality 5 and `lgwin=22`.

Decoded end-to-end throughput for the contiguous API:

| Codec | 1 worker | 2 workers | 4 workers | 8 workers |
|---|---:|---:|---:|---:|
| Raw identity | 15.27 GiB/s | 8.97 GiB/s | 14.33 GiB/s | 15.63 GiB/s |
| Zstandard | 3.01 GiB/s | 3.44 GiB/s | 5.33 GiB/s | 5.64 GiB/s |
| Brotli | 0.98 GiB/s | 1.63 GiB/s | 2.65 GiB/s | 3.33 GiB/s |

Raw is a zero-decode identity fast path, so worker differences there are
measurement noise rather than useful scaling. Representative one-worker
stored-byte gathers reached 14.59/19.04 GiB/s for raw, 12.52/22.39 GiB/s for
Zstandard, and 12.03/19.89 GiB/s for Brotli, reported as
`list[bytes]`/contiguous encoded gather respectively.

For one 64 MiB call, contiguous output reduced peak RSS relative to the list
API: approximately 169 vs 185 MiB for raw, 192 vs 207 MiB for Zstandard, and
254 vs 270 MiB for Brotli at one worker. Results vary by allocator and machine.
Accessing the packed buffer for 1,025 offsets took about 0.2 microseconds;
materializing the optional Python tuple took about 8 microseconds. The measured
compressed paths are therefore decode-bound, not offset-organization-bound.

The byte denominator is explicit: stored-read throughput uses selected encoded
bytes; decode and end-to-end throughput use decoded bytes. The benchmark never
divides decoded size by pure read time. Reproduce the complete staged timings,
Python object construction, contiguous assembly, and fresh-process peak RSS:

```bash
conda run -n iac-dev python benchmarks/bench_native_batch_decode.py
```

## Documentation

The [IatroCache Wiki](docs/index.md) covers:

- [design and architecture](docs/design.md);
- [the v2 file format](docs/file-format.md);
- [the complete core API](docs/core-api.md) and [codecs](docs/codecs.md);
- [domain adapters](docs/adapters.md) for
  [pathology data](docs/modalities/pathology.md),
  [clinical text](docs/modalities/clinical-text.md), and
  [DICOM](docs/modalities/dicom.md);
- [validation](docs/validation.md),
  [performance and concurrency](docs/performance.md), and
  [contributing](docs/contributing.md).

## Format at a glance

```text
[ fixed 64 KiB header ] magic + version + JSON layout and schema fields
[ slide table         ] Arrow IPC stream
[ index table         ] Arrow IPC stream
[ data segment        ] variable payloads or a fixed-width matrix
```

The current package version is `0.1.4`; the on-disk format version is `2`.
Those version domains evolve independently.

## License

IatroCache is released under the MIT License.

