Metadata-Version: 2.4
Name: iatro-base-iac
Version: 0.1.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: pyarrow
Requires-Dist: imagecodecs
Requires-Dist: numpy
Requires-Dist: brotli ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: pylibjpeg-libjpeg>=2.1,<3 ; extra == 'dicom'
Provides-Extra: dev
Provides-Extra: dicom
License-File: LICENSE
Summary: IatroCache (.iac): a lightweight medical data cache format
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/iatrode/iatro-base-iac
Project-URL: Repository, https://github.com/iatrode/iatro-base-iac

# iatro-base-iac

**IatroCache (`.iac`)** — a lightweight, high-performance binary container format for offline caching of multimodal medical datasets (image tiles, feature vectors, clinical text, expert tokens, etc.).

This package provides the core container format and readers/writers for `.iac` files. It is designed for high-concurrency training pipelines, with explicit owned-byte and memory-mapped (`mmap`) zero-copy read APIs.

---

## Installation

```bash
pip install iatro-base-iac
```

## Python API

```python
from iatro.iac import Codec, PackReader, build_pack
```

## Format Layout

```
[ fixed header        ] 65536 bytes  — magic "IATROC", JSON header (codec, payload_type, offsets, ...)
[ slide table         ] Arrow IPC    — slide_idx / slide_id / patient_id
[ index table         ] Arrow IPC    — caller-defined columns + offset / length / crc32
[ data segment        ] raw bytes    — concatenated payloads, indexed by the index table
```

### Key Technical Features
*   **Explicit Boundaries & Integrity**: Each record carries `offset` / `length` / `crc32`. Payload boundaries are explicit, eliminating the need to scan for framing markers. This works seamlessly for codecs without self-delimiting frames (e.g. raw Brotli).
*   **High Performance**: `PackReader.read_payload_views()` exposes zero-copy, lock-free mmap views for batch decoding, while `read_payload()` and `read_payloads()` always return owned `bytes`.
*   **Metadata Flexibility**: `payload_type` and `codec` are free-form header fields; the low-level container does not interpret the payload bytes directly.

---

## Quick Start (Core API)

Below is an example of writing raw bytes to an `.iac` pack and reading them back:

```python
import pyarrow as pa
from iatro.iac import build_pack, PackReader

# 1. Define metadata tables
slide_table = pa.table({
    "slide_idx": pa.array([0], pa.uint8()),
    "slide_id": ["s0"], 
    "patient_id": ["p0"]
})

# Offset, length, and crc32 columns are populated automatically
index_table = pa.table({
    "item_id": ["item_a", "item_b"]
})

# 2. Build the cache file
build_pack(
    output_path="out.iac",
    header_json={"payload_type": "raw_bytes", "codec": Codec.NONE},
    slide_table=slide_table,
    index_table=index_table,
    payloads=[b"first_payload_data", b"second_payload_data"]
)

# 3. Read payloads concurrently
reader = PackReader("out.iac")
print(reader.read_payload(1))  # Output: b"second_payload_data"

# Stable owned-data API: always list[bytes]
owned = reader.read_payloads([0, 1])

# Explicit zero-copy API: always list[memoryview], or raises if mmap is unavailable
views = reader.read_payload_views([0, 1])
reader.close()
```

For large-scale or streaming datasets, refer to `build_pack_streaming`, `build_pack_data_segment`, and `build_pack_data_segment_from_file`.

Pack-level codecs use one public interface. Native byte codecs are selected
without a Python implementation wrapper, while user codecs may inherit
`Codec` and register a reconstruction factory:

```python
codec = Codec.create(Codec.ZSTD, level=3)
```

Reusable concrete schemas are distributed separately in
`iatro-iac-adapters`. The adapter package depends on this core, so one command
installs both:

```bash
pip install iatro-iac-adapters
```

---

## Contributing & License

This project is licensed under the MIT License. Codec and Pack contributions are welcome.

