Metadata-Version: 2.4
Name: loaderx
Version: 2.5.4
Summary: Rebuildable high-performance ordered record containers
Author-email: Ben0i0d <ben0i0d@foxmail.com>
License-Expression: MIT
Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
Project-URL: Source, https://codeberg.org/eoelab/loaderx
Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
Keywords: flax,python,dataloader
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: msgpack
Provides-Extra: converter
Requires-Dist: datasets[vision]>=2.19; extra == "converter"
Dynamic: license-file

# Loaderx
Zrecord is a rebuildable, typed, ordered record sequence built from authoritative
source data and scripts. A creator consumes records in append order, ``close``
publishes one immutable container, and readers support scalar access, natural
iteration, slices, and indexed gather without changing record identity. A scalar
read returns the record itself, with no batch wrapper. To change content or
order, rebuild it at a new path.

Zrecord is the typed on-disk container; Loaderx is the sampler and prefetch
loader that consumes Zrecord streams. They currently ship together while both
layers mature, but their public responsibilities remain separate.

```
pip install loaderx
```

Wheels are published for CPython 3.10+ on glibc Linux (x86-64 and ARM64), Apple
Silicon macOS, and Windows AMD64. Free-threaded CPython 3.14 is also supported.

## Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. NumPy-native typed records with explicit schemas.
4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. **Dense and ragged are separate contracts.** Dense batches are arrays;
   ragged batches are lists of arrays. Loaderx never pads either representation.
6. **Logical IDs are stable sequence positions.** Append order defines
   ``0..N-1``; published containers are immutable.

## Usage

### Quick Start
```python
import numpy as np

from loaderx.zrecord import Dense
from loaderx.dataloader import DataLoader

data = np.load('data.npy', mmap_mode='r')
label = np.load('label.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:], codec='zstd') as ds:
    ds.append(data)
with Dense.create('train_label', label.dtype, label.shape[1:], codec='zstd') as ds:
    ds.append(label)

data_store = Dense.open('train_data')
label_store = Dense.open('train_label')
loader = DataLoader({'data': data_store, 'label': label_store},
                    transform=lambda batch: batch)
for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

loader.close()
data_store.close()
label_store.close()
```

A batch is a dict ``{name: values}``: each value is the stacked
``(batch_size, *item_shape)`` array for that stream.  Every stream is gathered
at the same indices, so record ``i`` lines up across them.  The ``transform``
callback is the collate step — reshape, cast, stack — where ``values`` is the
plain dense batch ready for the model.

### Records
`Dense` stores one fixed-shape array per record. A scalar read or one step of
iteration returns the record itself; a multi-record read returns a stacked
array:

```python
import numpy as np
from loaderx.zrecord import Dense

data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
with Dense.create('data', data.dtype, data.shape[1:], codec='zstd') as ds:
    ds.append(data)
with Dense.open('data') as ds:
    record = ds[0]                    # ndarray (2, 4), no leading batch axis
    batch = ds[0, 5, 2]               # ndarray (3, 2, 4), requested order retained
    for record in ds:                 # each record is ndarray (2, 4)
        consume(record)
```

A store is an ordered sequence of records, not an ndarray, so ``ds[0, 5, 2]``
selects sequence positions 0, 5 and 2 — never ``ds[0][5][2]``. A scalar selects
one unwrapped record, while any collection of indices selects a batch. Indices
must be in ``0..len(ds)-1``. The ragged example below reads the same way.

`Ragged` stores variable-shape arrays with one shared `dtype` and `ndim`.
Multi-record reads return a list; scalar reads and iteration return one array.
Every record's exact shape is restored and every axis may vary; scalar records
use Dense with `item_shape=()`. Padding remains an application policy:

```python
from loaderx.zrecord import Ragged

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
with Ragged.create('tokens', np.int32, ndim=1, codec='zstd') as rs:
    rs.append(seqs)                       # dtype/rank fixed; lengths remain per-record
with Ragged.open('tokens') as rs:
    record = rs[2]                    # ndarray (4,), not a one-element list
    records = rs[0, 2, 4]             # list of ndarray, exact per-record shapes
    for record in rs:                 # each item is one shape-restored ndarray
        consume(record)

lengths = np.array([len(r) for r in records])
padded = np.zeros((len(records), lengths.max()), dtype=records[0].dtype)
for i, r in enumerate(records):
    padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop
```

Logical ID is the stable sequence position. Each `append` preserves input order,
and successive calls extend the sequence. After `close`, the sequence is
immutable: there is no delete, update, compaction, or reopen-for-append.

### Creating containers

`Dense.create` and `Ragged.create` return append-only builders. The required
`codec` is `"raw"`, `"zstd"`, or `"zstd_dict"`. Schema, shape, and dtype are
explicit and are never inferred from the input. Dense accepts the complete
NumPy array or memory map; Ragged accepts the complete finite iterable of NumPy
arrays. Pass the complete source to one `append`; the native Zig writer bounds
its scheduling and scratch memory internally, so callers do not need to slice
the source into memory-control batches.

A creator consumes a finite build stream and publishes an immutable sequence;
readers do not tail an active writer. Append is synchronous and returns after
the complete input has been persisted.

```python
from loaderx.zrecord import Dense, Ragged

with Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28),
                  codec='zstd', data_shards=4) as ds:
    ds.append(images)

with Ragged.create('tokens', dtype=np.int32, ndim=1, codec='zstd') as tok:
    tok.append(sequences)

with Dense.open('mnist/x') as ds:
    first_four = ds[:4]               # opened containers are read-only
```

`data_shards` controls write parallelism, accepts `1..255`, and defaults to four.
Readers discover it automatically.

Append errors are reported by the current call. A writer cannot be read and a
reader cannot be appended to. Writer `close()` publishes the container; reader
`close()` releases it. Dense, Ragged, and DataLoader support `with`.

Structured, subarray, object, metadata-bearing, and zero-itemsize dtypes are not
supported. Append does not coerce Python containers or bytes; convert them with
`np.asarray` or `np.frombuffer` first. Raw bytes and encoded files can be stored
as `np.uint8` Ragged records.

### Codec notes

`"zstd"` compresses each record independently with plain zstd (level 3). Use it
for general-purpose compression; it is fast and must be selected explicitly.

`"zstd_dict"` uses a shared dictionary trained from representative settled data.
It is useful for large corpora of small, similar records such as token sequences
and image tiles. Plain `"zstd"` or `"raw"` is usually better for large records
and small corpora.

Train the dictionary manually with `train_dict` on settled data, then pass its
bytes to `dict_bytes`. The source may be a NumPy array, an iterable of records,
or an existing typed container. `zstd_dict` never trains automatically:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense, Ragged

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array)

# then any new store can install it and append explicitly
with Ragged.create('tokens', np.int32, ndim=1,
                   codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(token_generator)
with Dense.create('data', data.dtype, data.shape[1:],
                       codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(data)
```

Changing an existing store's codec requires writing a new store:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense

with Dense.open("src") as s, \
     Dense.create("dst", dtype=s.dtype, item_shape=s.item_shape,
                       codec="zstd_dict", dict_bytes=train_dict(s)) as d:
    d.append(s[:])                             # ndarray -> dense append
```

For Ragged, `s[:]` already returns the list accepted by `append`; create the
destination with `s.dtype` and `s.ndim`. The destination path must be new.

**Important:** Train the dictionary from settled authoritative input before
building the container. Training it before preprocessing is complete wastes
compression and does not describe the final records.

### Offline Hugging Face conversion

The optional converter uses Hugging Face Datasets for remote discovery,
download, caching, revision handling and source-format decoding. It then writes
explicit typed Zrecord streams, so training needs neither `datasets` nor Arrow:

```bash
pip install 'loaderx[converter]'
```

```python
from loaderx.converter import convert, huggingface

dataset = huggingface("ylecun/mnist", revision="main", token=None)
convert(
    dataset,
    "mnist",
    codec="zstd",
)
```

Mainland China or a private Hub can select an endpoint directly without setting
process environment variables:

```python
dataset = huggingface(
    "ylecun/mnist",
    endpoint="https://hf-mirror.com",
)
```

`huggingface` resolves the requested revision to an immutable snapshot and
returns its `DatasetDict`. Pass `config=...` when a repository has no default
configuration, and `token="hf_..."` for private or gated repositories.

`convert` preserves numeric dtypes and record shapes. Fixed-shape columns become
Dense stores and variable-shape columns become Ragged stores. Unsupported
features fail explicitly; conversion does not filter, pad, tokenize, reshape,
or apply a transform.

The result groups aligned streams under one published root:

```text
mnist/
  train/
    image/
    label/
  test/
    image/
    label/
```

The output is published only after all columns have been written and their
record counts agree. Each column remains an ordinary Zrecord store.

### DataLoader and multi-stream stores

A zrecord store is one stream; a training sample is usually several named
streams (skeleton + label + id, tokens + label, ...). Composition is a plain
Python ``dict`` passed to ``DataLoader``. There is no persistent wrapper,
manifest, directory convention or bundle mutation API. ``DataLoader`` verifies
that all streams have the same length, then gathers every stream with the same
indices.

```python
from loaderx.zrecord import Dense, Ragged
from loaderx.dataloader import DataLoader

root = "xsub/train"
with Dense.create(root + "/joint", joint.dtype, joint.shape[1:], codec="zstd") as s:
    s.append(joint)
with Dense.create(root + "/label", label.dtype, label.shape[1:], codec="zstd") as s:
    s.append(label)
with Ragged.create(root + "/token", np.int32, ndim=1, codec="zstd") as s:
    s.append(seqs)

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
streams["joint"][0, 5, 2]            # each stream keeps its own index API
loader = DataLoader(streams, batch_size=256)
batch = next(loader)                  # {name: values}, index-aligned

loader.close()
for stream in streams.values():
    stream.close()
```

Equal-length and variable-length are both just stores — `Dense` (one
fixed-shape record per sample) and `Ragged` (variable row count per
record). The loader does not interpret either: it fetches by index and packs a
dict, so a dense stream's batch value is the stacked `(B, *item_shape)` array
and a ragged stream's is a list of per-record arrays. No padding is imposed —
densify to a fixed shape however the model needs (a plain numpy loop), or
reshape/stack in the `transform` collate.

Collation is the `transform`: a batch dict in, an arbitrary result out.

```python
def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

loader = DataLoader({'tokens': dense_tokens, 'label': labelset}, batch_size=32,
                    transform=collate)
batch = next(loader)
loader.close()
```

The transform runs once per gathered batch. Calls may run concurrently and
complete out of order, so the callback must be thread-safe. Its return value is
passed to the consumer unchanged, and exceptions propagate to the consumer.

Numba can optionally accelerate a CPU-heavy Dense transform while releasing the
GIL. Compile it before timing, then call it from the ordinary transform:

```python
import numba
import numpy as np

@numba.njit(nogil=True, parallel=False)
def normalize_u8(x):
    out = np.empty(x.shape, dtype=np.float32)
    for i in range(x.size):
        out.flat[i] = x.flat[i] / 255.0
    return out

normalize_u8(np.zeros((1, 3, 224, 224), dtype=np.uint8))  # compile warmup

def transform(batch):
    batch["image"] = normalize_u8(batch["image"])
    return batch
```

Numba is optional. Compile it before measuring loader throughput.

### CPU → GPU transfer

Loaderx produces CPU batches. Device transfer belongs in `transform`, where the
training framework is already available:

```python
import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_numpy(v).to(device, non_blocking=True)
            for k, v in batch.items()}

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
}
loader = DataLoader(streams, transform=to_device)
for batch in loader:
    model(batch)                        # already on device
```

Call `loader.close()` when the training loop exits, then close its caller-owned
streams. Do not close either from `transform`.

A `non_blocking=True` copy is asynchronous only from pinned memory. Loaderx does
not pin memory; use the framework's API in the transform when needed:

```python
def to_device(batch):
    return {k: torch.from_numpy(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}
```

For JAX, use `jax.device_put`:

```python
import jax

def to_device(batch):
    return {k: jax.device_put(v) for k, v in batch.items()}
```

For a practical JAX/Flax integration, see the
**[MNIST layer-representation example](examples/mnist_latent/)**.

### Zsampler
Zsampler is the batch index generator used by DataLoader. A run resumes exactly
with `seek(step)`, without tracking an epoch.

```python
from loaderx.zsampler import Sampler

sampler = Sampler(1_000_000, 256, Sampler.Mode.IID, seed=42)
indices = sampler.next()       # borrowed until this sampler's next draw
saved = indices.copy()         # retain across draws only when needed
```

`next()` and iteration return a view of one reusable `uint64` batch buffer.
The contents stay unchanged until the next explicit draw from that Sampler;
copy only plans that must outlive it. DataLoader consumes each view synchronously
before drawing again.

1. **Sequential** traverses records in order and wraps at the end.
2. **IID** samples uniformly with replacement.
3. **Cyclic** traverses a fresh permutation without replacement. It does not
   allocate a dataset-sized permutation. Full batches are returned; a different
   remainder is omitted on each cycle when the length is not divisible by the
   batch size.

## Benchmarks

Dense and Ragged are measured separately because they expose different
contracts, but every store table uses the same columns. `scripts/bench_dense.py`
measures fixed-shape random gather, `scripts/bench_ragged.py` measures
variable-shape records, and `scripts/bench.py` covers machine, sampler, and the
end-to-end loader comparison. Every path runs through the public Python binding,
including output allocation and Ragged reconstruction.

### Methodology

The tables are one complete pass on a warm page cache, refreshed on 2.5.1 with
the default `data_shards=4`. They are not three-run medians. Every Store backend
receives the same prepared real records, then a full sweep and independent IID
batches verify exact dtype, shape, order, and values before timing. Loader
backends use the same source and seed but their own shipped samplers, so exact
permutations may differ.

`logical write` and `logical gather` report uncompressed NumPy payload bytes per
elapsed second, not physical storage bandwidth. Write timing includes public-API
adaptation and logical finalization, but excludes source preparation, dictionary
training, writer setup, cleanup, and stable-media sync. Zrecord writers receive
the complete prepared source in one append and bound execution internally.
Gather timing includes
allocation, reads, decompression, and reconstruction; it excludes open, close,
sampler time, and destruction after return. Fresh uniform IID gathers accumulate
at least two timed seconds. `krecords/s` is record throughput, `p95` is one
random batch's 95th-percentile latency, and `disk` is allocated blocks. Compare
results within a workload table, not across different record geometries. Every
listed backend and codec is required for the published matrix.

The vision source is the Oxford-IIIT Pet train split prepared by
`scripts/prepare_vision.py`. Dense uses RGB photographs resized on the short side
to 256 and center-cropped to `(3, 224, 224)`; Ragged uses the same ordered source
at native RGB resolution. Both are mmap-loaded uint8 CHW records.

**Machine** — one local workstation (AMD Ryzen AI 9 HX PRO 370, 12 cores / 24
threads):

| machine   | value |
|-----------|-------|
| CPU       | AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M, 1 socket, 12 cores / 24 threads |
| frequency | 605–5158 MHz |
| caches    | L1d 576 KiB, L1i 384 KiB, L2 12 MiB, L3 24 MiB |
| NUMA      | 1 node |
| memory    | 31 GiB (not limited by cgroup) |
| shared memory | 16 GiB `/dev/shm` |
| OS        | Debian GNU/Linux forky/sid, kernel 7.1.8+deb13-amd64, x86_64 |
| python    | CPython 3.14.7 (standard GIL build), numpy 2.5.2 |

The process sees all 24 threads, is not memory-limited by cgroup, and uses the
ordinary page cache. The 16 GiB shared-memory mount accommodates the Torch run.

### Large Vision Records

#### Fixed-Shape Dense

Zrecord against array-store alternatives: random batch gather over the first
2,500 Oxford-IIIT Pet train images, batch 256. Every prepared image is
`uint8[3,224,224]`; `metadata.json` records the source revision, transform,
decoder versions and logical SHA-256.

Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 4493 MiB/s | 15501 MiB/s | 108.0 | 2.67 ms | 358.9 MiB | 1.00x |
| npy-mmap-raw | 1814 MiB/s | 4989 MiB/s | 34.8 | 8.77 ms | 358.9 MiB | 1.00x |
| hdf5-raw | 2562 MiB/s | 2095 MiB/s | 14.6 | 20.72 ms | 359.0 MiB | 1.00x |
| lmdb-raw | 1582 MiB/s | 4244 MiB/s | 29.6 | 10.25 ms | 361.4 MiB | 0.99x |
| arrow-ipc-raw | 1038 MiB/s | 3752 MiB/s | 26.1 | 11.01 ms | 358.9 MiB | 1.00x |
| parquet-raw | 1085 MiB/s | 451 MiB/s | 3.1 | 90.42 ms | 358.9 MiB | 1.00x |
| arrayrecord-raw | 1625 MiB/s | 3280 MiB/s | 22.8 | 13.25 ms | 359.2 MiB | 1.00x |
| zrecord-zstd | 1698 MiB/s | 6912 MiB/s | 48.1 | 6.01 ms | 311.1 MiB | 1.15x |
| zrecord-zstdict | 1460 MiB/s | 8138 MiB/s | 56.7 | 5.42 ms | 320.1 MiB | 1.12x |
| hdf5-gzip | 47 MiB/s | 246 MiB/s | 1.7 | 161.55 ms | 301.8 MiB | 1.19x |
| arrow-ipc-zstd | 240 MiB/s | 106 MiB/s | 0.7 | 364.25 ms | 305.9 MiB | 1.17x |
| parquet-zstd | 232 MiB/s | 89 MiB/s | 0.6 | 463.77 ms | 305.9 MiB | 1.17x |
| arrayrecord-zstd | 221 MiB/s | 2461 MiB/s | 17.1 | 16.81 ms | 320.1 MiB | 1.12x |

At 147 KiB per record, Zrecord-raw reaches 15.1 GiB/s and is 3.1x npy-mmap-raw.
Decoded photographs have little remaining redundancy: plain zstd reduces them
only 1.15x, while dictionary mode falls to 1.12x. This is
why large real images should use plain zstd or raw. LMDB and Arrow
IPC are competitive raw record
stores, while codecs tied to whole IPC batches or Parquet row groups pay read
amplification on random gathers. Dense demonstrates that
typed record ownership and per-record compression do not turn fixed tensors
into an object-store slow path.

#### Native-Resolution Ragged

This workload contains the first 2,500 native-resolution Oxford-IIIT Pet CHW RGB
images. Height is 108..2606 (median 375) and width is 117..3264 (median 500),
totaling 1252.3 MiB of logical uint8 payload. Each of 50 random
batches contains 256 records. Every backend persists payload plus exact shape
and must return an ordered `list[np.ndarray]` of `(3, H, W)` arrays; a flat byte
list or a one-dimensional variable-length abstraction is not enough.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 5900 MiB/s | 13834 MiB/s | 27.7 | 14.13 ms | 1252.4 MiB | 1.00x |
| hdf5-raw | 1978 MiB/s | 2292 MiB/s | 4.6 | 69.15 ms | 1253.1 MiB | 1.00x |
| lmdb-raw | 1833 MiB/s | 9540 MiB/s | 19.1 | 20.17 ms | 1257.0 MiB | 1.00x |
| arrow-ipc-raw | 1293 MiB/s | 4683 MiB/s | 9.3 | 38.50 ms | 1252.4 MiB | 1.00x |
| parquet-raw | 647 MiB/s | 433 MiB/s | 0.9 | 359.87 ms | 1252.4 MiB | 1.00x |
| arrayrecord-raw | 1735 MiB/s | 2812 MiB/s | 5.6 | 61.46 ms | 1253.0 MiB | 1.00x |
| zrecord-zstd | 1562 MiB/s | 5130 MiB/s | 10.2 | 43.79 ms | 1032.2 MiB | 1.21x |
| zrecord-zstdict | 918 MiB/s | 5102 MiB/s | 10.2 | 45.26 ms | 1032.9 MiB | 1.21x |
| arrayrecord-zstd | 210 MiB/s | 2010 MiB/s | 4.0 | 96.52 ms | 1054.6 MiB | 1.19x |

Zrecord-raw is 1.5x LMDB and 3.0x Arrow IPC in logical gather. Plain and
dictionary zstd both reach 1.21x, confirming that dictionary mode is not useful
for these large photographs. The Ragged matrix is intentionally asymmetric:
HDF5 gzip, Arrow IPC zstd and Parquet zstd adapters are not implemented because
the real 256-record batch took 1.1–1.7 seconds; TileDB variable queries took
8.6–9.6 seconds and the backend was removed entirely. ArrayRecord zstd remains
because its batch p95 is 97 ms.

### Small Token Records

Both token workloads come from the same real corpus: WikiText-103 raw train,
tokenized with GPT-2 and stored as `int32` IDs. Preparation is outside every
measurement. `scripts/prepare_tokens.py` preserves nonempty text boundaries,
combines fragments shorter than 16 tokens, and splits records at 512 tokens into
`tokens.npy` plus `offsets.npy`; both benchmark scripts mmap those files.

#### Fixed Token Blocks

The Dense workload ignores text boundaries and packs the stream into 200,000
fixed `int32[512]` records: 2 KiB per record and 390.6 MiB logical payload.
Each IID batch gathers 256 records; fresh draws continue until timed gathers
accumulate at least two seconds.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 4300 MiB/s | 7552 MiB/s | 3866.6 | 0.09 ms | 393.7 MiB | 0.99x |
| npy-mmap-raw | 1873 MiB/s | 9909 MiB/s | 5073.5 | 0.07 ms | 390.6 MiB | 1.00x |
| lmdb-raw | 599 MiB/s | 1111 MiB/s | 569.0 | 0.72 ms | 786.3 MiB | 0.50x |
| arrow-ipc-raw | 1230 MiB/s | 227 MiB/s | 116.3 | 2.85 ms | 390.8 MiB | 1.00x |
| arrayrecord-raw | 853 MiB/s | 148 MiB/s | 76.0 | 5.58 ms | 401.4 MiB | 0.97x |
| zrecord-zstd | 1201 MiB/s | 2224 MiB/s | 1138.7 | 0.29 ms | 193.2 MiB | 2.02x |
| zrecord-zstdict | 1426 MiB/s | 2530 MiB/s | 1295.5 | 0.25 ms | 168.5 MiB | 2.32x |
| arrayrecord-zstd | 122 MiB/s | 131 MiB/s | 67.0 | 4.64 ms | 201.3 MiB | 1.94x |

The contiguous NumPy baseline is strongest for gather when the whole corpus is
one fixed typed matrix. Zrecord-raw reaches 3.87 Mrecords/s while retaining
independent record semantics and writes 2.3x faster than npy-mmap-raw;
dictionary zstd writes 19% faster than plain zstd, uses 13% less disk and gathers
14% faster in this pass. LMDB's B-tree/page overhead is visible in both throughput
and disk.

#### Variable Token Sequences

The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28,
median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record,
random plan and every backend must return ordered `list[np.ndarray]`
with exact `int32` values and original one-dimensional shapes.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 1902 MiB/s | 919 MiB/s | 1735.5 | 0.19 ms | 110.5 MiB | 0.96x |
| lmdb-raw | 322 MiB/s | 117 MiB/s | 221.2 | 1.46 ms | 151.8 MiB | 0.70x |
| arrow-ipc-raw | 317 MiB/s | 44 MiB/s | 83.9 | 3.82 ms | 109.1 MiB | 0.97x |
| arrayrecord-raw | 304 MiB/s | 25 MiB/s | 48.2 | 8.16 ms | 118.0 MiB | 0.90x |
| zrecord-zstd | 546 MiB/s | 621 MiB/s | 1173.5 | 0.29 ms | 67.4 MiB | 1.57x |
| zrecord-zstdict | 961 MiB/s | 799 MiB/s | 1509.8 | 0.21 ms | 53.0 MiB | 2.00x |
| arrayrecord-zstd | 61 MiB/s | 26 MiB/s | 49.2 | 7.49 ms | 76.0 MiB | 1.39x |

Here the record contract, not bulk byte bandwidth, is the useful scale.
Zrecord's three codecs return 1.17–1.74 Mrecords/s with 0.19–0.29 ms p95.
Dictionary zstd writes 76% faster than plain zstd, uses 21% less disk and gathers
29% faster in this pass. Raw reaches 1902 MiB/s write.

### Sampler

Index generation on its own, IID (with replacement), 1M index space, against
NumPy's modern API. The µs-scale figures fluctuate with box load; this pass shows
a 2.25–10.64x margin across batch sizes.

| sampler | batch | per batch | vs default_rng |
|---|---:|---:|---:|
| numpy default_rng | 256 | 5.8 µs | 1.00x |
| **zsampler** | 256 | 0.5 µs | **10.64x** |
| numpy default_rng | 1024 | 6.3 µs | 1.00x |
| **zsampler** | 1024 | 1.1 µs | **5.75x** |
| numpy default_rng | 8192 | 17.9 µs | 1.00x |
| **zsampler** | 8192 | 7.9 µs | **2.25x** |

### End-to-End DataLoader

The full pipeline comparison (sample, fetch, collate, handoff) uses the Dense
vision source above: 2,500 `(3,224,224)` uint8 records, 4 workers, batch 256
(36.8 MiB), and one 200-batch pass after warmup. `peak PSS` apportions shared and
copy-on-write mappings across the process tree; `peak RSS` sums each process's
resident set and therefore double-counts shared pages on Linux. These are
process-memory diagnostics, not total pipeline physical memory.

This is an end-to-end systems comparison with each loader's normal storage:
Torch uses read-only NumPy mmap, Loaderx uses Zrecord, and Grain uses
ArrayRecord. Fork represents normal Linux Torch operation; spawn is the no-fork
control. Grain is selected explicitly with `--only grain`.

| loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
|---|---|---|---:|---:|---:|---:|---:|
| **loaderx** | threads | zrecord-zstd | 112.2 | 14.49 ms | 633 MiB | 665 MiB | 669 MiB |
| loaderx-raw | threads | zrecord-raw | 171.4 | 11.79 ms | 668 MiB | 668 MiB | 671 MiB |
| torch | fork | npy-mmap-raw | 76.3 | 47.67 ms | 1400 MiB | 1542 MiB | 3860 MiB |
| torch-spawn | spawn | npy-mmap-raw | 99.7 | 34.89 ms | 2627 MiB | 2705 MiB | 4676 MiB |
| grain | processes | arrayrecord-zstd | 44.5 | 93.95 ms | 1698 MiB | 1764 MiB | 1886 MiB |

At 36.8 MiB per batch, gather dominates sampler cost and overlaps with
transform work. With source geometry and entropy fixed, raw Loaderx is 1.53x
compressed Loaderx. Compressed Loaderx is 1.47x Torch fork, 1.13x Torch spawn,
and 2.52x Grain; raw Loaderx is 2.25x, 1.72x, and 3.85x faster, respectively.
RSS must not be read as a physical-memory ratio because Linux counts shared
mappings repeatedly and page-cache accounting differs from mmap.

The Torch-only worker sweep runs each count once. Worker 0 is the in-process
baseline: 67.7 and 68.2 batches/s with 1411/1413 MiB peak PSS in two equivalent
runs. The process comparison starts at one worker:

| workers | fork batches/s | fork peak PSS | fork peak RSS | spawn batches/s | spawn peak PSS | spawn peak RSS |
|---:|---:|---:|---:|---:|---:|---:|
| 1 | 41.0 | 1429 MiB | 1730 MiB | 39.3 | 1708 MiB | 1946 MiB |
| 2 | 64.6 | 1510 MiB | 2476 MiB | 61.3 | 2064 MiB | 2887 MiB |
| 4 | 98.5 | 1611 MiB | 3899 MiB | 105.0 | 2714 MiB | 4674 MiB |
| 8 | 114.4 | 1799 MiB | 6615 MiB | 121.3 | 4038 MiB | 8187 MiB |

Spawn peak PSS grows from 1708 to 4038 MiB as workers rise from one to eight,
while fork grows from 1429 to 1799 MiB because it retains COW sharing. At eight
workers spawn uses 2.24x fork's peak PSS and throughput has already flattened.
This demonstrates no-fork memory pressure, not OOM: the 31 GiB machine completed
the run. Aggregate RSS remains diagnostic rather than physical memory.

**CPU-heavy transform solutions.** The same Python per-sample transform exposes
the GIL bottleneck on standard CPython. Numba is an explicit solution, not the
default: `--transform numba-nogil` compiles the equivalent batch transform with
`nogil=True`. The other explicit solution runs the unchanged Python transform on
free-threaded CPython. Numba compilation is warmed before timing; both
interpreters use the same prepared real corpus.

| loader | GIL Python | GIL + Numba nogil | free-threaded Python | Numba gain | free-threaded gain |
|---|---:|---:|---:|---:|---:|
| **loaderx** | 48.0 batches/s | 95.9 batches/s | 78.0 batches/s | **2.00x** | **1.63x** |
| loaderx-raw | 52.9 batches/s | 117.5 batches/s | 89.2 batches/s | **2.22x** | **1.69x** |

Peak PSS for compressed/raw was 860/871 MiB with GIL Python, 933/957 MiB with
Numba, and 879/887 MiB with free-threaded Python. These are two deployment
solutions to the transform bottleneck, not claims that Store itself was
optimized for either runtime.

**Conclusion.** Loaderx batches sampling, gather, decompression, and Ragged
reconstruction while preserving exact record semantics. IID sampling remains
uniform with replacement. Dense serves fixed-shape arrays directly; Ragged
restores each record's original shape. This favors random training gathers,
where formats organized around contiguous scans or larger storage groups can
pay read amplification.

Compression results depend on the data. In the Dense photograph workload,
plain zstd reaches only 1.15x and dictionary zstd 1.12x; dictionary mode is most
useful for repeated small records, not as a universal default.

The loader table combines execution model and storage, rather than comparing
schedulers over one shared backend. Compressed Loaderx is 1.47x Torch fork,
1.13x Torch spawn, and 2.52x Grain; raw Loaderx is 2.25x, 1.72x, and 3.85x
faster, respectively. CPU-heavy Python transforms remain limited by the GIL;
the measured alternatives are Numba `nogil=True` and free-threaded Python.

These numbers measure warm-cache access, not cold disk or durability. Disk means
allocated blocks; write timing excludes stable-media sync and dictionary
training. Arrow IPC and Parquet use 256-record groups, so their random gathers
include group-level read amplification. This is one complete benchmark pass;
Store reads accumulate at least two timed seconds, while sampler and loader
figures can fluctuate with machine load. Treat absolute values as ballpark and
cross-backend margins as the main signal.

## Real-data verification

Production dataset-specific preprocessing and verification remain in the
**[DataPipe](https://codeberg.org/eoelab/DataPipe)** repository. The built-in
converter covers standardized Hugging Face datasets; DataPipe handles sources
without a common remote protocol and implements derived modalities as loader
transforms without NumPy dump intermediates. Oxford-IIIT Pet is only the shared
benchmark fixture.

## Current Limitations
* Single-host only; multi-host training is not supported.
* A single sample must be at most 2 GiB. Store size is practically bounded by
  disk capacity and platform file limits.
* Stores are not portable between machines with different byte orders. All
  published platforms are little-endian.

## 设计文档

- [性能优化与工程决策记录](docs/性能优化.md)
- [分布式架构草案](docs/分布式架构.md)

## Build
```
python3 setup.py build_ext --inplace
zig build test
python3 scripts/test_loaderx.py
```

Build the release wheel matrix with:

```
python3 scripts/build_release.py
```

Source distributions are intentionally not published. The wheel matrix covers
the supported platforms. Source builds require Zig.
