Metadata-Version: 2.4
Name: loaderx
Version: 2.7.6
Summary: Rebuildable high-performance ordered record containers
Author-email: Ben0i0d <ben0i0d@foxmail.com>
License-Expression: MIT
Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
Project-URL: Source, https://codeberg.org/eoelab/loaderx
Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
Keywords: flax,python,dataloader
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.0
Requires-Dist: msgpack
Provides-Extra: converter
Requires-Dist: datasets[vision]>=2.19; extra == "converter"
Dynamic: license-file

# Loaderx
Zrecord is a rebuildable, typed, ordered record sequence built from authoritative
source data and scripts. A creator consumes records in append order, `close`
publishes one immutable container, and readers support integer indexing, natural
iteration, slices, and indexed gather without changing record identity. Dense
integer indexing unwraps the leading batch axis; integer indexing of Ragged
returns a one-record `RaggedBatch`. To change content or order, rebuild it at a
new path.

Zrecord is the typed on-disk container; Loaderx is the sampler and data loader
that consumes Zrecord streams. They currently ship together while both layers
mature, but their public responsibilities remain separate.

```
pip install loaderx
```

Wheels are published for CPython 3.10+ on glibc Linux (x86-64 and ARM64), Apple
Silicon macOS, and Windows AMD64. Free-threaded CPython 3.14 is also supported.

## Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. NumPy-native typed records with explicit schemas.
4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. **Dense and Ragged are explicit physical layouts.** Applications may
   regularize records into one fixed shape or preserve their variable geometry.
   Loaderx stores and delivers that choice without converting between layouts.
6. **Logical IDs are stable sequence positions.** Append order defines
   `0..N-1`; published containers are immutable.

## Usage

### Quick Start
```python
import numpy as np

from loaderx.zrecord import Dense
from loaderx.dataloader import Loader
from loaderx.sampler import Sampler

data = np.load('data.npy', mmap_mode='r')
label = np.load('label.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:], codec='zstd') as ds:
    ds.append(data)
with Dense.create('train_label', label.dtype, label.shape[1:], codec='zstd') as ds:
    ds.append(label)

data_store = Dense.open('train_data')
label_store = Dense.open('train_label')
sampler = Sampler(len(data_store), 256, Sampler.Mode.CYCLIC, seed=42)
loader = Loader({'data': data_store, 'label': label_store}, sampler,
                transform=lambda batch: batch)
for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

loader.close()
data_store.close()
label_store.close()
```

A batch is a dict `{name: values}`: a Dense value is a `DLTensor` of shape
`(batch_size, *item_shape)`, and a Ragged value is a `RaggedBatch` containing
`values`, `cu_seqlens`, and `shapes`. Every stream is gathered at the same
indices, so record `i` lines up across them. The `transform` callback is the
collate step: reshape, cast, stack, or hand values to a framework. Its return
value is the batch ready for the model.

### Records
The Dense or Ragged contract follows the geometry an application chooses to
persist. Fixed-shape records have one schema-known stride and stack directly;
records whose shapes vary require explicit payload boundaries and per-record
shapes. Resizing, padding, truncating, or otherwise converting between these
layouts is application policy, not loader behavior.

`Dense` stores one fixed-shape array per record. An integer-index read or one
step of iteration returns the record itself; a multi-record read returns one
stacked `DLTensor`:

```python
import numpy as np
from loaderx.zrecord import Dense

data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
with Dense.create('data', data.dtype, data.shape[1:], codec='zstd') as ds:
    ds.append(data)
with Dense.open('data') as ds:
    record = ds[0]                    # DLTensor (2, 4), no leading batch axis
    batch = ds[0, 5, 2]               # DLTensor (3, 2, 4), requested order retained
    for record in ds:                 # each record is a DLTensor (2, 4)
        consume(record)
```

A store is an ordered sequence of records, not an ndarray, so `ds[0, 5, 2]`
selects sequence positions 0, 5 and 2, never `ds[0][5][2]`. An integer index
selects one unwrapped Dense record, while any collection of indices selects a
batch. Reads return `loaderx.zrecord.DLTensor`, a DLPack-only interchange object. It has no
`to_numpy()` or `__array__`; consume a host value with `np.from_dlpack(record)`
or any value with a compatible framework such as `torch.from_dlpack(record)`.
Indices must be in `0..len(ds)-1`.

`Ragged` stores variable-shape arrays with one shared `dtype` and `ndim >= 1`.
It does not support zero-dimensional records. Scalar-valued records have a fixed
0-D item shape and therefore belong in `Dense` with `item_shape=()`.
Reads return a `loaderx.zrecord.RaggedBatch`: `values` (packed data),
`cu_seqlens` (record start offsets), and `shapes`, using one contract on host
and device. An empty batch has `cu_seqlens == [0]`; integer indexing returns a
single-record `RaggedBatch` (`shapes.shape[0] == 1`). Host batches provide `to_tensors()` for
explicit per-record reconstruction; device batches must be consumed in packed
form or converted by the device framework. Padding remains
an application policy:

```python
from loaderx.zrecord import Ragged, RaggedBatch

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
with Ragged.create('tokens', np.int32, ndim=1, codec='zstd') as rs:
    rs.append(RaggedBatch.from_records(seqs))
with Ragged.open('tokens') as rs:
    batch = rs[0, 2, 4]               # RaggedBatch: values + cu_seqlens + shapes
    values = batch.values            # packed data, ready for a ragged-aware consumer
    tensors = batch.to_tensors()     # explicit per-record DLTensor views
    for tensor in tensors:           # each is one shape-restored DLTensor
        consume(tensor)
```

Logical ID is the stable sequence position. Each `append` preserves input order,
and successive calls extend the sequence. After `close`, the sequence is
immutable: there is no delete, update, compaction, or reopen-for-append.

### Creating containers

`Dense.create` and `Ragged.create` return append-only builders. The required
`codec` is `"raw"`, `"zstd"`, or `"zstd_dict"`. Schema, shape, and dtype are
explicit and are never inferred from the input. Dense consumes contiguous NumPy
arrays directly, external CPU DLPack producers through NumPy, and Loaderx
`DLTensor` through its retained CPU mapping. Ragged accepts
the packed `RaggedBatch` representation also returned by reads;
`RaggedBatch.from_records(records)` explicitly adapts a finite iterable of NumPy
arrays. Arrow or kernel producers can construct packed `values`, `cu_seqlens`,
and `shapes` directly. Pass the complete source to one
`append`; the native Zig writer bounds its scheduling and scratch memory
internally, so callers do not need to slice the source into memory-control batches.

A creator consumes a finite build stream and publishes an immutable sequence;
readers do not tail an active writer. Append is synchronous and returns after
the complete input has been persisted. If a device framework populated an
allocation-backed `DLTensor`, synchronize that producer before calling
`append`; Loaderx does not own the framework stream or insert its events.

```python
from loaderx.zrecord import Dense, Ragged, RaggedBatch

with Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28),
                  codec='zstd', data_shards=4) as ds:
    ds.append(images)

with Ragged.create('tokens', dtype=np.int32, ndim=1, codec='zstd') as tok:
    tok.append(RaggedBatch.from_records(sequences))

with Dense.open('mnist/x') as ds:
    first_four = ds[:4]               # opened containers are read-only
```

`data_shards` controls write parallelism, accepts `1..255`, and defaults to four.
Readers discover it automatically.

Append errors are reported by the current call. A writer cannot be read and a
reader cannot be appended to. Writer `close()` publishes the container; reader
`close()` releases it. Dense, Ragged, and both loaders support `with`.

Schemas accept native-endian bool, integer, floating-point, and complex dtypes
that DLPack can represent without loss. Structured, subarray, object,
metadata-bearing, non-native-endian, and zero-itemsize dtypes are not supported.
Append does not use NumPy's general array coercion. Dense accepts a contiguous
NumPy array, an external CPU DLPack producer, or a Loaderx `DLTensor` with its
retained CPU mapping. Other device producers require an explicit transfer path.
Build a packed `RaggedBatch` directly or convert record objects explicitly with
`RaggedBatch.from_records`. Raw bytes and encoded files can be represented as
`np.uint8` records.

### Codec notes

`"zstd"` compresses each record independently with plain zstd (level 3). Use it
for general-purpose compression; it is fast and must be selected explicitly.

`"zstd_dict"` uses a shared dictionary trained from representative settled data.
It is useful for large corpora of small, similar records such as token sequences
and image tiles. Plain `"zstd"` or `"raw"` is usually better for large records
and small corpora.

Train the dictionary manually with `train_dict` on settled data, then pass its
bytes to `dict_bytes`. The source may be a NumPy array, an iterable of records,
or an existing typed container. `zstd_dict` never trains automatically:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense, Ragged, RaggedBatch

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array)

# then any new store can install it and append explicitly
with Ragged.create('tokens', np.int32, ndim=1,
                   codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(RaggedBatch.from_records(token_generator))
with Dense.create('data', data.dtype, data.shape[1:],
                       codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(data)
```

Changing an existing store's codec requires writing a new store:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense

with Dense.open("src") as s, \
     Dense.create("dst", dtype=s.dtype, item_shape=s.item_shape,
                       codec="zstd_dict", dict_bytes=train_dict(s)) as d:
    d.append(s[:])                             # DLTensor -> DLTensor
```

For both geometries, a batch read is already the exact append input. Dense uses
`DLTensor`; Ragged uses the packed `RaggedBatch`. Integer indexing unwraps one
record and therefore omits Dense's leading batch axis. Create the destination
with the source schema. The destination path must be new.

**Important:** Train the dictionary from settled authoritative input before
building the container. Training it before preprocessing is complete wastes
compression and does not describe the final records.

### Offline Hugging Face conversion

The optional converter uses Hugging Face Datasets for remote discovery,
download, caching, revision handling and Arrow/Parquet materialization. It
converts Arrow-backed `Dataset` and `DatasetDict` objects directly into typed
Zrecord streams, so training needs neither `datasets` nor Arrow:

```bash
pip install 'loaderx[converter]'
```

```python
from datasets import Dataset, Features, Sequence, Value
from loaderx.converter import convert

dataset = Dataset.from_dict(
    {"tokens": [[1, 2], [3], [4, 5, 6]]},
    features=Features({"tokens": Sequence(Value("int32"))}),
)
convert(dataset, "tokens", codec="zstd")
```

Mainland China or a private Hub can select an endpoint directly without setting
process environment variables:

```python
dataset = huggingface(
    "ylecun/mnist",
    endpoint="https://hf-mirror.com",
)
```

`huggingface` resolves the requested revision to an immutable snapshot and
returns its `DatasetDict`. Pass `config=...` when a repository has no default
configuration, and `token="hf_..."` for private or gated repositories.

`convert` consumes Arrow batches, not decoded Python records. Numeric scalar and
fixed-shape columns become Dense stores; numeric List and Binary columns become
one-dimensional Ragged stores. A multidimensional Ragged column uses an
explicit Arrow struct with `values: List<primitive>` and
`shape: FixedSizeList<uint64, ndim>`. Null, empty, nested dynamic, decoded Image
and Audio columns fail explicitly. Materialize preprocessing first with
`Dataset.map` and then call `convert`; `IterableDataset` is intentionally not a
converter input.

The result groups aligned streams under one published root:

```text
dataset/
  train/
    tokens/
    label/
  test/
    tokens/
    label/
```

The output is published only after all columns have been written and their
record counts agree. Each column remains an ordinary Zrecord store.

### Loaders and multi-stream stores

A zrecord store is one stream; a training sample is usually several named
streams (skeleton + label + id, tokens + label, ...). Composition is a plain
Python `dict` passed to `Loader`. There is no persistent
wrapper, manifest, directory convention or bundle mutation API. A loader verifies
that all streams have the same length, then gathers every stream with the same
indices.

```python
from loaderx.zrecord import Dense, Ragged, RaggedBatch
from loaderx.dataloader import Loader
from loaderx.sampler import Sampler

root = "xsub/train"
with Dense.create(root + "/joint", joint.dtype, joint.shape[1:], codec="zstd") as s:
    s.append(joint)
with Dense.create(root + "/label", label.dtype, label.shape[1:], codec="zstd") as s:
    s.append(label)
with Ragged.create(root + "/token", np.int32, ndim=1, codec="zstd") as s:
    s.append(RaggedBatch.from_records(seqs))

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
streams["joint"][0, 5, 2]            # each stream keeps its own index API
sampler = Sampler(len(streams["joint"]), 256, Sampler.Mode.CYCLIC, seed=42)
loader = Loader(streams, sampler)
batch = next(loader)                  # {name: values}, index-aligned

loader.close()
for stream in streams.values():
    stream.close()
```

Fixed-shape and variable-shape records are both just stores — `Dense` has one
schema-known item shape, while `Ragged` preserves each record's geometry. The
loader does not interpret either: it fetches by index and packs a
dict, so a dense stream's batch value is a `DLTensor` of shape
`(B, *item_shape)` and a ragged stream's is a
`loaderx.zrecord.RaggedBatch`: `values` (packed data),
`cu_seqlens` (record start offsets), and `shapes`, the same contract on host
and device, only the location differs. No padding is imposed. Densify
to a fixed shape however the model needs, or reshape/stack in the `transform`
collate.

Collation is the `transform`: a batch dict in, an arbitrary result out. With
positive prefetch, each worker serializes sampler draw and gather under one lock,
then runs transform concurrently and puts its result directly into the bounded
output queue. Completed batches may arrive out of sampling order, so `transform`
must be thread-safe. Set `prefetch=0` to run gather and transform synchronously
in the caller's thread when strict sampling order or minimal live allocation
pressure matters. Allocator-backed device streams support both policies;
prefetch simply keeps more allocations live concurrently.

Choose the execution policy from the workload, not from the allocation device:

| Workload | Loader policy | Why |
|---|---|---|
| DLPack-only handoff or another very light transform | `prefetch=0` | Avoid thread and queue overhead |
| Strict sampling order or minimum live memory | `prefetch=0` | Gather and transform complete in the caller thread |
| Blocking I/O, tokenization, decoding, or native CPU transform | default `prefetch=4`, `transform_workers=4` | Overlap gather and concurrent transforms |
| Device allocation with a substantial parallel transform | positive `prefetch` and measured worker count | Device batches are independent and may be queued safely |

```python
# Light GPU handoff: synchronous delivery is usually the lower-overhead path.
loader = Loader(streams, sampler, prefetch=0, transform=to_torch)

# Expensive thread-safe transform: use the asynchronous default, or tune both
# bounds from an end-to-end profile.
loader = Loader(streams, sampler, prefetch=4, transform_workers=4,
                transform=tokenize_and_collate)
```

`prefetch` is both a scheduling and memory-backpressure parameter. With a
positive depth, up to roughly `prefetch + transform_workers` batches may remain
live in the output queue and workers. This applies equally to CPU and device
allocators. A Python queue copies no tensor bytes: it holds object references,
and each `DLTensor` or `RaggedBatch` retains its exclusive `Allocation`. An
AMDGPU batch therefore remains in the same device-consumable allocation while
queued and is imported by `torch.from_dlpack` or JAX without an implicit D2H
transfer. Only an explicit transform such as `.cpu()` requests a readback.

```python
def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

sampler = Sampler(len(dense_tokens), 32, Sampler.Mode.CYCLIC, seed=42)
loader = Loader({'tokens': dense_tokens, 'label': labelset}, sampler,
                transform=collate)
batch = next(loader)
loader.close()
```

The transform runs once per gathered batch; its return value is passed to the
consumer unchanged, and exceptions propagate to the consumer.

Numba can optionally accelerate a CPU-heavy Dense transform. Compile it before
timing, then call it from the ordinary transform:

```python
import numba
import numpy as np

@numba.njit(nogil=True, parallel=False)
def normalize_u8(x):
    out = np.empty(x.shape, dtype=np.float32)
    for i in range(x.size):
        out.flat[i] = x.flat[i] / 255.0
    return out

normalize_u8(np.zeros((1, 3, 224, 224), dtype=np.uint8))  # compile warmup

def transform(batch):
    batch["image"] = normalize_u8(np.from_dlpack(batch["image"]))
    return batch
```

Numba is optional. Compile it before measuring loader throughput.

### CPU → GPU transfer

Data reaches the GPU through one of two memory placements, chosen by where the
computation runs (数据发生地): if the model consumes CPU batches and the framework copies
them over, that is the host allocation below; the AMDGPU allocator instead
places the Store output in a device-consumable allocation. Loaderx's job in
both is to hand the consumer a correct representation — it moves and
reconstructs records, it does not run GPU operators.

The host path produces CPU batches. Device transfer belongs in `transform`,
where the training framework is already available:

```python
import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_dlpack(v).to(device, non_blocking=True)
            for k, v in batch.items()}

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
}
sampler = Sampler(len(streams["joint"]), 256, Sampler.Mode.CYCLIC, seed=42)
loader = Loader(streams, sampler, transform=to_device)
for batch in loader:
    model(batch)                        # already on device
```

Call `loader.close()` when the training loop exits, then close its caller-owned
streams. Do not close either from `transform`.

A `non_blocking=True` copy is asynchronous only from pinned memory. Loaderx does
not pin memory; use the framework's API in the transform when needed:

```python
def to_device(batch):
    return {k: torch.from_dlpack(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}
```

For JAX, use `jax.device_put`:

```python
import jax
import jax.dlpack

def to_device(batch):
    return {k: jax.device_put(jax.dlpack.from_dlpack(v)) for k, v in batch.items()}
```

For complete integrations, see **[data2latent](examples/data2latent/)**, which
runs the same prepared Dense and Ragged images through Torch or JAX, and
**[Word2Vec](examples/word2vec/)**, which trains equivalent padded and packed
CBOW models on WikiText-103.

Loaderx targets Torch and JAX through the standard DLPack interchange protocol.
It does not vendor either framework or ROCm libraries, and TensorFlow is not a
maintained target. Both examples expose host and AMDGPU allocations for Torch
and JAX on ROCm hardware.

#### Allocator-selected output memory

The host path above asks the framework to create a separate device tensor. An
AMDGPU allocation instead lets Store produce a device-consumable batch directly.
Zrecord computes each output's
exact layout and asks an allocator for an aligned region; live batches retain
exclusive allocations and released regions return to their allocator:

```python
from loaderx.allocator import amdgpu
from loaderx.zrecord import Dense
import torch

allocator = amdgpu.Allocator(device=0)
try:
    with Dense.open(path, allocator=allocator) as reader:
        batch = reader[idxs]                         # lazily sized allocation
        gpu_tensor = torch.from_dlpack(batch)
        consume(gpu_tensor)
finally:
    allocator.close()
```

JAX consumes the same device object directly with
`jax.dlpack.from_dlpack(batch)`. Loaderx accepts DLPack's `stream` argument for
framework compatibility, but the synchronous gather does not create a producer
stream or insert framework-specific events. Every live batch has an exclusive
allocation, so a later gather cannot overwrite it.

`loaderx.allocator` defines the common `Allocator` / `Allocation` contract;
ordinary reads use `loaderx.allocator.host.Allocator` internally.
`loaderx.allocator.amdgpu.Allocator` implements the same contract with pooled
AMDGPU allocations. Dense computes exact bytes directly;
Ragged reads physical lengths, computes aligned shapes/values/cu_seqlens layout,
then acquires the final region. Users never provide a batch capacity.

Concrete placement implementations live under `loaderx.allocator`; importing
Loaderx does not discover optional GPU runtimes. An implementation supplies
writable regions to the common allocator contract and does not add a Store or
Loader execution path.

Dense reads return `DLTensor`; writes consume contiguous NumPy arrays, CPU
DLPack producers, or returned allocation-backed tensors. Read placement is selected once with
`Dense.open(path, allocator=allocator)`. Each allocation's immutable `(DLPack
device type, logical device id)` is inherited by its DLTensors.

Loaderx imports the process's already-loaded HIP runtime when present. Otherwise
it checks `LOADERX_HIP_LIBRARY`, `ROCM_PATH`/`ROCM_HOME`, an installed
package-owned runtime from `rocm-sdk-core` or legacy Torch, `/opt/rocm`, and
finally the system loader. Multiple distinct package-owned runtimes are rejected
rather than guessed; set `LOADERX_HIP_LIBRARY` to resolve the choice. No Torch or
JAX module is imported during discovery. HIP logical devices and DRM render
nodes are matched by PCI BDF, so visibility remapping does not depend on
render-node order.

Raw stores read directly into the selected allocation. Compressed stores use
host decoding scratch but place the final Dense/Ragged layout in that same
allocation; codec and memory location are independent. `Dense.append` consumes
the returned host-writable `DLTensor` directly, so host and AMDGPU sources share
one write path for every codec.
Use `Loader(..., prefetch=0)` for synchronous delivery and minimal live device
memory, or a positive prefetch depth when overlap is worth the allocator pool
growth. See the benchmark section for the end-to-end comparison.

Loaderx **supports computation, it does not implement it**: allocator-selected
delivery preserves the same data layout on host or device, never the operators
themselves. Ragged
delivers tightly packed `values`, element boundaries in `cu_seqlens`, and exact
per-record `shapes`. Ragged-aware or custom kernels can consume this native
layout without first materializing a Python list or padding it into Dense;
kernel-specific views and metadata remain the consumer's responsibility.
The [examples](examples/) apply this distinction to two representative
workloads. `data2latent` checks equivalent image-to-latent computation, while
`word2vec` checks equivalent embedding training from padded and packed contexts;
both expose Torch and JAX through the same Loader over host or AMDGPU allocations.

### Sampler
Sampler is a borrowed Python buffer over a stateless Cython batch function.
Loader receives a sampler object rather than duplicating its batch size,
mode, or seed. A run resumes in O(1) with `seek(step)`, without replaying draws
or tracking an epoch.

```python
from loaderx.sampler import Sampler

sampler = Sampler(1_000_000, 256, Sampler.Mode.IID, seed=42)
indices = sampler.next()       # borrowed until this sampler's next draw
saved = indices.copy()         # retain across draws only when needed
```

`next()` and iteration return a view of one reusable `uint64` batch buffer.
The contents stay unchanged until the next explicit draw from that Sampler;
copy only plans that must outlive it. The loaders consume each view before
drawing again. They borrow the sampler and never close it. `Sampler` exposes
only `next()`, `seek()`, and iteration; it has no public `indices` property,
`close()`, or context-manager protocol. Any
user object can be injected instead; its `next()` should return a borrowed
one-dimensional contiguous NumPy index array. This is a trusted hot-path
contract rather than a normalized protocol, so custom policies can be ordinary
NumPy code without Loaderx adapters or lifecycle methods.

1. **Sequential** traverses records in order and wraps at the end.
2. **IID** samples uniformly with replacement.
3. **Cyclic** traverses a fresh permutation without replacement. It does not
   allocate a dataset-sized permutation. Full batches are returned; a different
   remainder is omitted on each cycle when the length is not divisible by the
   batch size.

## Benchmarks

Dense and Ragged are measured separately because they expose different
contracts. `scripts/bench_dense.py` measures fixed-shape random gather;
`scripts/bench_ragged.py` measures variable-shape records as compute-ready
`values + cu_seqlens + shapes`; and `scripts/bench.py` covers machine, sampler,
and the end-to-end loader comparison. Every path runs through the public Python
binding, including output and metadata allocation. Every Ragged write backend
starts from the same Arrow List payload and fixed-size shape column. Zrecord
constructs its `RaggedBatch` from those Arrow buffers inside the timed write;
there are no Python-record benchmark variants. List-returning competitors are
packed into the common representation inside the timed gather call.

### Methodology

The Dense store tables below are historical results recorded for 2.6.1. The
Ragged tables were refreshed after D-073 with the common Arrow write source and
the current `RaggedBatch` gather contract. Each is one complete pass on a warm
page cache with the default `data_shards=4`, not a three-run median. The script
verifies exact dtype, shape, order, and values before timing.

The `logical write` and `logical gather` columns report uncompressed NumPy payload
bytes per elapsed second, not physical storage bandwidth. Write timing
includes public-API adaptation and logical finalization, but excludes source
preparation, dictionary training, writer setup, cleanup, and stable-media sync.
Zrecord writers receive the complete prepared source in one append and bound
execution internally.
Gather timing includes allocation, reads, decompression, and output construction;
Ragged packing and metadata production are included. It excludes open, close,
sampler time, and destruction after return. Fresh uniform IID gathers accumulate
at least two timed seconds. `krecords/s` is record throughput, `p95` is one
random batch's 95th-percentile latency, and `disk` is allocated blocks. Compare
results within a workload table, not across different record geometries. Every
listed backend and codec is required for the published matrix.

The vision source is the Oxford-IIIT Pet train split prepared by
`scripts/prepare_vision.py`. Dense uses RGB photographs resized on the short side
to 256 and center-cropped to `(3, 224, 224)`; Ragged uses the same ordered source
at native RGB resolution. Both are mmap-loaded uint8 CHW records.

**Machine** — one local workstation (AMD Ryzen AI 9 HX PRO 370, 12 cores / 24
threads):

| machine   | value |
|-----------|-------|
| CPU       | AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M, 1 socket, 12 cores / 24 threads |
| frequency | 605–5158 MHz |
| caches    | L1d 576 KiB, L1i 384 KiB, L2 12 MiB, L3 24 MiB |
| NUMA      | 1 node |
| memory    | 31 GiB (not limited by cgroup) |
| shared memory | 16 GiB `/dev/shm` |
| OS        | Debian GNU/Linux forky/sid, kernel 7.1.8+deb13-amd64, x86_64 |
| python    | CPython 3.14.7 (standard GIL build), numpy 2.5.2 |

The process sees all 24 threads, is not memory-limited by cgroup, and uses the
ordinary page cache. The 16 GiB shared-memory mount accommodates the Torch run.

### Large Vision Records

#### Fixed-Shape Dense

Zrecord against array-store alternatives: random batch gather over the first
2,500 Oxford-IIIT Pet train images, batch 256. Every prepared image is
`uint8[3,224,224]`; `metadata.json` records the source revision, transform,
decoder versions and logical SHA-256.

Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 2924 MiB/s | 15889 MiB/s | 110.7 | 2.61 ms | 358.9 MiB | 1.00x |
| npy-mmap-raw | 2167 MiB/s | 5411 MiB/s | 37.7 | 7.42 ms | 358.9 MiB | 1.00x |
| hdf5-raw | 2543 MiB/s | 2180 MiB/s | 15.2 | 20.64 ms | 359.0 MiB | 1.00x |
| lmdb-raw | 1537 MiB/s | 4459 MiB/s | 31.1 | 11.34 ms | 361.4 MiB | 0.99x |
| arrow-ipc-raw | 1734 MiB/s | 3726 MiB/s | 26.0 | 13.35 ms | 358.9 MiB | 1.00x |
| parquet-raw | 1251 MiB/s | 497 MiB/s | 3.5 | 88.21 ms | 358.9 MiB | 1.00x |
| arrayrecord-raw | 1853 MiB/s | 2303 MiB/s | 16.0 | 19.17 ms | 359.2 MiB | 1.00x |
| zrecord-zstd | 1350 MiB/s | 5108 MiB/s | 35.6 | 9.25 ms | 311.1 MiB | 1.15x |
| zrecord-zstdict | 1203 MiB/s | 4654 MiB/s | 32.4 | 9.13 ms | 320.1 MiB | 1.12x |
| hdf5-gzip | 45 MiB/s | 226 MiB/s | 1.6 | 193.76 ms | 301.8 MiB | 1.19x |
| arrow-ipc-zstd | 240 MiB/s | 105 MiB/s | 0.7 | 367.56 ms | 305.9 MiB | 1.17x |
| parquet-zstd | 238 MiB/s | 90 MiB/s | 0.6 | 431.77 ms | 305.9 MiB | 1.17x |
| arrayrecord-zstd | 228 MiB/s | 1778 MiB/s | 12.4 | 24.90 ms | 320.1 MiB | 1.12x |

At 147 KiB per record, Zrecord-raw reaches 15.5 GiB/s and is 2.9x npy-mmap-raw.
Decoded photographs have little remaining redundancy: plain zstd reduces them
only 1.15x, while dictionary mode falls to 1.12x. This is
why large real images should use plain zstd or raw. LMDB and Arrow
IPC are competitive raw record
stores, while codecs tied to whole IPC batches or Parquet row groups pay read
amplification on random gathers. Dense demonstrates that
typed record ownership and per-record compression do not turn fixed tensors
into an object-store slow path.

#### Native-Resolution Ragged

This workload contains the first 2,500 native-resolution Oxford-IIIT Pet CHW RGB
images. Height is 108..2606 (median 375) and width is 117..3264 (median 500),
totaling 1252.3 MiB of logical uint8 payload. Each of 50 random
batches contains 256 records. Every backend persists payload plus exact shape
and produces an ordered packed batch. Zrecord returns its native `RaggedBatch`;
list-returning competitors allocate and fill equivalent packed values, int32
offsets, and uint64 shapes inside the timed call. Throughput counts payload bytes
only, while metadata work remains timed.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 911 MiB/s | 14487 MiB/s | 28.9 | 13.75 ms | 1252.4 MiB | 1.00x |
| hdf5-raw | 1986 MiB/s | 1693 MiB/s | 3.3 | 96.08 ms | 1253.1 MiB | 1.00x |
| lmdb-raw | 1716 MiB/s | 3324 MiB/s | 6.6 | 50.07 ms | 1257.0 MiB | 1.00x |
| arrow-ipc-raw | 1489 MiB/s | 2470 MiB/s | 4.9 | 68.34 ms | 1252.4 MiB | 1.00x |
| parquet-raw | 789 MiB/s | 503 MiB/s | 1.0 | 299.43 ms | 1252.4 MiB | 1.00x |
| arrayrecord-raw | 1672 MiB/s | 2199 MiB/s | 4.3 | 92.28 ms | 1253.0 MiB | 1.00x |
| zrecord-zstd | 1111 MiB/s | 2441 MiB/s | 4.8 | 69.68 ms | 1032.2 MiB | 1.21x |
| zrecord-zstdict | 800 MiB/s | 2470 MiB/s | 4.9 | 72.78 ms | 1032.9 MiB | 1.21x |
| arrayrecord-zstd | 211 MiB/s | 1612 MiB/s | 3.1 | 132.20 ms | 1054.6 MiB | 1.19x |

Zrecord-raw is 4.4x LMDB and 5.9x Arrow IPC in logical gather because it already
produces the contiguous compute representation. Plain and dictionary zstd both
reach 1.21x storage reduction and about 2.4 GiB/s, confirming that dictionary
mode is not useful for these large photographs. The Ragged matrix is
intentionally asymmetric: HDF5 gzip, Arrow IPC zstd and Parquet zstd adapters
are not implemented because the real 256-record batch took 1.1–1.7 seconds;
TileDB variable queries took 8.6–9.6 seconds and the backend was removed
entirely. ArrayRecord zstd remains as the compressed record-store comparison.

### Small Token Records

Both token workloads come from the same real corpus: WikiText-103 raw train,
tokenized with GPT-2 and stored as `int32` IDs. Preparation is outside every
measurement. `scripts/prepare_tokens.py` preserves nonempty text boundaries,
combines fragments shorter than 16 tokens, and splits records at 512 tokens into
`tokens.npy` plus `offsets.npy`; both benchmark scripts mmap those files.

#### Fixed Token Blocks

The Dense workload ignores text boundaries and packs the stream into 200,000
fixed `int32[512]` records: 2 KiB per record and 390.6 MiB logical payload.
Each IID batch gathers 256 records; fresh draws continue until timed gathers
accumulate at least two seconds.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 1724 MiB/s | 5655 MiB/s | 2895.3 | 0.11 ms | 393.7 MiB | 0.99x |
| npy-mmap-raw | 2205 MiB/s | 11436 MiB/s | 5855.1 | 0.06 ms | 390.6 MiB | 1.00x |
| lmdb-raw | 696 MiB/s | 1158 MiB/s | 592.8 | 0.69 ms | 786.3 MiB | 0.50x |
| arrow-ipc-raw | 2183 MiB/s | 240 MiB/s | 122.7 | 2.61 ms | 390.8 MiB | 1.00x |
| arrayrecord-raw | 947 MiB/s | 174 MiB/s | 89.1 | 4.84 ms | 401.4 MiB | 0.97x |
| zrecord-zstd | 764 MiB/s | 1673 MiB/s | 856.5 | 0.37 ms | 193.2 MiB | 2.02x |
| zrecord-zstdict | 808 MiB/s | 1882 MiB/s | 963.8 | 0.32 ms | 168.5 MiB | 2.32x |
| arrayrecord-zstd | 122 MiB/s | 150 MiB/s | 76.6 | 4.26 ms | 201.3 MiB | 1.94x |

The contiguous NumPy baseline is strongest for gather when the whole corpus is
one fixed typed matrix. Zrecord-raw reaches 2.90 Mrecords/s while retaining
independent record semantics; dictionary zstd writes 6% faster than plain zstd,
uses 13% less disk and gathers 12% faster in this pass. LMDB's B-tree/page
overhead is visible in both throughput and disk.

#### Variable Token Sequences

The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28,
median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses 100 independent
correctness batches followed by fresh timed IID batches of 256 records and the
same `RaggedBatch` output contract as native-resolution vision.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-raw | 820 MiB/s | 691 MiB/s | 1304.7 | 0.32 ms | 110.5 MiB | 0.96x |
| lmdb-raw | 221 MiB/s | 96 MiB/s | 181.7 | 1.75 ms | 151.8 MiB | 0.70x |
| arrow-ipc-raw | 282 MiB/s | 40 MiB/s | 76.1 | 4.18 ms | 109.1 MiB | 0.97x |
| arrayrecord-raw | 195 MiB/s | 24 MiB/s | 45.9 | 8.25 ms | 118.0 MiB | 0.90x |
| zrecord-zstd | 338 MiB/s | 444 MiB/s | 839.6 | 0.46 ms | 67.4 MiB | 1.57x |
| zrecord-zstdict | 377 MiB/s | 521 MiB/s | 985.1 | 0.37 ms | 53.0 MiB | 2.00x |
| arrayrecord-zstd | 56 MiB/s | 25 MiB/s | 46.7 | 6.92 ms | 76.0 MiB | 1.39x |

Here the record contract, not bulk byte bandwidth, is the useful scale.
Zrecord-raw returns 1.30 Mrecords/s and is 7.2x LMDB and 17.1x Arrow IPC in this
logical-gather comparison. Starting from Arrow buffers also raises Zrecord raw
write throughput from the old Python-record table's 357 to 820 MiB/s. Dictionary
zstd writes 12% faster than plain zstd, uses 21% less disk, and gathers 17%
faster in this pass.

### Sampler

Index generation on its own, IID (with replacement), 1M index space, against
NumPy's modern API. The counter-based Cython function is stateless at every step.
The µs-scale figures fluctuate with box load; three 100K-draw runs show a
1.9–7.7x median margin across batch sizes.

| sampler | batch | per batch | vs default_rng |
|---|---:|---:|---:|
| numpy default_rng | 256 | 3.2 µs | 1.00x |
| **sampler** | 256 | 0.5 µs | **7.67x** |
| numpy default_rng | 1024 | 4.3 µs | 1.00x |
| **sampler** | 1024 | 1.3 µs | **3.39x** |
| numpy default_rng | 8192 | 17.5 µs | 1.00x |
| **sampler** | 8192 | 9.2 µs | **1.94x** |

### End-to-End Loader

The current loader benchmark measures host and AMDGPU output allocations
separately. Host rows use prefetched `Loader`; AMDGPU rows use
`Loader(..., prefetch=0)` to limit simultaneously live allocations. Benchmark
row names are `loaderx-host`, `loaderx-amdgpu`, and `torch`; Ragged row names
are `loaderx-host-ragged`, `loaderx-amdgpu-ragged`, and
`torch-packed-ragged`. The Torch ragged path is a packed equivalent with no
padding. The current one-pass results cover:

- **Vision (Dense)** — 2,500 `(3,224,224)` uint8 records, batch 256
  (36.8 MiB), one 200-batch pass.
- **Tokens (Ragged)** — 100,000 variable-length WikiText token sequences
  (p10/p50/p90 ≈ 28/129/255), batch 256, no padding.

Backends, per workload:

- `loaderx-host` / `loaderx-host-ragged`: zrecord raw → prefetched `Loader` →
  DLPack → CPU Torch tensors.
- `loaderx-amdgpu` / `loaderx-amdgpu-ragged`: zrecord raw → allocator-owned dma-buf
  → HIP import → DLPack → `torch.from_dlpack`; ragged yields `values` +
  `cu_seqlens + shapes` (the packed record form).
- `torch`: four-worker `torch.utils.data.DataLoader` over one `.npy` mmap, then
  `.to("cuda")`.
- `torch-packed-ragged`: in-process `DataLoader` over per-record `.npy` files,
  packed collation, then `.to("cuda")`.

Correctness and ten warmup batches precede timing. Memory tracking starts after
warmup and initial GPU synchronization and stops after the timed pass's final
GPU synchronization. `peak PSS` approximates physical CPU memory by
proportionally charging shared pages across the complete Linux process tree;
`peak agg RSS` sums every process's RSS and therefore double-counts shared
pages. Both are absolute peaks, so retained setup allocations contribute to the
baseline even though setup and teardown are outside the sampled pass. Neither
metric includes GPU device memory. Measured on the AMD Ryzen AI 9 HX PRO 370 / Radeon
890M (gfx1150) under ROCm 10 / torch 2.12.

**Vision (Dense):**

| loader | storage | batches/s | samples/s | krecords/s | peak PSS | peak agg RSS | device |
|---|---|---:|---:|---:|---:|---:|---:|
| **loaderx-host** | zrecord-raw | 192.4 | 49264 | 49.3 | 1193.9 MiB | 1198.1 MiB | cpu |
| **loaderx-amdgpu** | zrecord-raw | 408.4 | 104539 | 104.5 | 1154.2 MiB | 1158.4 MiB | amdgpu |
| torch | npy-mmap-raw | 72.5 | 18557 | 18.6 | 3182.4 MiB | 5718.5 MiB | amdgpu |

**Tokens (Ragged, no pad):**

| loader | storage | batches/s | samples/s | krecords/s | peak PSS | peak agg RSS | device |
|---|---|---:|---:|---:|---:|---:|---:|
| **loaderx-host-ragged** | zrecord-raw | 2295.0 | 587529 | 587.5 | 756.3 MiB | 760.4 MiB | cpu |
| **loaderx-amdgpu-ragged** | zrecord-raw | 1993.8 | 510402 | 510.4 | 895.7 MiB | 899.9 MiB | amdgpu |
| torch-packed-ragged | npy-list-packed | 108.7 | 27829 | 27.8 | 900.9 MiB | 905.1 MiB | amdgpu |

In these passes, loaderx AMDGPU is **5.6x** Torch on Dense and **18.3x** on ragged
tokens. Host `Loader` is **2.7x** Torch on Dense and **21.1x** on ragged
tokens. Dense Torch also peaks at 3.2 GiB process-tree PSS versus 1.2 GiB or less
for both loaderx paths. The throughput margins include storage layout,
collation, and transfer differences; they are end-to-end loader comparisons,
not isolated claims about one component.

**Current contract.** Loaderx batches sampling, gather, and AMDGPU device
delivery while preserving exact record semantics. Dense serves fixed-shape
arrays directly; Ragged delivers `values + cu_seqlens + shapes` (no padding,
host and device alike). The AMDGPU path lets Store fill the final
device-consumable allocation and avoids an intermediate host batch and explicit
framework H2D transfer; file I/O remains the ordinary Store path.

These numbers measure warm-cache access, not cold disk or durability. Store
reads accumulate at least two timed seconds, while sampler and loader figures
can fluctuate with machine load. Treat absolute values as ballpark and
cross-backend margins as the main signal.

## Real-data verification

Production dataset-specific preprocessing and verification remain in the
**[DataPipe](https://codeberg.org/eoelab/DataPipe)** repository. The built-in
converter covers materialized Arrow-backed Hugging Face datasets; DataPipe
handles sources without a common remote protocol and implements derived
modalities as loader transforms without NumPy dump intermediates. Oxford-IIIT
Pet is only the shared benchmark fixture.

## Current Limitations
* Single-host only; multi-host training is not supported.
* A single sample must be at most 2 GiB. Store size is practically bounded by
  disk capacity and platform file limits.
* Stores are not portable between machines with different byte orders. All
  published platforms are little-endian.

## 设计文档

- [性能优化与工程决策记录](docs/性能优化.md)
- [分布式架构草案](docs/分布式架构.md)

## Build
```
python3 setup.py build_ext --inplace
zig build test
python3 scripts/test_loaderx.py
```

The regular suite is CPU-only and uses mocks or memfd regions for device
contracts. On a machine configured with an AMD GPU and ROCm Torch, run the
separate mandatory hardware certification; its JAX checks run when a ROCm JAX
backend is installed:

```bash
python3 scripts/test_rocm.py
```

Build the release wheel matrix with:

```
python3 scripts/build_release.py
```

Source distributions are intentionally not published. The wheel matrix covers
the supported platforms. Source builds require Zig.
