Metadata-Version: 2.4
Name: dataio-rs
Version: 0.5.4
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Free Threading :: 2 - Beta
Requires-Dist: numpy>=2.0,<3 ; extra == 'bench'
Requires-Dist: pillow>=10,<12 ; extra == 'bench'
Requires-Dist: torch>=2.6,<2.10 ; extra == 'bench'
Requires-Dist: torchvision>=0.21,<0.25 ; extra == 'bench'
Requires-Dist: opencv-python-headless>=4.10,<5 ; extra == 'bench'
Requires-Dist: safetensors>=0.4.5,<0.7 ; extra == 'bench'
Requires-Dist: dataio-rs[bench] ; extra == 'bench-dali'
Requires-Dist: nvidia-dali-cuda120>=1.44,<1.52 ; extra == 'bench-dali'
Requires-Dist: dataio-rs[bench] ; extra == 'bench-spdl'
Requires-Dist: spdl>=0.0.8,<0.1 ; python_full_version >= '3.10' and extra == 'bench-spdl'
Requires-Dist: dataio-rs[bench] ; extra == 'bench-tensorflow'
Requires-Dist: tensorflow-cpu>=2.18,<2.21 ; extra == 'bench-tensorflow'
Provides-Extra: bench
Provides-Extra: bench-dali
Provides-Extra: bench-spdl
Provides-Extra: bench-tensorflow
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES
Summary: High-performance data processing library for ML workloads.
License-Expression: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Package, https://pypi.org/project/dataio-rs/
Project-URL: Repository, https://github.com/Mikubill/dataio-rs

# dataio

[![CI](https://github.com/Mikubill/dataio-rs/actions/workflows/ci.yml/badge.svg)](https://github.com/Mikubill/dataio-rs/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/dataio-rs.svg)](https://pypi.org/project/dataio-rs/)
[![Python](https://img.shields.io/pypi/pyversions/dataio-rs.svg)](https://pypi.org/project/dataio-rs/)

dataio is a Rust-backed data loader and IO layer for Python. It reads and writes
local files, S3, R2, GCS, Azure, HTTP(S) and the Hugging Face Hub, runs
decode/transform/batch pipelines on native worker pools, and returns NumPy,
PyTorch, JAX, DLPack or Arrow data. It supports free-threaded CPython and ships a
command line that replaces s5cmd.

## Install

```bash
pip install dataio-rs
python -m dataio  # report this host's capabilities
```

Python 3.11 or newer on Linux x86_64 (glibc 2.28+) gets a prebuilt wheel: abi3 for
CPython with the GIL, cp314t for free-threaded CPython 3.14; other platforms build
from the sdist ([Develop](#develop)). Nothing else is required: install NumPy,
PyTorch, JAX or PyArrow for the outputs you use. CUDA, nvJPEG, cuFile (GDS) and
libfabric load on first use, so the wheels run without them. To upgrade, read the
[changelog](CHANGELOG.md) and the [migration guide](docs/migration.md).

## Quick start

This runs without input files or a framework:

```python
import dataio

records = [{"key": "a", "payload": b"red"}, {"key": "b", "payload": b"sky"}]
pipe = dataio.pipeline(dataio.slot("payload").decode_bytes(), dataio.batch(2))
with pipe.bind(records).run("dlpack") as loader:
    for batch in loader:
        print(batch.keys, batch["payload"].shape)  # ['a', 'b'] [2, 3]
```

`output=` selects the framework. The loader's `run()` defaults to `"torch"` and
also takes `"numpy"`, `"jax"`, `"dlpack"`, a name registered with
`register_output_adapter`, or a callable. Readers default to `"numpy"` (`read_dcp`
to `"torch"`): `read_image` takes the loader's choices, and tensor readers such as
`read_safetensors` take `"numpy"` or `"torch"`. Column-batch iterators
(`iter_batches`, `iter_arrow_batches`) yield `{column: [values]}` unless
`output="numpy"`. The [examples](examples/README.md) are complete scripts.

## Build a pipeline

A pipeline is optional record maps, one or more slot chains, one batch stage
(`batch`, `collate` or `group_by`), then optional batch transforms and maps.
Building it does no IO; `.bind(records).run(...)` starts a loader.

```python
pipe = dataio.pipeline(
    dataio.map(lambda row: {"image": row["path"], "label": row["class_id"]}),
    dataio.slot("image").decode_image(mode="rgb").resize_short(size=256)
        .center_crop(h=224, w=224).normalize("imagenet"),
    dataio.slot("label").passthrough(),
    dataio.batch(64, drop_last=True, prefetch=4),
)
with pipe.bind(records).run("torch", concurrency=64) as loader:
    for batch in loader:
        train_step(batch["image"], batch["label"])
```

Slot methods are shorthand for the factories in these namespaces:

| Namespace | Operations |
| --- | --- |
| `dataio.decoders` | Image, audio (`resampling="sinc"` anti-aliases), JSON, NPY, safetensors, text, bytes and PT decoding |
| `dataio.transforms` | Resize, crop, normalize, augment and conditional transforms |
| `dataio.batch_transforms` | MixUp, CutMix, padding, shuffle and normalization |
| `dataio.cond` | Field and shape predicates for slot activation and transforms |

A `dataio.map` before the batch stage gets input records, one after it the decoded
`Batch`: a passthrough slot plus `dataio.map(lambda b: collate_fn(b["sample"]))`
reuses an existing collator. `collate()` keeps the batches a source yields,
`group_by(key)` batches each run of equal keys, and
`from_sampler(batch_sampler, row_fn)` tags a batch sampler's rows for
`group_by("batch_id")`. Examples: [batch boundaries](examples/stream_shapes.py),
[conditional slots](examples/conditional_slots.py),
[conditional transforms](examples/conditional_branch.py); contracts:
[callbacks](docs/arch.md#composition-and-callbacks).

## Choose sources

`bind` takes a reusable iterable or a zero-argument factory and reads it once per
epoch; `from_iterable(...)` is single-pass. Wrap a source to index, compose,
shuffle or mix it:

```python
photos = dataio.records(dataset, indexed=True)  # len() + integer __getitem__
art = dataio.webdataset(manifest="s3://b/art/manifest.json")
records = dataio.mix([photos, art], [0.7, 0.3], seed=0, stop="first")
```

| Call | Effect |
| --- | --- |
| `records(dataset, indexed=True)` | Integer row access, bypassing the dataset's iterator; with `batch(N)`, `__getitems__(indices)` fetches rows together |
| `.map_records(fn)`, `.flat_map_records(fn)`, `.concat(*sources)` | Replace one record, emit zero or more, append sources without opening them early |
| `.shuffle(seed=42, buffer_size=4096)` | Mix nearby records in a buffer of that many records; no `buffer_size` means a full permutation |
| `mix(sources, weights, seed=, stop=)` | Interleave indexed sources by weight into an indexed source; `stop="first"` ends before a source repeats, `"all"` once every source was read, `"never"` after `length=` samples |
| `webdataset("shard-{0000..0099}.tar")`, `webdataset(manifest=...)` | Tar shards; a `shard_writer` manifest opens as an indexed, seekable source |

Expanded and streaming sources need a buffered shuffle. Contracts:
[record sources](docs/arch.md#record-sources), [bulk reads](docs/arch.md#indexed-and-bulk-reads),
[weighted mix](docs/arch.md#weighted-mix); examples:
[callable sources](examples/callable_source.py), [composition](examples/iterable_source.py).

## Configure execution

Loader options and process-wide environment settings bound concurrency and
memory; `dataio.env_vars()` lists every setting with its default.

| Setting | Effect |
| --- | --- |
| `run(..., concurrency=N)` | Per-shard stage admission; start with the default `"auto"`, measure higher values for high-latency sources |
| `batch(..., prefetch=K)` | Batches in flight |
| `run(..., max_prefetch_bytes=B)` | Logical bytes of pending native outputs with a fixed layout; not an RSS limit |
| `run(..., map_workers=N)` | Threads for record maps, sized-source `__getitem__` and `from_sampler` record functions; output equals `N=1`, callbacks must be thread-safe |
| `run(..., order="completion")` | Deliver batches as they finish instead of in submit order; no mid-epoch resume |
| `DATAIO_CPU_WORKERS` / `DATAIO_IO_WORKERS` | Process-wide CPU / async IO worker pools |
| `DATAIO_HTTP_MAX_INFLIGHT_READS` | Process-wide GET/HEAD limit; `0` means unlimited |
| `DATAIO_OBJECT_SHARDS` | HTTP and object-store client shard fanout |

From async code, run `async for batch in batches` inside
`async with loader.aiter() as batches`: it reuses native prefetch and closes on
early exit. One iteration owns a loader at a time. Scheduling, memory bounds and
cleanup: [plans and execution](docs/arch.md#plans-and-execution),
[iteration and cancellation](docs/arch.md#iteration-and-cancellation).

## Handle errors

`run(..., on_failure=...)` decides what a failed row does; `batch.errors` lists each
failure (`index`, `key`, `stage`, `op`, `message`).

| `on_failure=` | Effect |
| --- | --- |
| `"skip"` (default) | Drop failed rows; raise when no row of a batch survives |
| `"strict"` | Raise on a failed row |
| `N` | Drop failed rows; raise when fewer than `N` rows of a batch survive |

`log_errors=True` logs each failure to the `dataio.rows` logger, or pass a callable.
Source-iteration and Python batch-map errors propagate as raised
([callback contracts](docs/arch.md#composition-and-callbacks)).

## Shard and resume

`shard=` splits each epoch across ranks; `state_dict()` / `load_state_dict()`
checkpoint a loader mid-epoch. A keyed shuffle reshuffles a huge indexed dataset
every epoch without materializing its index, and restores on any world size:

```python
records = dataio.records(dataset, indexed=True, resume="seek")
loader = pipe.bind(records).run(
    "torch", shard={"id": rank, "of": world_size}, seed=42, shuffle="keyed",
)  # or shuffle=("block", 1024, 8) to read 1024-record blocks within 8-block windows
states = [None] * world_size
torch.distributed.all_gather_object(states, loader.state_dict())  # save `states`
# After restarting on any world size, every rank restores the gathered list:
loader.load_state_dict(states)
```

Ranks save after the same step; on an unchanged layout a rank can restore its own
`state_dict()` alone. `"pad": True` gives every rank the same batch count. A
restore replays the consumed prefix unless `resume="seek"` lets an indexed source
with index-determined lookups skip it. Contracts:
[resume](docs/arch.md#replay-and-indexed-resume), [checkpoint identity](docs/arch.md#checkpoint-identity),
[elastic resume](docs/arch.md#keyed-shuffles-and-elastic-resume).

## Read and write objects

Local paths, `s3://`, `r2://`, `gs://`, `az://`, `hf://` and HTTP(S) URIs share
these operations where the backend supports them:

```python
dataio.configure_endpoint("r2", "https://<account>.r2.cloudflarestorage.com")
dataio.configure_transport("https", timeout_secs=30, readall_concurrency=32)
chunk = dataio.read("s3://bucket/file.bin", offset=0, size=1 << 20)
parts = dataio.fetch("s3://bucket/file.bin", ranges=[(0, 64), (4096, 256)])  # one GET
dataio.write("s3://bucket/out.bin", chunk, if_none_match="*")  # create only
etag = dataio.copy("s3://bucket/out.bin", "s3://bucket/copy.bin")
dataio.exists("s3://bucket/out.bin")  # False only when missing; other errors raise
dataio.ls("s3://bucket/prefix/", recursive=False, detail=True)  # one level, size/etag/mtime
dataio.glob("s3://bucket/**/*.safetensors")
payload = dataio.reader("s3://bucket/shard.tar.gz").gunzip().tar_entry("0001.jpg").read_all()
with dataio.writer("s3://bucket/checkpoint.bin") as writer:
    writer.write_chunks(huge_generator())
with dataio.shard_writer("s3://bucket/out/shard-{:06d}.tar", max_bytes=1 << 30) as shards:
    for key, jpg, meta in samples():
        shards.write({"__key__": key, "jpg": jpg, "json": meta})
records = dataio.webdataset(manifest=shards.manifest)  # indexed; no tar headers read
```

`fetch_into(uri, buffer)` fills an existing buffer, `stream(uri)` reads a body on
demand (bytes, lines or server-sent events), `head_many(uris)` sends HEADs
concurrently, and `copy(recursive=True)`, `sync` and `delete(recursive=True)` work
on trees like the [command line](#command-line). Requests retry throttling, 5xx,
resets and timeouts; `max_attempts=` and `deadline_secs=` bound one call.
On `s3://` and `r2://`, the store verifies every upload's CRC64NVME, and every
whole-object download is checked against the stored one.

`s3://` uses the AWS credential chain (environment, profiles, SSO, web identity,
ECS/EKS Pod Identity, instance metadata); `configure_credentials(scheme, ...)` sets
explicit credentials, and `clear_credentials(scheme)` re-reads them. Details:
[network configuration](docs/arch.md#network-configuration),
[byte and format IO](docs/arch.md#byte-and-format-io).

## Command line

`python -m dataio` runs the same operations with the same credentials, so shell
scripts need no other S3 tool; `python -m dataio <command> --help` lists its flags.

```bash
python -m dataio cp model.safetensors s3://bucket/models/     # -> .../models/model.safetensors
python -m dataio cp -r --workers 32 s3://bucket/models/vae /nvme/vae
python -m dataio cp -n 's3://bucket/ckpt/__3_*.distcp' /nvme/ckpt/
python -m dataio sync --workers 128 --delete --no-follow-symlinks /fsx/run s3://bucket/archive/run
python -m dataio rm -r s3://bucket/scratch/run-1              # prints "deleted 1234"
python -m dataio sync --dry-run --delete --exclude '*.tmp' /fsx/run s3://bucket/run   # plan only
python -m dataio --json sync s3://bucket/data /nvme/data      # {"copied": .., "skipped": .., "deleted": ..}
python -m dataio head --max-attempts 1 --deadline 5 s3://bucket/ckpt/COMMIT   # a fast probe
```

| Command or flag | Effect |
| --- | --- |
| `cp SRC DST` | Copy a file or object (into `DST/<name>` when `DST` ends in `/` or is a local directory), a directory or prefix with `-r`, or every match of a wildcard |
| `sync SRC DST` | Copy the files of a directory or prefix that are missing or differ under `DST` |
| `rm URI` | Delete a file or object, or a directory or prefix with `-r` |
| `head URI`, `cat URI`, `ls URI` | Print metadata (exit 1 when missing), the body (`-o FILE` saves it), or the URIs below a prefix (`--long` adds sizes) |
| `info` (the default), `parquet PATH`, `safetensors PATH` | Print host capabilities, Parquet metadata or safetensors entries |
| `cp -n` | Skip destinations that exist |
| `cp --etag ETAG` | Copy one object only if it still has this ETag |
| `sync --delete` | Remove destination files the source lacks |
| `sync --strategy S` | `size` (default) compares sizes; `mtime` also copies files modified after their copy, between local trees; `checksum` also compares equal-size files by CRC64NVME (s3, r2, local) |
| `sync --no-follow-symlinks` | Skip symlinks in a local source |
| `--include PATTERN`, `--exclude PATTERN` (`cp -r` or a wildcard, `sync`, `rm -r`) | Select files by their path relative to the source: an include pattern, if any, must match and no exclude pattern may (`*` also matches `/`; repeatable) |
| `--dry-run` (`cp`, `sync`, `rm`) | Print each planned `copy SRC DST` / `delete URI` and the summary; change nothing |
| `cp - DST` | Stream stdin to `DST` (`-n`: create only) |
| `--workers N` (`cp`, `sync`) | Transfers in flight (default 64) |
| `--max-attempts N` (not `info`, `parquet`, `safetensors`) | Attempts per request on throttling, 5xx, resets and timeouts (default `DATAIO_OBJECT_MAX_ATTEMPTS`, 4) |
| `--deadline SECS` (likewise) | One bound for the whole command: listing, transfers, retries and backoff |
| `--json` (before the command) | Print JSON instead of text |

**Output.** `cp`, `sync` and `rm -r` print one summary line (`copied 3, skipped 1`),
or one JSON object with `--json`, and nothing per file. A failure prints one stderr
line and exits 1; the first failure stops the remaining transfers, and re-running
`sync` resumes. A usage error exits 2. Printed URLs drop userinfo and query
strings, so a presigned signature never reaches a terminal or log.

Recursive commands treat a prefix as a directory (`s3://b/run` never reaches
`s3://b/run2/`), and `sync` refuses a source with no file, so a mistyped source
never empties the destination under `--delete`. Wildcards, filters, dry runs, path
checks and retry limits: the [command-line contract](docs/arch.md#command-line).

Replacing s5cmd 2.2.2:

| s5cmd | dataio |
| --- | --- |
| `cp 's3://b/p/*' dir/`, `cp 'dir/*' s3://b/p/` | `cp -r s3://b/p dir`, `cp -r dir s3://b/p` (local hidden files included) |
| `cp 's3://b/p/*.parquet' dir/`, `cp -n 's3://b/p/__3_*' dir/` | the same pattern (`-n` as well) |
| `cp FILE s3://b/p/`, `cp s3://b/k /tmp/f` | the same |
| `sync 's3://b/p/*' dir/`, `sync dir/ s3://b/p/` | `sync s3://b/p dir`, `sync dir s3://b/p` (no wildcards) |
| `--numworkers N` | `--workers N` (default 64) |
| `--retry-count N` | `--max-attempts N+1` (per command; default 4 attempts) |
| `timeout SECS s5cmd ...` | `--deadline SECS` (exit 1 with a `TimeoutError` line) |
| `sync --size-only` | default `--strategy size` |
| `sync --size-only`, then an `ls` count/bytes check | `sync --strategy checksum`: exit 0 means every file's content matches |
| `sync` (size + newer mtime), local trees | `--strategy mtime` |
| `sync --delete`, `--no-follow-symlinks` | the same flags |
| `--exclude P`, `--include P` (`cp`, `sync`, `rm`) | the same flags (paths relative to the source; `*` also matches `/`) |
| `--dry-run` | the same flag (`copy` / `delete` lines, then the summary line) |
| `--exit-on-error` | always |
| `--log error`, `--stat` | default: no per-file lines, one summary line |
| `--concurrency N`, `--part-size MiB` (per-file parts) | `DATAIO_MULTIPART_CONCURRENCY` (upload parts in flight per process, 64) / `DATAIO_READALL_CONCURRENCY` (ranges per download, 32), `DATAIO_MULTIPART_PART_MIB` (16; parts grow past it to fit 10 000) / `DATAIO_READALL_CHUNK_MAX_MIB` |
| `rm 's3://b/p/*'`, `rm 's3://b/p/*.tmp'` | `rm -r s3://b/p`, `rm -r --include '*.tmp' s3://b/p` |
| `ls 's3://b/p/*'` (count, bytes) | `--json ls --long s3://b/p/` (`count`, `bytes`) |
| `cat s3://b/k` | `cat s3://b/k` |
| `pipe DST` | `cp - DST` (`-n`: create only) |
| `ls s3://b/k` as an existence test | `head s3://b/k` (exit 1 when missing), or `dataio.exists` in Python |
| `run FILE` (a batch of commands) | not a command: call `dataio.copy` from Python, or `cp -r` / a wildcard |
| `credential_process` shim for Pod Identity | not needed |

## Read structured formats

Typed readers return arrays, images, columns and Arrow data without a pipeline:

```python
array = dataio.read_numpy("/data/x.npy")
image = dataio.read_image("s3://bucket/image.png", mode="rgb")
columns = dataio.read_parquet("/data/shard.parquet", columns=["key", "height"])
tall = dataio.scan_parquet("/data/shard.parquet").filter(lambda r: r["height"] > 512).collect()
stream = dataio.scan_feather("/data/table.arrow").select(["key"]).collect_arrow_stream()
for batch in dataio.scan_parquet("/data/features.parquet").iter_batches(output="numpy"):
    features, tokens = batch["features"], batch["tokens"]  # ndarray, RaggedArray
for example in dataio.iter_tfrecord("/data/train.tfrecord"):
    label = example["label"]
```

Scans keep Arrow types (`collect_arrow_stream()`, or PyArrow through
`collect_arrow_reader()` / `collect_arrow()`). `iter_batches(output="numpy")` needs
only NumPy: structs become dicts, fixed-size lists add dimensions, and
variable-length lists become `RaggedArray(values, offsets)` (row `i` is
`values[offsets[i]:offsets[i + 1]]`). `iter_arrow_batches(source, output="numpy")`
does the same for any `__arrow_c_stream__` producer, such as a DuckDB relation.
Types, nulls and views: [columnar NumPy batches](docs/arch.md#columnar-numpy-batches).

## Read or fill safetensors weights

Single-file and sharded checkpoints use the same API:

```python
dataio.write_safetensors("checkpoints/model/", state_dict, max_shard_size="5GB")
subset = dataio.read_safetensors("checkpoints/model/", keys=["layer.weight"], output="torch")
with dataio.open_safetensors("weights.safetensors") as source:
    tensor = source.read(keys="layer.weight", output="torch")["layer.weight"]
    source.read_into(destinations, chunk_bytes=16 << 20)
```

`read_into` checks every shape, dtype and layout, then fills existing NumPy arrays
or CPU/CUDA Torch tensors (`destinations` maps names to them) in bounded chunks,
without another full copy. On local GDS-ready storage,
`read_safetensors(..., output="torch", device="cuda:0")` reads straight to the GPU,
falling back to CPU with a warning; `gds_available()` and `gds_info()` report
support. Views, copies and CUDA staging: [buffer ownership](docs/arch.md#buffer-ownership-and-dlpack).

## Convert distributed checkpoints

`scan_dcp` reassembles a PyTorch distributed checkpoint (DCP) into safetensors;
`read_dcp` returns its tensors:

```python
dataio.scan_dcp("s3://bucket/checkpoint", keys="model.", bf16=True).rename(
    lambda name: name.removeprefix("model.")
).write("weights.safetensors", strategy="stream")
```

DCP metadata is a Python pickle: load only trusted checkpoints. Selection,
streaming and layouts: [distributed checkpoints](docs/arch.md#distributed-checkpoints).

## Transfer with RDMA

`dataio.rdma` does one-sided reads and writes through libfabric, loaded on first use:

```python
import dataio.rdma as rdma

with rdma.serve(bind="0.0.0.0:0", buffer_size=1 << 20) as server:
    with rdma.connect(server.local_addr) as client:
        client.write_bytes(0, b"hello rdma")
        assert client.read_bytes(0, 10) == b"hello rdma"
```

`serve(bind=..., file=path)` exposes file ranges; `rdma.read_into` fills
caller-owned CPU or CUDA buffers. `available()` also succeeds over TCP software
providers, so check `info()` or `diagnose()` for physical RDMA. GPUDirect RDMA
needs an HMEM provider and libfabric API 1.9+ ([RDMA transport](docs/arch.md#rdma-transport)).

## API index

| Task | Entry points |
| --- | --- |
| Build a pipeline | `pipeline`, `slot`, `slots`, `chain`, `batch`, `collate`, `group_by` |
| Extend processing | `map`, `custom_op`, `register_output_adapter` |
| Bind records | `records`, `files`, `urls`, `manifest`, `from_callable`, `from_iterable`, `from_sampler`, `webdataset`, `archive`, `mix` |
| Work with bytes | `read`, `fetch`, `fetch_into`, `write`, `copy`, `sync`, `head`, `head_many`, `stream`, `ls`, `glob`, `exists`, `delete` |
| Compose IO | `reader`, `writer`, `shard_writer`, `scan`, `from_iterator` |
| Work with tensors | `read_safetensors`, `open_safetensors`, `write_safetensors`, `scan_dcp`, `read_dcp`, `scan_pt`, `read_pt` |
| Work with arrays/tables | `read_*` / `scan_*` / `write_*` for `numpy`, `npz`, `parquet`, `feather`, `tfrecord`; `iter_arrow_batches`, `read_image`, `iter_tfrecord` |
| Configure IO | `configure_credentials`, `configure_endpoint`, `configure_transport`, `clear_credentials` |
| Use GDS | `gds_open`, `gds_register_buffer`, `gds_read_into_tensor` |

The [documentation index](docs/README.md) links the contracts, the
[benchmark guide](docs/benchmark.md) and the roadmap.

## Develop

```bash
uv run --no-sync --with maturin maturin develop --profile staging
cargo test
uv run --no-sync python -m pytest python/tests -q
uv run --no-sync --with maturin maturin build --profile staging --out dist  # a wheel
```

The toolchain is pinned in `rust-toolchain.toml`. Source builds omit RDMA: install
libfabric headers and add `--features rdma` (published wheels include it). Under a
free-threaded interpreter (`uv venv --python 3.14t`) the same commands build the
cp314t extension; `maturin build -i python3.11 python3.14t` builds both wheels. The
native IO, network, Arrow and format crates build without Python.

## License

[Apache-2.0](LICENSE). Bundled dependency licenses are in
[THIRD_PARTY_NOTICES](THIRD_PARTY_NOTICES).

