Metadata-Version: 2.4
Name: zarr-vlen-ndarray
Version: 0.1.2
Summary: Typed variable-length ndarray data type (vlen-ndarray) for Zarr v3
Project-URL: Homepage, https://github.com/espg/zarr-vlen-ndarray
Project-URL: Repository, https://github.com/espg/zarr-vlen-ndarray
Project-URL: Issues, https://github.com/espg/zarr-vlen-ndarray/issues
Project-URL: Specification, https://github.com/espg/zarr-vlen-ndarray/blob/main/SPEC.md
Author-email: Shane Grigsby <refuge@rocktalus.com>
License: MIT License
        
        Copyright (c) 2026 Shane Grigsby
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Requires-Dist: numcodecs>=0.14
Requires-Dist: numpy>=1.26
Requires-Dist: zarr>=3.1.0
Provides-Extra: test
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

# zarr-vlen-ndarray

Typed variable-length ndarray data type (`vlen-ndarray`) for Zarr v3.

Each array element is a variable-length ndarray of shape `(n, *inner_shape)`:
the leading dimension `n` varies per element; the trailing `inner_shape` and
the scalar `dtype` are fixed in the data type's configuration. One
parameterized type covers, e.g., `(n, 2)` float32 centroid sets and `(n,)`
uint64 location lists.

```json
{
  "name": "vlen-ndarray",
  "configuration": {"dtype": "float32", "inner_shape": [2]}
}
```

The package provides:

- **`VlenNDArray`** — a zarr-python `ZDType` for the data type above;
- **`VlenNDArrayCodec`** — the paired parameter-free `array -> bytes` codec,
  which serializes each element as its raw little-endian C-order bytes and
  frames chunks exactly like zarr's `vlen-bytes` codec (numcodecs `VLenBytes`
  framing: u32le count, then per element u32le length + payload).

The normative extension text is in [SPEC.md](SPEC.md) and the
[`registry/`](registry/) directory (zarr-extensions house style).

## Why: byte identity with `bytes` + `vlen-bytes`

For any chunk, the encoded bytes are **byte-identical** to what zarr's
`vlen-bytes` codec produces for the equivalent raw-bytes payloads (proved in
`tests/test_byte_identity.py`, including through a zstd chain). A store that
keeps ragged data as the `bytes` data type with raw-LE-bytes elements — the
[`zagg-ragged/1`](https://github.com/englacial/zagg/issues/340) convention —
can be upgraded to the typed form by rewriting `zarr.json` only; chunk
objects are untouched. Downgrade is equally metadata-only, which is the
escape hatch for readers without this package.

This package is phase 1 of
[englacial/zagg#210](https://github.com/englacial/zagg/issues/210) and
defines the element type for the `zagg-ragged/2` store revision
([englacial/zagg#340](https://github.com/englacial/zagg/issues/340)).

## Install

From [PyPI](https://pypi.org/project/zarr-vlen-ndarray/):

```sh
pip install zarr-vlen-ndarray
```

Or from source:

```sh
pip install git+https://github.com/espg/zarr-vlen-ndarray
```

Requires Python >= 3.11, `zarr >= 3.1.0` (the ZDType API), `numcodecs >= 0.14`.

## Usage

```python
import numpy as np
import zarr
import zarr_vlen_ndarray  # registers the data type and codec
from zarr_vlen_ndarray import VlenNDArray, VlenNDArrayCodec

zdtype = VlenNDArray(dtype="float32", inner_shape=(2,))
arr = zarr.create_array(
    store="example.zarr",
    shape=(4,),
    chunks=(4,),
    dtype=zdtype,
    serializer=VlenNDArrayCodec(),   # required: zarr has no default serializer for extension dtypes
    compressors=zarr.codecs.ZstdCodec(level=3),
)

cells = np.empty(4, dtype=object)   # object-array staging, as with zarr's vlen types
cells[:] = [
    np.random.random((3, 2)).astype("float32"),
    np.empty((0, 2), dtype="float32"),          # empty cell
    np.random.random((7, 2)).astype("float32"),
    np.random.random((1, 2)).astype("float32"),
]
arr[:] = cells

arr[:][0]        # -> float32 ndarray of shape (3, 2)
```

Reading requires `import zarr_vlen_ndarray` first (see the registration
section below); after that, plain `zarr.open` works.

### Semantics worth knowing

- **Slice reads** return object arrays whose cells are float32/uint64/...
  ndarrays. Decoded cells are **read-only views** over the decoded payload
  (zero-copy); call `.copy()` to mutate.
- **Scalar reads** (`arr[0]`) return the cell wrapped in a 0-d object array —
  zarr-python does the same for its own vlen types. `zarr_vlen_ndarray.unbox`
  recovers the cell.
- **Fill values**: the default fill is the empty cell `(0, *inner_shape)`
  (metadata `fill_value: ""`, base64 of zero bytes). Cells materialized from
  the fill are `VlenScalar` instances — ndarray subclasses whose `==`/`!=`
  compare the whole cell and return a plain bool (this is what makes zarr's
  empty-chunk elision work for ndarray cells; in every other respect they
  behave like regular ndarrays).

## Registration and the no-package failure mode

The package declares both zarr entry points:

```toml
[project.entry-points."zarr.data_type"]
vlen-ndarray = "zarr_vlen_ndarray:VlenNDArray"

[project.entry-points."zarr.codecs"]
vlen-ndarray = "zarr_vlen_ndarray:VlenNDArrayCodec"
```

and also registers eagerly on import. Status with current zarr-python
(3.1.0–3.2.1, pinned by `tests/test_registration.py`): the **codec** entry
point is discovered lazily by zarr as designed, but zarr collects
`zarr.data_type` entry points into the data type registry's lazy-load list
**without ever flushing it**, so data type discovery via entry points is
currently inert upstream. Practical rule: **`import zarr_vlen_ndarray` before
opening a store**. When upstream flushes the lazy list, the explicit import
becomes unnecessary automatically.

A vanilla zarr user *without* this package installed who opens a
`vlen-ndarray` store gets (verbatim):

```
ValueError: No Zarr data type found that matches {'name': 'vlen-ndarray', 'configuration': {'dtype': 'float32', 'inner_shape': [2]}}
```

The fix is `pip install zarr-vlen-ndarray` + `import zarr_vlen_ndarray` — or,
without installing anything, the metadata-only downgrade to `bytes` +
`vlen-bytes` described above.

## Registry status

The `vlen-ndarray` name is **submitted but not yet registered** with
[zarr-developers/zarr-extensions](https://github.com/zarr-developers/zarr-extensions):
see [zarr-extensions#71](https://github.com/zarr-developers/zarr-extensions/pull/71).
Until that PR merges, treat the name as provisional. The registry-formatted
spec files in [`registry/`](registry/) are the ones under review.

## Development

```sh
uv sync --extra test
uv run pytest -v
uv run ruff check src tests && uv run ruff format --check src tests
uv run mypy src
```

CI runs the test matrix on Python 3.11–3.13 against latest zarr, plus a floor
job against `zarr==3.1.0`.

## License

MIT
