Metadata-Version: 2.5
Name: sporadik
Version: 1.1.0
Summary: a sparse array wire format: one axis made contiguous, written as a zarr prefix any object store can serve
Author-email: jhnnsrs <jhnnsrs@gmail.com>
License-Expression: MIT
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: zarr>=3.0
Provides-Extra: build
Requires-Dist: uv>=0.7.12; extra == 'build'
Provides-Extra: complete
Requires-Dist: obstore>=0.10; extra == 'complete'
Requires-Dist: scipy>=1.11; extra == 'complete'
Provides-Extra: obstore
Requires-Dist: obstore>=0.10; extra == 'obstore'
Provides-Extra: scipy
Requires-Dist: scipy>=1.11; extra == 'scipy'
Description-Content-Type: text/markdown

# sporadik

**A sparse array wire format.** One zarr prefix, one child per *axis made contiguous*, and a block
written last that says which children the writer finished.

It is strictly a *format*: no network, no client, no storage opinions, and no idea what the numbers
mean. Two dependencies is the whole of it.

It defines no container of its own. A layout is anndata's spelling, so at rank two the three arrays
are **byte-identical** to what `scanpy` writes and a browser reads one with `zarrita`'s `open` and
`get` rather than a decoder. What sporadik adds is the part anndata has no opinion about: which axis
a layout makes contiguous, how to hold more than one of them, how to tell a finished upload from a
torn one, and what all of that means above rank two.

---

## The idea

A **layout** is one axis made contiguous. Pick the axis a reader will select along, ravel the
remaining axes into `indices`, and `indptr` names one run per position along the chosen one.

At rank two that construction *is* CSR (axis 0) and CSC (axis 1). At rank three it is the same
construction with more axes folded into the same `indices`. One invariant holds at every rank, and
it is the spine of the format:

```
len(indptr) == shape[indexed_axis] + 1
```

So an array of rank *n* has up to *n* layouts, and each buys exactly one question — *"everything at
this position along axis k"* — at the cost of another copy of the nonzeros. Ask a layout the
question it does not compress for and there is no range to read at all, only a scan: **1 777 ms
against 2.2 ms**, measured on a 16 µm spatial-transcriptomics matrix.

---

## The layout on disk

```text
<prefix>/
  zarr.json                     attributes: {"sporadik": {...}}   <- written LAST
  layouts/
    axis0/                      a sparse group, anndata-spelled
      zarr.json                 encoding-type, encoding-version, shape
      data/ indices/ indptr/
    axis1/                      the same array, contiguous along another axis
```

Children are named for **the axis they make contiguous**, not for an encoding: `csr`/`csc` name the
two cases of rank two and say nothing at rank three. A reader recomputes the name and compares, so a
layout filed under the wrong one is refused rather than silently indexed along the wrong axis.

---

## The block — spec 1

Under the key `sporadik` in the root group's own `attributes`:

<!-- spec-block: parsed by tests/test_spec_document.py -- keep in step with sporadik.spec -->
```json
{
  "spec": "1",
  "complete": true,
  "shape": [91033, 19059],
  "layouts": [
    {"path": "layouts/axis0", "indexed_axis": 0, "index_order": [1]},
    {"path": "layouts/axis1", "indexed_axis": 1, "index_order": [0]}
  ]
}
```

| key | declared or derived | meaning |
|---|---|---|
| `spec` | declared | how to read the prefix. **Unknown ⇒ refuse.** |
| `complete` | declared | the writer finished. See *Why the block is last*. |
| `shape` | declared | the array's own shape, at its own rank. Checked against every layout. |
| `layouts[].path` | declared | the child group. Must equal `layouts/axis{indexed_axis}`. |
| `layouts[].indexed_axis` | declared | which axis this layout makes contiguous. |
| `layouts[].index_order` | declared | the axes it did *not* compress, in the order `indices` was raveled over them. |

Everything else a reader needs — each layout's encoding, nonzero count, dtype, chunking, and whether
it is byte-addressable — is **derived from the artifact and never declared.** That distinction is
the design: a fact read off the bytes cannot be stated wrongly.

`index_order` is the one exception, and it has to be stated because it **cannot be recovered from
the bytes**. At rank two it has one member and says nothing. Above it, a wrong `index_order` does not
fail anywhere — it reads a different cell.

---

## A layout child

Each named child is an anndata-spelled sparse group: attributes `encoding-type`, `encoding-version`
and `shape`, plus three 1-D arrays `data`, `indices`, `indptr`.

**`encoding-type` by rank.** At rank two it is anndata's exactly — `csr_matrix` compresses axis 0,
`csc_matrix` axis 1. Above rank two there is no anndata spelling of the thing, so the child holds the
array *raveled to two axes*, which genuinely is a `csr_matrix`, and says so.

**Declared `shape` by rank.** At rank two the child declares the array's own shape, so it is a real
anndata group over the real matrix. Above it, the raveled pair it literally holds:
`[shape[indexed_axis], prod(shape[a] for a in index_order)]`.

A layout group may also hold a **`maxima`** array: one value per slice, the largest absolute value in
it, computed once at write time so a reader never reduces over every value to get it. It is optional
and unannounced — a reader that wants it looks for it, and one that does not is unaffected — which is
why carrying it does not move the spec version.

---

## Why the block is last

Everything else in the prefix declares something *before* it is true: zarr writes an array's
`zarr.json` ahead of its chunks, and a layout's `encoding-type` when its group is created. Only the
block — one object, written once, after every chunk is durable — is a statement made **after** the
thing it describes exists.

That matters more than it sounds, because zarr substitutes the fill value for a chunk it cannot
fetch. That is a legitimate convention for genuinely-sparse arrays and, for an interrupted upload,
indistinguishable from success. Measured, before the block existed: deleting every chunk of `data`
left a store that passed every other check, recorded the right `nnz`, and returned the right *number*
of values for a slice — all of them zero, with nothing raised anywhere.

---

## What a conforming reader checks

Each one is silent if skipped, which is why each is listed.

1. **`len(indptr) == shape[indexed_axis] + 1`**, at every rank — what makes `indptr[i:i+2]` a run.
2. **`len(data) == len(indices)`** — parallel arrays; one without the other stopped partway.
3. **`path == layouts/axis{indexed_axis}`** — else it is read along the wrong axis.
4. **`index_order` is a permutation** of the axes other than `indexed_axis`.
5. **The child's declared shape and `encoding-type`** match the rank rules above.
6. **At most one layout per axis**, and at most *rank* of them.
7. **The block is present and `complete`.**

---

## Chunking

All three arrays are chunked, `indptr` included, at ~128 KB (32 768 four-byte elements) — sized for
**one object-store request**, because on S3 the cost is round trips rather than bytes. Measured: one
slice costs 0.95 ms at 32 768-element chunks, 3.00 ms at 512, and 23.55 ms at 4 Mi.

A chunk is also a **cache unit**: consecutive slices are adjacent in the array, so a reader walking
nearby positions hits chunks it already holds, which an exact byte range never does. `indptr` is
chunked for the same reason rather than written whole — over 5.4 M positions it is a ~22 MB object
that two entries would otherwise pull entirely.

`write_store(..., byte_addressable=True)` writes one uncompressed chunk per array instead, so the
stored object *is* the raw little-endian buffer and `indptr` names an exact byte range. Fewer bytes
(376 against 131 072 for one 94-nonzero slice), the same number of round trips, no reuse — worth it
for a reader that makes one cold lookup and caches nothing. **Which was written is derived from the
codecs, never declared.**

---

## anndata interop

At **rank two** a layout is a genuine anndata sparse group and its arrays are byte-identical to
anndata's own — asserted in the test suite, not remembered. `read_elem(group["layouts"]["axis0"])`
returns the matrix.

Above rank two, `read_elem` on a child returns the **raveled two-axis view**, which is what the child
literally holds. A degradation, not a lie; the real shape is in the block.

`read_elem` on the **prefix root** does not work at any rank: the root is a container for layouts,
not itself a sparse group.

---

## Install

```bash
pip install sporadik              # numpy + zarr
pip install sporadik[scipy]       # + read_layout(), which returns a scipy.sparse matrix
pip install sporadik[obstore]     # + writing into an object store rather than a directory
pip install sporadik[complete]    # both of the above
```

## Use

```python
import scipy.sparse as sp
import sporadik

counts = sp.random(20_000, 1_200, density=0.01, format="csr")
sporadik.write_store("expression.zarr", [counts, counts.tocsc()])

with sporadik.open_store("expression.zarr", axis=1) as feature_major:
    positions, values = feature_major.slice_at(7)      # two reads, nothing else fetched
```

Above rank two, build a layout per axis and unravel what comes back:

```python
layout = sporadik.layout_over(shape, axis, data=..., indices=..., indptr=..., index_order=(0, 2))
with sporadik.open_store(path, axis=1) as reader:
    (a, c), values = reader.coords_at(7)               # through index_order, not axis order
```

### The convenience layer

Both of the above are the format itself. `sporadik.SparseArray` holds an array's layouts and builds
them from whatever the values came in as; `sporadik.open_array` opens a written store and keeps a
reader per axis. Nothing here is normative — it builds the same layouts by the same construction, and
a caller that already holds its own is unaffected.

```python
array = sporadik.SparseArray.from_matrix(counts)       # both layouts, from one matrix
array = sporadik.SparseArray.from_coords(shape, (cells, metabolites, adducts), intensity)
array.write("expression.zarr")                         # one layout per axis

with sporadik.open_array("expression.zarr") as store:
    positions, values = store.slice_at(7, axis=1)      # the axis is still named, never defaulted
```

### Reading many slices

One position at a time is the wrong unit for a caller with a list of them. `slices_at` reads a whole
batch in **two waves whatever its size** — one for the `indptr` brackets, one for every `indices` and
`data` run together — against two *per slice* otherwise.

```python
with sporadik.open_array("expression.zarr") as store:
    selection = store.slices_at(cell_ids, axis=0)      # two waves, whatever len(cell_ids) is
    selection = store.slices_over(10, 40, axis=0)      # contiguous: already one byte range
    batch = store.dense_slices(cell_ids, axis=0)       # dense over the uncompressed axes
```

Two rather than one is the floor, not an implementation limit: the second wave's byte ranges are
computed from the first wave's bytes, which is what `indptr` is for.

A `Selection` carries the positions along with the bytes, because **row `i` of a batch is
`positions[i]`, not `i`** — `selection.coords()` converts back to the original array's frame, and
`selection.as_layout()` is for a caller who means "these slices are my array now" and says so.

Order is the caller's, repeats are answered rather than refused, and an empty batch is an empty
selection. Only exactly adjacent byte ranges are merged here; merging across a *gap* is left to the
object store's own range-coalescing, which is measured, rather than guessed at a second time.

---

## Conformance

`sporadik.spec` is the normative half; everything else in the package is one implementation of it.
A second implementation — in another language, or in a server that must not depend on this package —
reproduces `spec` and nothing more.

That independence is deliberate. A reader that imports its writer inherits the writer's dependencies
and its release cycle, and a version skew between them becomes an outage rather than a refusal. Two
independent implementations of a written-down format is the only arrangement in which *"the format is
specified"* is a testable claim rather than a shared object file.

The block example above is parsed by `tests/test_spec_document.py` and asserted against
`sporadik.spec`, so this document cannot drift from the code it documents.
