Metadata-Version: 2.5
Name: signal-dataset
Version: 0.2.0
Summary: An open, profile-free format and Python library for large-scale multidimensional signal datasets.
Project-URL: Documentation, https://superpose-labs.github.io/signal-dataset/
Project-URL: Issues, https://github.com/superpose-labs/signal-dataset/issues
Project-URL: Repository, https://github.com/superpose-labs/signal-dataset
Author: Superpose Labs
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: <3.14,>=3.11
Requires-Dist: array-record<0.9,>=0.8.3
Requires-Dist: numpy<2.5,>=1.26
Requires-Dist: safetensors<0.9,>=0.5
Provides-Extra: gcs
Requires-Dist: google-cloud-storage<4,>=3; extra == 'gcs'
Description-Content-Type: text/markdown

# Signal Dataset

Immutable, indexed storage for multidimensional signal records on local filesystems and GCS.
Numerical fields use SafeTensors; ArrayRecord provides random access.

> **Status:** experimental 0.x software. The Python API may make documented breaking changes
> between minor releases. Persisted-format compatibility is versioned separately.

```python
import numpy as np
import signal_dataset as sds

record = sds.Record(
    id="capture-0042",
    fields={
        "iq": sds.Field(
            np.zeros((4, 4096), dtype=np.complex64),
            axes=(sds.Axis("channel", 4), sds.Axis("time", 4096)),
        )
    },
    metadata={"sample_rate_hz": 20_000_000},
)

shard = sds.write_shard([record], "captures.sds", work_id="worker-000")
dataset = sds.publish(
    "captures.sds",
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

assert dataset[0].id == "capture-0042"
assert dataset.record_metadata[0]["iq"].shape == (4, 4096)
```

`dataset.record_metadata[index]` reads the aligned metadata shard without fetching signal tensors.
Readers follow generation-pinned manifests and never list directories or GCS prefixes.

See the [quickstart](docs/quickstart.md), [annotation example](docs/how-to/annotations.md), and
[distributed-writing guide](docs/how-to/distributed-writing.md).

## Install

```bash
uv add signal-dataset
uv add 'signal-dataset[gcs]'  # GCS transport
```

Equivalent pip commands are `pip install signal-dataset` and
`pip install 'signal-dataset[gcs]'`.

Python 3.11–3.13 on Linux and macOS are supported. Windows is not currently supported because the
local atomic-publication implementation uses POSIX filesystem primitives.

## Design

- Immutable, root-last publication: readers observe no dataset or a complete snapshot.
- Lazy ordinal random access without directory or bucket-prefix discovery.
- Tensor-free aligned metadata reads.
- Storage and shard-container extension contracts.
- Profile-free records: no modality, training framework, or split policy is embedded in the core.

The project does not provide mutable datasets, distributed scheduling, authorization, retention,
or batching policy. See the [architecture](docs/architecture.md), [format compatibility
contract](docs/concepts/compatibility.md), and [roadmap](ROADMAP.md).

Signal Dataset is use-case agnostic. It has no training-framework, modality, or split-policy
dependency.

## Community

Read [CONTRIBUTING.md](CONTRIBUTING.md) before proposing changes. Use GitHub Issues for reproducible
bugs and design discussion, and follow [SECURITY.md](SECURITY.md) for private vulnerability reports.
Participation is governed by the [Code of Conduct](CODE_OF_CONDUCT.md).
