Metadata-Version: 2.4
Name: osfdata
Version: 1.1.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Dist: numpy>=1.20
Summary: Python bindings for the Open Streaming Format (OSF) — read, write, and inspect .osf and .osfz files.
Author: Optimeas GmbH
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://github.com/optimeas/osf/issues
Project-URL: Changelog, https://github.com/optimeas/osf/blob/main/implementations/python/CHANGELOG.md
Project-URL: Documentation, https://github.com/optimeas/osf/tree/main/docs
Project-URL: Homepage, https://github.com/optimeas/osf
Project-URL: Repository, https://github.com/optimeas/osf

# OSF — Python Implementation

![Status](https://img.shields.io/badge/status-in%20progress-orange.svg)
[![CI](https://github.com/optimeas/osf/actions/workflows/ci.yml/badge.svg)](https://github.com/optimeas/osf/actions/workflows/ci.yml)

## Target Platform

Data analytics, scientific computing, and AI/ML pipelines. The Python
bindings are the primary entry point for data exploration and the
foundation for ecosystem integrations (Arrow, PyTorch, TensorFlow,
LangChain).

The crate sits on top of the Rust [`osf-core`](../rust/osf-core/) library
via [PyO3](https://pyo3.rs) — see [DECISIONS §18](../../DECISIONS.md#18-rust-as-foundation-for-python).
One codebase, two audiences.

## Status

**In progress.** Reader, manager, and writer bindings are functional;
the Python wheel builds via [maturin](https://maturin.rs) with `abi3`
so a single artefact covers Python 3.9 / 3.10 / 3.11 / 3.12 / 3.13.

| Capability                                              | State                |
|---------------------------------------------------------|----------------------|
| `osf.load(path)` — read OSF or OSFZ                     | ✅                   |
| `osf.save(mgr, path)` — write OSF5                      | ✅                   |
| `DataManager.channel(name)` (DECISIONS §10)             | ✅                   |
| `DataManager.channels` / `channel_by_index`             | ✅                   |
| Channel `samples()` → NumPy array (numeric / GPS)       | ✅                   |
| Channel `samples()` → `list[str]` / `list[bytes]`       | ✅                   |
| Channel `timestamps_ns()` → NumPy `int64`               | ✅                   |
| Channel `segments` for equidistant channels             | ✅                   |
| `WriterBuilder` with chainable setters                  | ✅                   |
| Transparent OSFZ (gzip + zlib) on read                  | ✅                   |
| Type stubs (`*.pyi`) for IDE support                    | ✅                   |
| pandas `DataFrame` convenience                          | Pending (session 7b) |
| CI + wheel-build matrix + PyPI publishing               | ✅ (`pip install osfdata`) |

## Distribution name vs. import name

- **PyPI:** `pip install osfdata`
- **Python:** `import osf`

The split follows the established Python convention (scikit-learn
imports as `sklearn`, PyYAML imports as `yaml`, beautifulsoup4 imports
as `bs4`). The PyPI name `osf` is registered to an unrelated 2015
package; `osfdata` is the Optimeas distribution.

## Installation

### PyPI (recommended)

```bash
pip install osfdata
```

The PyPI distribution name is `osfdata` (the short name `osf` is taken by
an unrelated 2015 package); the Python import name is `osf` for brevity.
Wheels are published with `abi3` for Python 3.9–3.13 on Linux (x86_64 +
aarch64), macOS arm64, and Windows x64; other platforms build from the
sdist and need a local Rust toolchain.

### Development build (source checkout)

Build the native extension from a source checkout — the right path for
local Rust-side hacking or an unsupported platform.

```bash
git clone https://github.com/optimeas/osf
cd osf/implementations/python

# Create a virtual environment.
uv venv
.venv/Scripts/Activate.ps1            # Windows PowerShell
# source .venv/bin/activate           # Linux / macOS

# Install build + test tooling.
uv pip install maturin pytest

# Build the native extension and install it editably into the venv.
maturin develop --release

# Run the test suite.
pytest tests/
```

`maturin develop` produces an editable install — code changes to
`python/osf/*.py` and `python/osf/*.pyi` are picked up immediately;
Rust changes need another `maturin develop`.

For a complete walkthrough of the toolchain, build process, and
release pipeline, see [BUILD.md](BUILD.md) and
[RELEASE.md](RELEASE.md). They explain the PyO3 + maturin stack from
local setup through the Trusted-Publishing release to PyPI, intended
for developers new to Python's packaging conventions.

## Quick start

```python
import osf
import numpy as np

# Reader: convenience path
mgr = osf.load("examples/steam_loco.osf")
print(f"Channels: {len(mgr)}")
print(f"Compressed: {mgr.stats.compressed}")
print(mgr.stats)

# Channel access by name (DECISIONS §10)
ch = mgr.channel("GPS.PosFixMode")
print(f"{ch.name}: {ch.data_type}, {ch.sample_count} samples")

arr = ch.samples()           # NumPy float64 array
ts = ch.timestamps_ns()      # NumPy int64 array

# Equidistant segments are first-class
for seg in ch.segments:
    print(f"  start={seg.start_timestamp_ns} rate={seg.sample_rate_hz} n={seg.sample_count}")

# Writer: convenience path — round-trip an existing manager
osf.save(mgr, "out.osf")

# Writer: builder path — construct from scratch
b = osf.WriterBuilder().creator("my-app").tag("preview")
idx = b.add_channel(
    name="Sensor.Temp",
    data_type="double",
    channel_type="scalar",
    physical_unit="°C",
)
b.add_equidistant_segment(
    idx,
    start_ns=1_700_000_000_000_000_000,
    sample_rate_hz=1.0,
    values=np.array([18.4, 18.5, 18.6], dtype=np.float64),
)
b.write_to_file("synthetic.osf")
```

Both `osf.load()` and `osf.save()` always emit OSF5 (DECISIONS §6),
so an OSF4 source file becomes an OSF5 target after a round-trip.

## API surface

| Object                | Provides                                                                                      |
|-----------------------|-----------------------------------------------------------------------------------------------|
| `osf.load(path)`      | Open and parse an OSF or OSFZ file                                                            |
| `osf.save(mgr, path)` | Write a `DataManager` back as OSF5                                                            |
| `osf.DataManager`     | `channel(name)`, `channel_by_index(i)`, `channels`, `stats`, `len`                            |
| `osf.Channel`         | `index`, `name`, `data_type`, `channel_type`, `samples()`, `timestamps_ns()`, `segments`      |
| `osf.Segment`         | `start_timestamp_ns`, `sample_rate_hz`, `sample_count`                                        |
| `osf.ReaderStats`     | `compressed`, `compression_format`, channel/block counts, sizes, elapsed                      |
| `osf.WriterBuilder`   | Chainable file-info setters plus `add_channel`, `add_*_samples`, `write_to_file` (see stubs)  |
| `osf.OsfError`        | Single exception class for all reader / writer errors                                         |

NumPy is the data type for every numeric and `gpslocation` channel.
`gpslocation` arrives as a `(N, 3) float64` array with columns
`[latitude, longitude, altitude]`. `string` channels return
`list[str]`; `binary` channels return `list[bytes]`.

## Performance

`osf.load("examples/steam_loco.osf")` (123 channels, ~164 k samples)
measures **~3 ms** on a release-build extension on the dev box —
same order of magnitude as the underlying Rust read. Channel access
plus NumPy array conversion adds ~0.3 ms per channel. The clone
strategy (each `mgr.channel(name).samples()` call clones the
`Vec<T>` once) is fast enough that an `Arc<Channel>` optimisation is
unnecessary at this point.

## Spec revision tracked

OSF specification revision **2026-05-04** ([English](../../docs/en/osf_general.md),
[Deutsch](../../docs/de/osf_general.md)). All spec-level constraints
implemented in `osf-core` carry through automatically: removed
datatypes raise `OsfError`, deprecated channel-level fields produce a
log warning and are dropped, equidistant blocks limit to `float` /
`double`, and the writer never emits OSFZ.

## Dependencies

| Package          | Purpose                                            |
|------------------|----------------------------------------------------|
| `numpy>=1.20`    | Array data type for numeric channels               |

Build-time only:

| Crate              | Purpose                                          |
|--------------------|--------------------------------------------------|
| `pyo3 = "0.22"`    | Python C-API bindings (matched pair with numpy)  |
| `numpy = "0.22"`   | Rust → NumPy ndarray conversions                 |
| `osf-core` (path)  | The pure-Rust core library                       |

`pyo3` and `numpy` must agree on their major version per the
rust-numpy README; bumping one requires bumping the other.

## Next steps

1. **pandas `DataFrame` convenience** — build a DataFrame from a
   `DataManager` (one column per channel, optional time alignment).

CI builds the wheel matrix (Linux / macOS / Windows × Python 3.9–3.13)
and the release workflow publishes to PyPI automatically on a `v*` tag
(see [RELEASE.md](RELEASE.md)).

## Relationship to `python-osf`

`osfdata` is the modern successor to the existing
[python-osf](https://github.com/optimeas/python-osf) package. While
`python-osf` is a pure-Python implementation supporting OSF4 reading
only, `osfdata` provides:

- Full OSF4 and OSF5 support (read and write)
- Significantly higher performance via a Rust foundation
- Complete data type coverage including `binary`, `gpslocation`, and
  unsigned integers
- Compatibility with the current spec revision (2026-05-04)
- Transparent OSFZ decompression (zlib + gzip)

`python-osf` will be deprecated in favor of `osfdata` once feature
parity for all production use cases is verified.

## License

MIT. © 2026 Optimeas GmbH.

