Metadata-Version: 2.4
Name: datum-stream
Version: 0.10.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Requires-Dist: pyarrow>=14
Requires-Dist: cloudpickle==3.1.2
Requires-Dist: mypy==2.2.0 ; extra == 'dev'
Requires-Dist: pyright==1.1.411 ; extra == 'dev'
Requires-Dist: mypy==2.2.0 ; extra == 'test'
Requires-Dist: pyright==1.1.411 ; extra == 'test'
Provides-Extra: dev
Provides-Extra: test
Summary: Python bindings for Datum stream blueprints
License-Expression: Apache-2.0
Requires-Python: >=3.13
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# datum-stream

Python bindings for Datum stream blueprints.

Datum mirrors the `Source -> Flow -> Sink` and GraphDSL vocabulary from the Rust
crate while keeping Python execution explicit: building a pipeline creates an
immutable blueprint, and work starts only when a runnable graph is materialized
with a `Runtime`.

## Install

```sh
python -m pip install datum-stream
```

The package requires Python 3.13 or newer and depends on PyArrow. User Python
code always runs through Arrow UDF batches: integer-stream `map`, `filter`, and
`flat_map` wrap callables into single-column Arrow batches, while `map_batches`
is the vectorized `RecordBatch -> RecordBatch` tier.

For scalar integer hot paths, use the named kernels such as `map_add`,
`map_multiply`, and `filter_greater_than`. For Arrow batch streams, use typed
`col()` expressions for lowerable work and `map_batches` for arbitrary Python.
Terminal arithmetic is named too: `Sink.fold()` / `Sink.fold_sum()` sum values,
and `Sink.fold_product()` multiplies them. Graph partitions select the typed
`PartitionStrategy.MODULO` constant.

The package ships PEP 561 stubs (`py.typed`) with generic `Source`, `Flow`,
`Sink`, `RunnableGraph`, `Inlet`, and `Outlet` types. Public construction is
always strict: graph wiring and declared Arrow schemas are validated during
builder calls, and empty Arrow inputs require an explicit `schema=...`.

## Example

```python
import datum

with datum.Runtime() as runtime:
    graph = (
        datum.Source.range(1, 5)
        .map_add(1)
        .to_mat(datum.Sink.fold_product())
    )
    completion = graph.run(runtime)
    assert completion.wait() == 120
```

## Scope

The current Python surface covers integer linear streams, a focused GraphDSL
surface (`Broadcast`, `Balance`, `Merge`, `Partition`, `Zip`, `Concat`, and
`Interleave`), Arrow batch UDFs, and Datum Connect for trusted client/server
execution. Connect can run linear plans, supported junction graphs, tuple
FlowShape results, direct ZipShape runs, and Merge/Concat/Interleave FanInShape
runs. Local Connect defaults to `arrow-ipc`; remote Arrow payloads should prefer
`arrow-ipc-zstd` when compression is wanted. Those wire-format defaults were
chosen by measurement.

For the broader guide, see the Datum Python docs in
[`docs/guides/python`](../../docs/guides/python/index.md).

