Metadata-Version: 2.4
Name: turbine-fast
Version: 0.2.1a4
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Distributed Computing
Requires-Dist: pyarrow>=24
Requires-Dist: pydantic>=2
Requires-Dist: python-dotenv>=1.0
Requires-Dist: fastapi>=0.141 ; extra == 'fastapi'
Requires-Dist: uvicorn[standard]>=0.54 ; extra == 'fastapi'
Provides-Extra: fastapi
License-File: LICENSE
Summary: High-performance stateful stream processor with a Rust core and a Python SDK.
Keywords: kafka,streaming,stream-processing,stateful,arrow,rust,free-threading
Author-email: Léo Debret <leo@turbine.fast>
License-Expression: Apache-2.0
Requires-Python: >=3.14
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://docs.turbine.fast/

# Turbine

![Turbine](https://docs.turbine.fast/turbine.png)

[![PyPI](https://img.shields.io/pypi/v/turbine-fast?label=PyPI)](https://pypi.org/project/turbine-fast/)
[![Python](https://img.shields.io/pypi/pyversions/turbine-fast)](https://docs.turbine.fast/installation/)
[![License](https://img.shields.io/pypi/l/turbine-fast)](https://www.apache.org/licenses/LICENSE-2.0)

A high-performance stateful stream processor with a Rust core and a Python interface.

Kafka I/O, JSON/Avro serialization, windowing, state (RocksDB) and checkpointing run in
Rust. Your Python handler is called once per **Apache Arrow `RecordBatch`** — never per
message — so the ergonomics stay Python and the hot path never does.

**[Documentation](https://docs.turbine.fast/)** ·
[Installation](https://docs.turbine.fast/installation/) ·
[Quick start](https://docs.turbine.fast/quick-start/) ·
[Windowing](https://docs.turbine.fast/windowing/) ·
[Console & API](https://docs.turbine.fast/api/)

## Install

```bash
uv init my-pipeline && cd my-pipeline
uv python pin 3.14t                        # free-threaded CPython is mandatory
uv add --prerelease=allow turbine-fast
```

The distribution is **`turbine-fast`**; the import name is **`turbine`**.

> **Free-threaded Python 3.14 is required.** Turbine runs one partition per thread in a
> single process, which only scales without the GIL — so the published wheels are
> `cp314t` **exclusively** (Linux x86_64, `manylinux_2_28`). On a standard GIL
> interpreter there is no compatible wheel and no sdist fallback: the install fails
> rather than silently giving you a version that cannot scale. Other platforms build
> from source. Full details, including the source build, are in
> [Installation](https://docs.turbine.fast/installation/).

`--prerelease=allow` is required while Turbine is published as alphas (`0.2.0aN`).

## A first app

```python
from turbine import KafkaBroker, RecordBatch, Turbine, TurbineState

kafka = KafkaBroker(bootstrap="localhost:9092")
app = Turbine(kafka)

@app.subscribe(kafka.topic("events"), output=kafka.topic("counts"))
def aggregate(batch: RecordBatch, state: TurbineState) -> RecordBatch:
    total = state.get("total") or 0
    total += batch.num_rows
    state.put("total", total)
    return batch

app.run()
```

Point it at any Kafka-compatible broker (Kafka or Redpanda) and run it like any Python
script. State is checkpointed automatically and survives restarts; the node serves its
REST API and the web console on port 8400 with nothing else to deploy.

## What you get

- **Rust on the hot path** — broker I/O, serde, state and checkpointing. Python sees
  columnar Arrow batches, so per-message overhead never reaches it. Throughput grows
  roughly linearly as you add partitions; the absolute numbers depend on your handler.
- **Stateful processing with checkpoints** — one RocksDB per partition, snapshotted to
  a local directory or S3-compatible object storage. A restart restores the snapshot
  and resumes at the right offset.
- **Event-time windowing** — tumbling, sliding and session windows, persistent or
  in-memory, driven by a data-derived watermark, with late-data policies and mergeable
  accumulators (sum/mean/min/max, HLL, t-digest, …).
- **Delivery guarantees you choose per subscription** — at-least-once by default,
  opt-in `exactly_once` built on Kafka transactions, with dead-letter routing for bad
  payloads and failing handlers.
- **Models in the pipeline** — declare `infer=ml.OnnxModel(...)` on a subscription and
  every batch it publishes is scored in Rust, in the processing thread: no model server,
  no network hop. Train in PyTorch, scikit-learn, XGBoost or LightGBM, export to ONNX.
- **High availability** — embedded Raft replicates partition assignments; a node that
  dies has its partitions redistributed, and rejoins without manual intervention.
- **Observability that ships with the package** — `/health`, `/ready`, `/metrics`
  (Prometheus), a state-browsing REST API, and a web console served by every node at
  `/console`. Nothing extra to install or deploy.

## Status

**Alpha.** The API is stabilising and may still change between alpha releases; state
and internal topic formats can change in a way that requires dropping state and
replaying. Published wheels: Linux x86_64, free-threaded CPython 3.14 only — macOS,
Windows and arm builds are not published yet and need a source build.

## Positioning

Turbine aims to bring Kafka-Streams-level stateful stream processing to the Python
ecosystem, with performance that matches or exceeds JVM-based solutions.

- **Kafka Streams** — powerful, but JVM-only. Turbine targets similar semantics with a
  Python-native experience.
- **Faust** — Python stream processing, but slow, unmaintained and thin on tooling.
- **FastStream** and friends — better for per-message routing, request/reply and
  multi-broker work. Turbine is for high-volume aggregation, stateful pipelines and
  cluster-level HA. The trade-off is laid out in the
  [overview](https://docs.turbine.fast/).

Python dominates the AI/ML landscape, and the people who need stream processing wired
into inference pipelines, feature stores and real-time applications are already there.
Turbine gives them a fast, stateful stream processor without leaving their ecosystem.

## License

Licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). The full text ships in the distribution as `LICENSE`.

