# Turbine

**Contract-driven data quality for data products.**

[![PyPI](https://img.shields.io/pypi/v/turbine-data.svg)](https://pypi.org/project/turbine-data/)
[![Python](https://img.shields.io/pypi/pyversions/turbine-data.svg)](https://pypi.org/project/turbine-data/)
[![ODCS](https://img.shields.io/badge/ODCS-v3.1.0-green.svg)](https://github.com/bitol-io/open-data-contract-standard)

Turbine turns a YAML data contract into running quality checks. You declare a table's schema, ownership, freshness expectations, and validity rules in one file; Turbine validates the YAML offline, checks the live database matches it, runs every quality check against the data, scores the result, and exposes everything over a REST API and dashboard.

It uses [ODCS v3.1.0](https://github.com/bitol-io/open-data-contract-standard), so contracts round-trip with the rest of the data ecosystem.

## Example

A contract is a single YAML file:

```yaml
kind: DataContract
apiVersion: v3.1.0
id: orders
domain: sales
version: "1.0.0"
status: active

slaProperties:
  - property: latency
    element: public.orders.created_at
    value: 24
    unit: hour

schema:
  - name: public.orders
    properties:
      - name: order_id
        logicalType: integer
        required: true
        primaryKey: true
      - name: amount
        logicalType: number
        required: true
      - name: status
        logicalType: string
        required: true
      - name: created_at
        logicalType: timestamp
        required: true
```

Run every check on it:

```bash
turbine check --datasource default orders.yml
```

You get a per-check verdict, a quality score per dimension (completeness, accuracy, consistency, timeliness, validity), and the failing rows persisted for follow-up.

## Installation

```bash
uv add "turbine-data[duckdb]"        # local files, zero credentials
uv add "turbine-data[postgres]"      # PostgreSQL
uv add "turbine-data[snowflake]"     # Snowflake
uv add "turbine-data[all]"           # every driver + dashboard
```

Requires Python 3.12 or newer. The PyPI package is `turbine-data`; the CLI is `turbine`.

## Quick start

```bash
# 1. Scaffold a project
uv run turbine init --database duckdb

# 2. Configure credentials
cp .env.example .env

# 3. Validate and run
uv run turbine lint  src/<project>/contracts/example.yml
uv run turbine check src/<project>/contracts/example.yml --datasource default
```

`turbine init --demo` scaffolds a fully populated example project with real data and multiple contracts.

## Features

- **YAML contracts** in ODCS v3.1.0 — schema, ownership, SLAs, quality checks in one file
- **Quite a few check types** — missing, duplicate, invalid values, freshness, row count, custom SQL, Python, group, and window checks (z-score, spike, flatline)
- **Schema drift detection** — compare your contract to the live database before running a single check
- **Dimension-aware scoring** — every check is weighted by its quality dimension (completeness, accuracy, consistency, timeliness, validity)
- **Row-level flagging** — failing rows are persisted in a per-cell bitmap matrix; query which rows failed which checks across runs
- **Management API + dashboard** — `turbine serve` exposes runs, results, scores, and flagged rows over HTTP
- **Code generation** — scaffold SQLModel models and FastAPI routers from contracts
- **IDE support** — full LSP with VS Code and soon JetBrains extensions

## Management API

```bash
turbine serve --datasource default --port 8000
```

Endpoints under `/api/v1/manage/`: `/contracts`, `/checks/run`, `/runs/{id}`, `/runs/{id}/results`, `/flagged-rows/{table}`. Browse `/api/v1/manage/docs` for the interactive OpenAPI page.

```bash
turbine dashboard --port 5173
```

Renders the same data as charts, run history, and a flagged-rows explorer.

## CLI

```bash
turbine lint     <contract.yml>                     # validate the YAML offline
turbine validate <contract.yml> --datasource <name> # compare to live database schema
turbine check    <contract.yml> --datasource <name> # lint + validate + run every check
turbine status                                      # project health, flagged-row counts
turbine bump                                        # update contract versions
turbine new      contract|datasource|check          # scaffold a new resource
turbine generate                                    # SQLModel + FastAPI from contracts
```

## Ecosystem

**Orchestrators.** Run Turbine Check Runs as native steps in your workflow tool:

- [`dagster-turbine`](integrations/dagster-turbine/) — Check Definitions become Dagster `AssetCheckSpec`s. Partitioned assets scope each Check Run to their partition window.
- [`airflow-turbine`](integrations/airflow-turbine/) — `TurbineOperator` with a deferred trigger; tasks wait on a Check Run without blocking a worker.
- [`turbine-client`](integrations/turbine-client/) — sync + async Python client. Use directly when you need glue beyond the integrations above.

**Editors.**

- [VS Code extension](editors/vscode/) — diagnostics, autocomplete, quick fixes, run-from-editor. Install **Turbine** from the Marketplace.
- [JetBrains plugin](editors/jetbrains/) — same surface for IntelliJ, PyCharm, DataGrip.

## Documentation

Full documentation lives under [`docs/user/docs/`](docs/user/docs/):

- [Getting started](docs/user/docs/getting-started/index.md)
- [Concepts](docs/user/docs/concepts/index.md) — contracts, Check Definition, Check Run, Run Result, scoring
- [Guides](docs/user/docs/guides/index.md) — incremental mode, row flagging, CI workflows
- [CLI reference](docs/user/docs/reference/cli.md)

## Contributing

See [`docs/contributing/`](docs/contributing/) for dev setup, test layout, and code conventions.
