Metadata-Version: 2.5
Name: dataproduct-sdk
Version: 0.1.0rc6
Summary: Turn an ODCS data contract plus pure transforms into a running, quality-gated, multi-target data product
Author-email: Leandro Kellermann de Oliveira <lkellermann@leandroasaservice.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Keywords: data-contract,data-mesh,data-product,databricks,duckdb,ibis,odcs,write-audit-publish
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Requires-Python: >=3.12
Requires-Dist: datacontract-cli[duckdb]>=1.1.3
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: api
Requires-Dist: fastapi>=0.139.2; extra == 'api'
Requires-Dist: uvicorn>=0.51.0; extra == 'api'
Provides-Extra: databricks
Requires-Dist: datacontract-cli[databricks]>=1.1.3; extra == 'databricks'
Provides-Extra: delta
Requires-Dist: deltalake>=0.9.0; extra == 'delta'
Description-Content-Type: text/markdown

# Data Product SDK

[![CI](https://github.com/leandroasaservice/dataproduct-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/leandroasaservice/dataproduct-sdk/actions/workflows/ci.yml)
[![Security](https://github.com/leandroasaservice/dataproduct-sdk/actions/workflows/security.yml/badge.svg)](https://github.com/leandroasaservice/dataproduct-sdk/actions/workflows/security.yml)
[![codecov](https://codecov.io/gh/leandroasaservice/dataproduct-sdk/graph/badge.svg?token=2066Z0UU64)](https://codecov.io/gh/leandroasaservice/dataproduct-sdk)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Dependabot](https://img.shields.io/badge/Dependabot-enabled-blue?logo=dependabot&logoColor=white)](https://docs.github.com/en/code-security/dependabot)
[![License: AGPL-3.0-only](https://img.shields.io/badge/license-AGPL--3.0--only-blue.svg)](https://www.gnu.org/licenses/agpl-3.0.html)

Turn an [ODCS](https://bitol-io.github.io/open-data-contract-standard/) data contract plus a set
of pure transforms into a running, quality-gated, multi-target data product.

## Install

Requires Python 3.12+.

```bash
pip install dataproduct-sdk                 # DuckDB target (default)
pip install "dataproduct-sdk[databricks]"   # + the Databricks/Spark cloud target
pip install "dataproduct-sdk[api]"          # + the `api` port shape (`dataproduct serve` is not implemented for the DuckDB folder layout yet)
```

While the newest release is a release candidate, ask for it explicitly: `pip install --pre
dataproduct-sdk`, or with [uv](https://docs.astral.sh/uv), `uv tool install dataproduct-sdk
--prerelease=allow` (which puts `dataproduct` on your PATH globally).

## Start here

Build your first data product on your laptop, on DuckDB — no account, no credentials, no cost.

```bash
uv tool install dataproduct-sdk --prerelease=allow   # puts `dataproduct` on your PATH
dataproduct doctor                                   # is this machine ready? it names the fix if not
dataproduct init signups                             # scaffold a product
cd signups && uv sync
dataproduct run sample                               # publish sample rows through the quality gate
```

Then follow the path. The guides ship inside the package — `dataproduct learn <topic>` renders one
in your terminal — and each ends where the next begins:

| Step | `dataproduct learn …` | What you get |
|---|---|---|
| 1 | `concepts` | the five ideas everything follows from — 15 minutes, optional first read |
| 2 | `duckdb` | the quickstart: explain, run, inspect, audit and check a product on sample rows |
| 3 | `your-own-data` | your data in it: your data published through the gate, the sample gone, every check green — locally and in CI |

Going deeper, when you want it: `quality-gate` (shaping the gate), `troubleshooting`
(the failures people hit), and `from-scratch` (how it works underneath,
finished in `examples/duckdb-bikeshare`). Another target: `databricks`
(the same code on a free cloud workspace).

`--prerelease=allow` is needed only while the newest release is a release candidate. To try the SDK
without installing it: `uvx --prerelease=allow --from dataproduct-sdk dataproduct learn`. Inside a
product, `task ...` or `uv run dataproduct ...` runs the SDK version that product pins; plain
`dataproduct ...` runs the one on your PATH.

## The commands you will actually use

Every command runs from your product's root. On DuckDB a command works on one environment — which
data each input reads, and where the run publishes, both declared in the contract. It is `dev` unless
you say otherwise: the scaffold writes `DATAPRODUCT_ENVIRONMENT=dev` to the product's `.env`, and
`--environment` overrides it. DuckDB reads no config file; `--config` is for Databricks only.

**Every day**

| Command | What it does |
|---|---|
| `dataproduct explain` | what a run *would* publish, gate and quarantine — reads nothing, writes nothing, needs no warehouse. The best first command after any contract change. |
| `dataproduct run` | build every output port, audit it against the contract, publish only what passes. Bad rows are quarantined; a mass failure holds the whole run. |
| `dataproduct run ASSET` | run your deploy code, `deploy/duckdb/<ASSET>.py` — `dataproduct run sample` lands the scaffold's sample data, then publishes. `explain`, `audit`, `serve` and `doctor` take the same names to check only what that code produces (`serve` is not implemented for the DuckDB folder layout yet). |
| `dataproduct run --dry-run` | build and audit everything, report the counts, write **nothing** — the circuit breaker still bites. |
| `dataproduct check` | the fitness-function battery — transform purity, module boundaries, WAP coverage, contract lint, schema conformance, placement drift. Exit non-zero on any failure. |
| `dataproduct audit` | the authoritative *external* audit — `datacontract test` against each published port. |
| `dataproduct doctor` | environment preflight: toolchain, runtime, optional extras, and this product's shape. Names the fix for anything missing. |

**When something looks wrong**

| Command | What it does |
|---|---|
| `dataproduct quarantine inspect` | which rows were rejected, by which rule, with the original payload. Producer-only — never a consumer surface. |
| `dataproduct observability dashboard` | a static HTML dashboard from the SLI store (rows in/published/quarantined per port per run). |
| `dataproduct check --only FF-3 FF-5` | run just the checks you are chasing (`FF-3,FF-5` works too). |

**Authoring**

| Command | What it does |
|---|---|
| `dataproduct init [DIR]` | scaffold a product that is green on its first `check`. Interactive, or `--answers answers.yaml`. |
| `dataproduct contract draft PORT` | print a starter ODCS schema object for a port — column names and types derived from your transform, every constraint left as a `TODO(owner)`. |
| `dataproduct docs` | render the contract catalog + per-contract HTML, including the product ERD. |
| `dataproduct evolve plan` | derive the migration plan for a contract change. |
| `dataproduct evolve check` | fail a build whose contract change ships no matching, up-to-date plan. |

**Serving and deploying**

| Command | What it does |
|---|---|
| `dataproduct serve` | not implemented for the DuckDB folder layout yet: lists every `type: api` route it would serve and exits 1 (needs the `api` extra). |
| `dataproduct generate ddl` (also `bundle`, `job`, `all`) | re-derive the committed Databricks deploy artifacts from the contract. |

`dataproduct --help` lists everything; `dataproduct <command> --help` details one. `explain` and
`run --dry-run` take `--format json` for machine-readable output. `python -m dataproduct.checks` is
an equivalent entry point for `check` when the console script is not on PATH.

If you scaffolded with `dataproduct init`, every command above also has a `task` wrapper —
`task --list` shows them, plus `task tour` and `task demo:break`.

## What a product looks like

```python
# product.py
from dataproduct import Product
import transforms as T

product = Product(name="orders", domain="sales")


@product.table("orders")
def orders(src):
    return T.build_orders(src.read("orders_raw"))


@product.view("public_orders", reads=["orders"])
def public_orders(orders):
    return T.build_public_orders(orders)


product.enable_observability()
```

You write three things: **transformation logic** (pure `ibis` expression-to-expression functions),
**declared constraints** (an ODCS contract), and **topology** (which output ports exist and what
feeds them). The SDK supplies everything else.

`src.read("orders_raw")` names an *input*. On DuckDB the contract declares it in its `openlineage`
custom property — one OpenLineage dataset per environment, optionally bound to the upstream contract
it must conform to — so a product's inputs and outputs live in one reviewed file.

## What the SDK does for you

| | |
|---|---|
| **Declared inputs** | A product's inputs are OpenLineage datasets declared in its own contract, one per environment (`dev`, `prod`, …), each optionally bound to the upstream contract it must conform to — the check fails naming the upstream when it drifts. The environment comes from `--environment` or a gitignored `.env`, never from the product's committed files. |
| **Contract-derived quality** | Row and dataset rules are projected from the ODCS contract — no second place to keep them in sync. Each rule maps to an action: warn, quarantine the row, or hard-fail the dataset. |
| **Write-Audit-Publish** | Every port is staged, audited, and only then published — by shape. Tables replace their published data in one atomic Delta swap on Databricks, and merge into it on the contract's primary key on DuckDB; views are defined only after the audit passes, APIs get a black-box audit. A failed gate leaves the last good data serving. |
| **Forensic quarantine** | Rejected rows are kept with the rule that rejected them, and completeness is accounted for so nothing silently disappears. |
| **Portability** | Core and transforms are written against the `ibis` expression API only, so the identical pipeline and the identical gate run on DuckDB and on Databricks. Enforced, not just intended. |
| **Fitness functions** | `dataproduct check` runs a battery of architectural checks — transform purity, module boundaries, WAP coverage, contract lint, schema conformance, placement drift. |
| **Generated tests** | A pytest plugin synthesizes one gate-bite test *per contract rule*, so your test suite cannot drift from your contract. |
| **Schema evolution** | `dataproduct evolve` plans a contract change and gates on plan freshness. |
| **Offline introspection** | `dataproduct explain` projects what a run would do without touching a warehouse; `dataproduct run --dry-run` provably writes nothing. |

## Extending it

Targets are resolved through the `dataproduct.adapters` entry-point group, so a new backend is a
separate distribution — no fork and no change to this package. The core depends only on the
`Protocol`s in `dataproduct.ports`.

## License and support

[AGPL-3.0-only](https://www.gnu.org/licenses/agpl-3.0.html) — copyright (c) 2026 Leandro
Kellermann de Oliveira. See `LICENSE` for the additional permission covering scaffolded/generated
files. For a proprietary license, OEM bundling, or support:
**lkellermann@leandroasaservice.com**
