Metadata-Version: 2.4
Name: dagster-dataframely
Version: 0.12.0
Summary: Attach a Dataframely schema to a Dagster asset.
Keywords: dagster,data,data-quality,dataframely,polars,validation
Author: Ozan Ozbeker
Author-email: Ozan Ozbeker <github@ozanozbeker.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Dist: dagster>=1.13.6
Requires-Dist: dataframely>=3.0.0
Requires-Dist: polars>=1.44.2
Requires-Dist: universal-pathlib>=0.2.2
Maintainer: Ozan Ozbeker
Maintainer-email: Ozan Ozbeker <github@ozanozbeker.com>
Requires-Python: >=3.12
Project-URL: Changelog, https://ozanozbeker.com/dagster-dataframely/changelog.html
Project-URL: Homepage, https://ozanozbeker.com/dagster-dataframely/
Project-URL: Issues, https://github.com/ozanozbeker/dagster-dataframely/issues
Project-URL: Repository, https://github.com/ozanozbeker/dagster-dataframely
Description-Content-Type: text/markdown

# dagster-dataframely

[![PyPI](https://img.shields.io/pypi/v/dagster-dataframely)](https://pypi.org/project/dagster-dataframely/) [![CI](https://github.com/ozanozbeker/dagster-dataframely/actions/workflows/ci.yml/badge.svg)](https://github.com/ozanozbeker/dagster-dataframely/actions/workflows/ci.yml)

[Dataframely](https://github.com/Quantco/dataframely) validates [Polars](https://pola.rs) frames against a schema.
[Dagster](https://dagster.io) has two built-in places to show a schema: the Columns tab and asset checks.
`dagster-dataframely` attaches a Dataframely schema to a Dagster asset, so you declare a table once and Dagster shows it in both places.

<!-- snippet: demo/src/dagster_dataframely_demo/defs/silver/customers.py -->
```python
import dataframely as dy
import polars as pl

import dagster_dataframely as dd


class CustomersSchema(dy.Schema):
    """A storefront customer account."""

    customer_id = dy.String(primary_key=True, description="Account identifier.")
    email = dy.String(nullable=False, description="Address for receipts.")
    lifetime_value = dy.Float64(nullable=False, min=0.0, description="Spend to date.")


@dd.asset(CustomersSchema)
def customers(raw_customers: pl.DataFrame) -> pl.DataFrame:
    """Validate the storefront's customer accounts."""
    return raw_customers
```

That is the whole integration.
From one declaration you get the catalog's Columns tab, one asset check per rule, and a code version derived from the schema.
A row that fails an `"error"` rule fails the run.
With `filter=True`, the run writes that row to a separate table instead and writes the other rows to the asset's table.
[Get started](https://ozanozbeker.com/dagster-dataframely/user-guide/dagster-dataframely.html) covers each of those, with screenshots of the demo pipeline.

## Installation

```bash
uv add dagster-dataframely
```

You need Python 3.12 or newer.
The dependencies are `dagster`, `dataframely` and `polars`, plus `universal-pathlib`, which `dagster` already installs.

This package includes an IO manager only for its examples, `dd.data.ParquetIOManager`.
In your own project, bind a library one.
[`dagster-polars`](https://docs.dagster.io/integrations/libraries/polars) writes Polars frames to a filesystem or object store, and [`dagster-duckdb-polars`](https://docs.dagster.io/integrations/libraries/duckdb) writes them to a warehouse.
Any IO manager that stores by asset key works.

> **Pre-1.0.**
> A characterization test covers the public surface, so it never changes by accident.
> It can still change: a `0.x` minor release can include breaking changes.
> [Upgrading](https://ozanozbeker.com/dagster-dataframely/user-guide/upgrading.html) has the migration for each one.

## Documentation

[**https://ozanozbeker.com/dagster-dataframely**](https://ozanozbeker.com/dagster-dataframely/) has the guide, the reference and the changelog.
Contributor documentation stays in the repository: [`CONTEXT.md`](https://github.com/ozanozbeker/dagster-dataframely/blob/main/CONTEXT.md) is the glossary, and [`docs/pre-1.0.md`](https://github.com/ozanozbeker/dagster-dataframely/blob/main/docs/pre-1.0.md) records the decisions and the measurements the package was built on.
