Metadata-Version: 2.4
Name: statguardian
Version: 2.7.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: polars>=1.6,<1.32.3
Requires-Dist: pyarrow>=16.0
Requires-Dist: python-frontmatter>=1.0.0,<1.2.0
Requires-Dist: apache-airflow>=2.0 ; extra == 'airflow'
Requires-Dist: pandas==2.1.0 ; extra == 'all'
Requires-Dist: connectorx>=0.3 ; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9 ; extra == 'all'
Requires-Dist: pymysql>=1.0 ; extra == 'all'
Requires-Dist: sqlalchemy==2.0.54 ; extra == 'all'
Requires-Dist: pyyaml>=6.0 ; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.0 ; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.0 ; extra == 'all'
Requires-Dist: amazon-redshift-python-driver>=1.0 ; extra == 'all'
Requires-Dist: databricks-sql-connector>=1.0 ; extra == 'all'
Requires-Dist: clickhouse-driver>=0.2 ; extra == 'all'
Requires-Dist: duckdb>=0.8 ; extra == 'all'
Requires-Dist: trino>=0.20 ; extra == 'all'
Requires-Dist: pyspark>=3.0 ; extra == 'all'
Requires-Dist: polars[aws,gcp,azure]>=1.6,<1.32.3 ; extra == 'all'
Requires-Dist: polars[aws,gcp,azure]>=1.6,<1.32.3 ; extra == 'cloud'
Requires-Dist: pyyaml>=6.0 ; extra == 'dbt'
Requires-Dist: sqlalchemy==2.0.54 ; extra == 'dbt'
Requires-Dist: connectorx>=0.3 ; extra == 'dbt'
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
Requires-Dist: pyflink>=1.17 ; extra == 'flink'
Requires-Dist: confluent-kafka>=2.0 ; extra == 'kafka'
Requires-Dist: pandas==2.1.0 ; extra == 'pandas'
Requires-Dist: pyspark>=3.0 ; extra == 'spark'
Requires-Dist: connectorx>=0.3 ; extra == 'sql'
Requires-Dist: psycopg2-binary>=2.9 ; extra == 'sql'
Requires-Dist: pymysql>=1.0 ; extra == 'sql'
Requires-Dist: sqlalchemy==2.0.54 ; extra == 'sql'
Requires-Dist: google-cloud-bigquery>=3.0 ; extra == 'sql-bigquery'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-bigquery'
Requires-Dist: clickhouse-driver>=0.2 ; extra == 'sql-clickhouse'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-clickhouse'
Requires-Dist: databricks-sql-connector>=1.0 ; extra == 'sql-databricks'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-databricks'
Requires-Dist: duckdb>=0.8 ; extra == 'sql-duckdb'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-duckdb'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-mysql'
Requires-Dist: pymysql>=1.0 ; extra == 'sql-mysql'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-postgres'
Requires-Dist: psycopg2-binary>=2.9 ; extra == 'sql-postgres'
Requires-Dist: amazon-redshift-python-driver>=1.0 ; extra == 'sql-redshift'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-redshift'
Requires-Dist: snowflake-connector-python>=3.0 ; extra == 'sql-snowflake'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-snowflake'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-sqlite'
Requires-Dist: trino>=0.20 ; extra == 'sql-trino'
Requires-Dist: connectorx>=0.3 ; extra == 'sql-trino'
Provides-Extra: airflow
Provides-Extra: all
Provides-Extra: cloud
Provides-Extra: dbt
Provides-Extra: dev
Provides-Extra: flink
Provides-Extra: kafka
Provides-Extra: pandas
Provides-Extra: spark
Provides-Extra: sql
Provides-Extra: sql-bigquery
Provides-Extra: sql-clickhouse
Provides-Extra: sql-databricks
Provides-Extra: sql-duckdb
Provides-Extra: sql-mysql
Provides-Extra: sql-postgres
Provides-Extra: sql-redshift
Provides-Extra: sql-snowflake
Provides-Extra: sql-sqlite
Provides-Extra: sql-trino
License-File: LICENSE
Summary: Rust-native data quality engine with a declarative contract DSL: schema validation, drift detection, and anomaly detection for Pandas and Polars.
Keywords: data-quality,data-validation,schema-validation,data-testing,data-contracts,pandas,polars,duckdb,data-pipeline,etl,data-engineering,data-governance,drift-detection,anomaly-detection,outlier-detection,automated-testing,data-integrity,data-profiling,statistical-analysis,python-library,machine-learning,mlops,data-ops,data-observability
Author-email: Georgi Mammen Mullassery <mullassery@gmail.com>
Maintainer-email: Georgi Mammen Mullassery <mullassery@gmail.com>
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://github.com/Mullassery/statguardian/issues
Project-URL: Changelog, https://github.com/Mullassery/statguardian/releases
Project-URL: Discussions, https://github.com/Mullassery/statguardian/discussions
Project-URL: Documentation, https://github.com/Mullassery/statguardian#readme
Project-URL: Homepage, https://github.com/Mullassery/statguardian
Project-URL: Repository, https://github.com/Mullassery/statguardian
Project-URL: Source Code, https://github.com/Mullassery/statguardian/tree/main

# StatGuardian

A Rust-native data quality engine with a declarative contract DSL for schema validation, drift detection, and anomaly detection for Pandas and Polars DataFrames.

## What it is

StatGuardian validates data at runtime against versionable contracts written in a declarative `.sg` DSL. It catches schema violations, statistical drift, and anomalies before they reach downstream consumers. The core validation and statistics logic run in Rust while providing a Python API for integration into data pipelines.

## Features / Highlights

- **Declarative contracts**: Define schema, quality rules, statistical drift thresholds, and anomaly checks in one `.sg` file.
- **Rust-native execution**: Core checks and statistics execute in compiled Rust for performance.
- **Multi-backend support**: Validate Pandas and Polars DataFrames with the same contract. Delta Lake and Apache Iceberg tables are supported as inputs.
- **Drift detection**: Detect distribution shifts between datasets using PSI and KS tests via the DSL.
- **Anomaly detection**: Identify outliers and duplicates with built-in rules.
- **Blocking controls**: Severity tiers let you mark critical checks as `@blocking` to gate pipeline execution.
- **Detailed reports**: Structured violation messages, scores, and per-check results.

## Who it's for / Use cases

- **Data engineering pipelines**: Enforce contracts on batch or streaming inputs before writes.
- **MLOps/feature pipelines**: Catch schema or distribution drift between training and inference data.
- **ETL/data integration**: Validate external data against agreed schemas and quality thresholds.
- **Data governance**: Version and review contracts alongside code as the source of truth.

## What it is not

- **Not a full observability platform**: Focused on contract-based validation; does not include dashboards/alerting infrastructure beyond returning structured reports.
- **Not a generic testing framework**: Designed for data contract enforcement at runtime and in pipelines.
- **Benchmarks vary**: Performance depends on workload (row count, columns, checks). Run benchmarks against your own data before making comparisons.

## Project status

StatGuardian is in active development. Core features (schema validation, quality rules, drift, anomaly detection, multi-backend DataFrame support) are implemented. Streaming validation (StreamGuard) and additional integrations are documented under development paths; see [docs/ROADMAP_HONEST.md](docs/ROADMAP_HONEST.md) for known limitations and planned work.

## Installation

### From PyPI

```bash
pip install statguardian
```

### From source (Python + Rust)

```bash
pip install maturin
maturin develop
```

### System requirements

- Python 3.8+
- Rust toolchain (for source builds)

## Quick start

```python
import polars as pl
import statguardian

contract = statguardian.DataContract.from_dsl("""
dataset orders {
    schema {
        order_id: string, not_null, unique
        amount:   float,  positive
        status:   string, not_null, enum=["pending","paid","cancelled"]
    }
    quality {
        completeness(order_id) > 0.999
    }
}
""")

df = pl.read_parquet("orders.parquet")
report = statguardian.execute(contract, df)
print(report.summary())
print(f"Passed: {report.passed}")
```

## Usage

Validate a Polars DataFrame against a contract:

```python
import statguardian

contract = statguardian.DataContract.from_dsl("dataset orders { schema { id: string, not_null } }")
report = statguardian.execute(contract, df)
```

Drift detection between reference and incoming data:

```python
report = statguardian.execute(contract, incoming_df, reference=baseline_df)
for d in report.drift_results():
    if not d["passed"]:
        print(f"Drift: {d['column']} PSI={d.get('psi', 0):.4f}")
```

Load a contract from a `.sg` file:

```python
contract = statguardian.DataContract.from_file("contracts/orders.sg")
```

Note: When validating a Pandas DataFrame, convert it to Polars first (e.g., `pl.from_pandas(df)`) before calling `execute()`.

## Documentation

- [CLI documentation](docs/CLI.md)
- [StreamGuard (streaming validation)](docs/STREAMGUARD.md)
- [Data contract DSL](docs/contract-dsl.md)
- [Benchmarking](docs/bench/benchmark.py)
- [Honest roadmap & limitations](docs/ROADMAP_HONEST.md)

## Contributing

Contributions are welcome. Please review [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on development setup, code style, and testing.

## License

Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.

