Metadata-Version: 2.5
Name: eliza-dq
Version: 0.1.0
Summary: Fastest open-source data quality engine for Polars, SQL, and any DataFrame
Project-URL: Homepage, https://github.com/Se7enquick/eliza
Project-URL: Repository, https://github.com/Se7enquick/eliza
Project-URL: Issues, https://github.com/Se7enquick/eliza/issues
Author-email: Vladislav Koval <koval.vladislavv@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-engineering,data-quality,delta-lake,iceberg,polars,validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: polars>=1.0.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: all
Requires-Dist: connectorx>=0.3; extra == 'all'
Requires-Dist: fpdf2>=2.7; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'all'
Requires-Dist: pyathena>=3.0; extra == 'all'
Requires-Dist: rich>=13.0; extra == 'all'
Provides-Extra: athena
Requires-Dist: pyathena>=3.0; extra == 'athena'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'bigquery'
Provides-Extra: cli
Requires-Dist: rich>=13.0; extra == 'cli'
Provides-Extra: clickhouse
Requires-Dist: clickhouse-connect>=0.7; extra == 'clickhouse'
Provides-Extra: connectorx
Requires-Dist: connectorx>=0.3; extra == 'connectorx'
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector>=3.0; extra == 'databricks'
Provides-Extra: delta
Requires-Dist: deltalake>=0.15; extra == 'delta'
Provides-Extra: dev
Requires-Dist: fpdf2>=2.7; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: rich>=13.0; extra == 'dev'
Requires-Dist: ruff>=0.5.0; extra == 'dev'
Provides-Extra: iceberg
Requires-Dist: pyiceberg>=0.5; extra == 'iceberg'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: redshift
Requires-Dist: redshift-connector>=2.0; extra == 'redshift'
Provides-Extra: report
Requires-Dist: fpdf2>=2.7; extra == 'report'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.0; extra == 'snowflake'
Provides-Extra: test
Requires-Dist: pandas>=2.0; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

<div align="center">

# Eliza DQ

**The fastest open-source data quality engine for Python.**

*259M rows. 17 checks. Samples. 1.5 seconds.*

[![CI](https://github.com/Se7enquick/eliza/actions/workflows/ci.yml/badge.svg)](https://github.com/Se7enquick/eliza/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/eliza-dq/)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)

</div>

---

Eliza DQ validates DataFrames and warehouse tables with streaming execution, parallel SQL pushdown, and instant failure sampling. It runs on Polars LazyFrames (constant memory, no matter the data size), connects to 8 warehouses, and ships with 2 dependencies.

```bash
pip install eliza-dq
```

```python
from eliza import check

result = check("data.parquet", checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative"],
    "email":    ["is_email"],
})
print(result.summary())
# 3 passed, 0 warnings, 2 failed (1,000,000 rows, 3ms)
```

## Benchmarks

### DataFrame Engine

In-memory Polars DataFrame, 5 checks, warmup + 3 runs, min time.

| Rows | Eliza | Cuallee | Pandera | Pointblank | GX |
|------|-------|---------|---------|------------|-----|
| **3M** | **1.6ms** | 5.3ms | 5.7ms | 35ms | 331ms |
| **10M** | **4.5ms** | 13ms | - | - | - |
| **41M** | **14ms** | 29ms | - | - | - |
| **126M** | **40ms** | 89ms | 118ms | 1,453ms | 4,400ms |
| **259M** | **1.5s** | OOM | OOM | OOM | OOM |

> Eliza streams from disk via LazyFrames — constant memory regardless of file size. Competitors must load the entire dataset into RAM.

### SQL Pushdown Engine

Athena, 179M rows (crossref_silver), 8 not_null checks.

| | Eliza | Soda Core |
|---|---|---|
| **Without samples** | **~10s** | 22.4s |
| **With samples (10 rows)** | **16.8s** | DNF (killed after 5 min) |
| **17 checks + samples** | **18.4s** | - |

> Eliza batches all inline checks into one `SELECT`, runs separate checks and samples in parallel, and uses `LIMIT N` on sample queries. Soda runs queries sequentially and fetches ALL failing rows before truncating in memory.

## Eliza vs Competitors

### Features

| Feature | Eliza | Soda | GX | Pandera | Cuallee | Dataframely |
|---------|:-----:|:----:|:--:|:-------:|:-------:|:-----------:|
| Polars native | Yes | - | - | Yes | Yes | Yes |
| LazyFrame streaming | Yes | - | - | - | - | - |
| SQL pushdown | 8 DWH | Yes | Yes | - | - | - |
| Parallel SQL queries | Yes | - | - | - | - | - |
| Failed row samples | `LIMIT N` | All rows* | - | - | - | All rows* |
| YAML config | Yes | Yes | Yes | - | - | - |
| Inline dict API | Yes | - | - | Yes | Yes | Yes |
| CLI | Yes | Yes | Yes | - | - | - |
| Auto-learn from data | Yes | - | Yes | Yes | - | - |
| PDF report | Yes | - | - | - | - | - |
| Slack alerting | Yes | Cloud** | - | - | - | - |
| Partition filter | Yes | Yes | Yes | - | - | - |
| Schema validation | Yes | Yes | Yes | Yes | - | Yes |
| FK reference check | Yes | Yes | Yes | - | - | - |
| Core dependencies | **2** | 30+ | 30+ | 7+ | 3+ | 2 |

<sub>* Materializes all failing rows in memory before truncating — causes OOM/timeout on large failures.</sub><br>
<sub>** Soda Slack alerting requires Soda Cloud ($25k+/yr).</sub>

### Checks

| Check | Eliza | Soda | GX | Pandera | Cuallee |
|-------|:-----:|:----:|:--:|:-------:|:-------:|
| not_null | Yes | Yes | Yes | Yes | Yes |
| not_missing (custom) | Yes | Yes | Yes | - | - |
| unique | Yes | Yes | Yes | Yes | Yes |
| not_negative | Yes | Yes | Yes | Yes | Yes |
| between (range) | Yes | Yes | Yes | Yes | Yes |
| in_set | Yes | Yes | Yes | Yes | Yes |
| regex | Yes | Yes | Yes | Yes | Yes |
| is_email | Yes | - | - | - | - |
| is_url | Yes | - | - | - | - |
| min/max_length | Yes | Yes | Yes | - | - |
| freshness | Yes | Yes | - | - | - |
| row_count | Yes | Yes | Yes | - | - |
| cross_column | Yes | - | Yes | Yes | - |
| schema | Yes | Yes | Yes | Yes | - |
| reference (FK) | Yes | Yes | Yes | - | - |
| custom SQL | Yes | Yes | Yes | - | - |
| anomaly detection | - | Cloud | Yes | - | - |
| distribution | - | Cloud | Yes | - | - |
| change over time | - | Cloud | - | - | - |

### Warehouse Support

| Warehouse | Eliza | Soda | GX |
|-----------|:-----:|:----:|:--:|
| BigQuery | Yes | Yes | Yes |
| Athena | Yes | Yes | Yes |
| Snowflake | Yes | Yes | Yes |
| PostgreSQL | Yes | Yes | Yes |
| MySQL | Yes | Yes | Yes |
| ClickHouse | Yes | - | - |
| Databricks | Yes | Yes | Yes |
| Redshift | Yes | Yes | Yes |

```bash
pip install eliza-dq[bigquery]   # install only what you need
pip install eliza-dq[athena]
pip install eliza-dq[snowflake]
pip install eliza-dq[postgres]
pip install eliza-dq[clickhouse]
pip install eliza-dq[mysql]
pip install eliza-dq[databricks]
pip install eliza-dq[redshift]
```

## Quick Start

### Inline checks (notebook / script)

```python
import polars as pl
from eliza import check

df = pl.read_parquet("orders.parquet")

result = check(df, checks={
    "order_id": ["not_null", "unique"],
    "amount":   ["not_null", "not_negative", {"between": {"min": 0, "max": 100000}}],
    "email":    ["is_email"],
    "status":   [{"in_set": {"values": ["pending", "shipped", "delivered"]}}],
    "name":     [{"min_length": {"min": 2}}, {"max_length": {"max": 100}}],
})

print(result.summary())
result.raise_on_fail()  # exit code 1 on failure
```

### YAML config (production)

```yaml
# eliza_checks/orders.yaml
connection:
  type: bigquery
  project: my-project-123

table: my-project-123.analytics.orders

filter: "created_at >= '2024-01-01'"

samples:
  limit: 20

checks:
  - column: order_id
    check: not_null
  - column: order_id
    check: unique
  - column: amount
    check: not_negative
  - column: email
    check: not_missing
    missing_values: ["", "N/A", "null"]
  - column: updated_at
    check: freshness
    max_age: 24h
  - check: row_count
    min: 1000
```

```python
from eliza import check
result = check(config="orders")
```

### CLI

```bash
# Initialize project
eliza init

# Auto-learn checks from data
eliza learn data/orders.parquet --name orders

# Run checks (exit code: 0=pass, 1=fail, 2=error)
eliza check --config orders --source data/orders.parquet

# JSON output for orchestrators
eliza check --config orders --format json
```

## Alerting & Reporting

```python
from eliza import check
from eliza.alert import send_slack
from eliza.report import generate_pdf

result = check(config="orders")

# Slack message + PDF attachment
send_slack(result, token="xoxb-...", channel="C...", pdf=True, name="orders")

# PDF report with charts (donut, failure bars, sample tables)
generate_pdf(result, name="orders")
# -> eliza_orders_2026-09-07.pdf
```

```bash
pip install eliza-dq[report]  # for PDF reports
```

## CI/CD Integration

```yaml
# .github/workflows/dq.yml
- run: pip install eliza-dq
- run: eliza check --config orders --source data/orders.parquet
```

```python
# Airflow
@task
def dq_check():
    from eliza import check
    result = check(config="orders")
    result.raise_on_fail()
    return result.to_dict()
```

## Architecture

**Polars native** (DataFrames, files): Streaming engine with per-column grouping. Files are scanned as LazyFrames — data streams through without loading into RAM. Each column group runs one `collect(engine="streaming")` call. Failed row samples use `.filter().head(N).collect(engine="streaming")` — instant, no full materialization.

**SQL pushdown** (warehouses): All inline checks batched into one `SELECT COUNT(*), SUM(CASE WHEN ...) FROM table`. Separate checks (unique, freshness) and sample queries run in parallel via `ThreadPoolExecutor` with thread-local connections. Sample queries use `LIMIT N` in SQL — never fetches all failing rows.

## License

MIT
