Metadata-Version: 2.4
Name: spark-data-quality
Version: 1.0.1
Summary: SparkDQAgent — Data Quality validation package for K8s Spark pods
Author-email: khailas <khailas.rangath@saal.ai>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: great-expectations==0.18.12
Requires-Dist: trino>=0.320.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: spark
Requires-Dist: pyspark>=3.1.1; extra == "spark"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations and persists results to a DQ Engine API.

## Installation

```bash
pip install .            # core (GE + SQL)
pip install ".[spark]"   # includes PySpark
pip install -e ".[dev]"  # editable + dev tools
```

## Usage

### Single table

```python
from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="public",
    table="sales_data",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    table_id=13,
    test_suite_id=37,
    trino_host="sql-host:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.get_config()        # single GET /config call
results = agent.validate(df)    # hybrid SQL + GE execution
```

### Suite run (auto-discover tables)

```python
agent = SparkDQAgent(
    catalog="", schema="", table="",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    catalog_type="unmanaged",
    test_suite_id=37,
)
table_configs = agent.get_all_table_configs()  # returns list of per-table configs
```

## Hybrid execution

When SQL engine credentials are provided, the agent splits expectations into two paths:

| Path | Assertion types | How |
|------|----------------|-----|
| **SQL fast-path** | `not_be_null`, `be_unique`, `be_between`, `match_regex`, `row_count_to_be_between` | Single batched SQL query |
| **GE slow-path** | All others | Parallel in-memory GE validators |

Both paths run **concurrently**. Results are merged and saved via `POST /save-results`.

## License

MIT
