Metadata-Version: 2.4
Name: spark-data-quality
Version: 1.0.9
Summary: SparkDQAgent — Data Quality validation package for K8s Spark pods
Author-email: khailas <khailas.rangath@saal.ai>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: great-expectations==0.18.12
Requires-Dist: trino>=0.320.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: spark
Requires-Dist: pyspark>=3.1.1; extra == "spark"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# spark-data-quality

Data Quality validation library for Kubernetes Spark pods. Runs Great Expectations
and Trino SQL checks against any table, then persists results to a DQ Engine API.

## Installation

```bash
pip install spark-data-quality
```

## Usage

```python
from spark_dq.quality import SparkDQAgent

agent = SparkDQAgent(
    catalog="my_catalog",
    schema="my_schema",
    table="my_table",
    data_quality_url="http://dq-engine:8000/api/v1/spark",
    trino_host="trino:443",
    trino_user="user",
    trino_pwd="pwd",
)

cfg = agent.fetch_table_config()
results = agent.execute_data_quality(df)
```

### `fetch_table_config()`

Fetches the table's configuration from the DQ Engine, including the SQL query
to load data, active test suites with their expectations, and scan limits.
The config is cached after the first call.

### `execute_data_quality(df)`

Runs all test suites against the provided Spark DataFrame. Expectations are
split into two concurrent paths:

- **Trino fast-path** — null checks, uniqueness, range, regex via a single SQL query
- **GE slow-path** — all other expectation types via parallel Great Expectations validators

Suites run in parallel (up to 8 concurrently). Results are saved to the
DQ Engine and per-suite statistics are returned.

If Trino credentials are not provided, all expectations run through
Great Expectations only.

## License

MIT
