Metadata-Version: 2.4
Name: dq-engine
Version: 0.5.3
Summary: Framework de validação e limpeza de dados com PySpark
Author-email: Bruna Cataldo <bruna.cataldo@autoglass.com.br>
Project-URL: Homepage, https://bitbucket.org/ced_engenharia/data_quality
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pyspark<4.0.0,>=3.4.0
Provides-Extra: dev
Requires-Dist: pytest<9.0.0,>=8.0.0; extra == "dev"
Requires-Dist: pytest-mock<4.0.0,>=3.12.0; extra == "dev"
Provides-Extra: notebook
Requires-Dist: openpyxl<4.0.0,>=3.1.0; extra == "notebook"
Requires-Dist: pandas<3.0.0,>=2.0.0; extra == "notebook"

# dq-engine

Framework PySpark para validacao e tratamento de qualidade de dados.

## O que faz

- Resolve rule_set por nome de coluna
- Compila regras declarativas em expressoes Spark
- Gera score e status por coluna
- Aplica tratamento em lote
- Executa pipeline antes/depois com comparativo

## Requisitos

- Python 3.10+
- PySpark 3.4+

## Instalacao

```bash
pip install -e .
```

## Uso rapido

```python
from pyspark.sql import SparkSession
from dq_engine import DataQualityEngine
from dq_engine.config.conventions import CONVENTIONS

spark = SparkSession.builder.getOrCreate()
engine = DataQualityEngine(spark=spark, conventions=CONVENTIONS)

df = spark.table("lakehouse.clientes")
result = engine.validate_table(df=df, table_name="CLIENTES", run_id="2026-07-02")

result.output_df.orderBy("column_name").show(truncate=False)
```

## Pipeline completo

```python
df_tratado, result_before, result_after = engine.run_dq_pipeline(
    df=df,
    table_name="CLIENTES",
    output_dir="/mnt/dq_output",  # opcional
)
```

## API principal

- DataQualityEngine.validate_table
- DataQualityEngine.apply_treatment
- DataQualityEngine.run_dq_pipeline
- DataQualityEngine.get_column_treatment_contexts
- DataQualityResult.table_summary
- DataQualityResult.failed_columns

## Configuracao

Arquivo central de convencoes:

- src/dq_engine/config/conventions.py

## Documentacao

- docs/architecture.md
- docs/rules.md
