Metadata-Version: 2.4
Name: databricks-labs-dqx
Version: 0.16.0
Summary: Data Quality eXtended (DQX) is a Python library for data quality checks and data quality monitoring
Project-URL: Issues, https://github.com/databrickslabs/dqx/issues
Project-URL: Source, https://github.com/databrickslabs/dqx
Author-email: Databricks Labs <labs-oss@databricks.com>, Marcin Wojtyczka <marcin.wojtyczka@databricks.com>, Alex Ott <alexey.ott@databricks.com>
Maintainer-email: Databricks Labs <labs-oss@databricks.com>, Marcin Wojtyczka <marcin.wojtyczka@databricks.com>, Alex Ott <alexey.ott@databricks.com>, Greg Hansen <gregory.hansen@databricks.com>
License-File: LICENSE
License-File: NOTICE
Keywords: Databricks
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: databricks-labs-blueprint<=0.12,>=0.9.1
Requires-Dist: databricks-sdk~=0.73
Requires-Dist: pydantic<3,>=2.8.2
Requires-Dist: pyyaml~=6.0.3
Requires-Dist: sqlalchemy<3.0,>=2.0
Provides-Extra: anomaly
Requires-Dist: cloudpickle<4.0,>=3.0; extra == 'anomaly'
Requires-Dist: mlflow[databricks]<3.0,>=2.19.1; extra == 'anomaly'
Requires-Dist: scikit-learn<2.0,>=1.0; extra == 'anomaly'
Requires-Dist: shap<0.50,>=0.42.0; extra == 'anomaly'
Provides-Extra: cli
Requires-Dist: databricks-connect<18,>=15.4; extra == 'cli'
Provides-Extra: datacontract
Requires-Dist: datacontract-cli<1.0,>=0.11.1; extra == 'datacontract'
Provides-Extra: llm
Requires-Dist: databricks-langchain<1.0,>=0.1.0; extra == 'llm'
Requires-Dist: dspy~=3.1.3; extra == 'llm'
Requires-Dist: langchain-core<1.0,>=0.1.0; extra == 'llm'
Requires-Dist: litellm<=1.82.6,>=1.64.0; extra == 'llm'
Provides-Extra: pii
Requires-Dist: numpy<2.0,>=1.20; extra == 'pii'
Requires-Dist: presidio-analyzer~=2.2.359; extra == 'pii'
Requires-Dist: spacy~=3.8; extra == 'pii'
Description-Content-Type: text/markdown

DQX by Databricks Labs
===

<p align="center">
    <a href="https://github.com/databrickslabs/dqx">
        <img src="https://raw.githubusercontent.com/databrickslabs/dqx/refs/heads/main/docs/dqx/static/img/logo.svg" class="align-center" width="200" height="200" alt="logo" />
    </a>
</p>

Simplified Data Quality checking at Scale for PySpark Workloads on streaming and standard DataFrames.

[![build](https://github.com/databrickslabs/dqx/actions/workflows/push.yml/badge.svg)](https://github.com/databrickslabs/dqx/actions/workflows/push.yml) 
[![codecov](https://codecov.io/github/databrickslabs/dqx/graph/badge.svg)](https://codecov.io/github/databrickslabs/dqx) 
![linesofcode](https://aschey.tech/tokei/github/databrickslabs/dqx?category=code)
[![PyPI](https://img.shields.io/pypi/v/databricks-labs-dqx?label=pypi%20package&cacheSeconds=3600)](https://pypi.org/project/databricks-labs-dqx/) 
![PyPI Downloads](https://static.pepy.tech/personalized-badge/databricks-labs-dqx?period=month&units=international_system&left_color=grey&right_color=orange&left_text=PyPI%20downloads&cacheSeconds=3600)

# ✨ Key capabilities

- **Rule-based quality checks** — 80+ built-in checks (null, range, regex, referential, aggregate, geo, PII, and more) for row-level and column/dataset-level validation, with support for custom check functions.
- **Code or config checks** — define checks programmatically as code or declaratively as YAML/JSON configuration.
- **Check levels** — mark failed checks as warning or error.
- **Custom reactions to failed checks** — drop, mark, or quarantine invalid data flexibly.
- **Detailed failure info** — get detailed insights into why each check failed.
- **AI-assisted rule generation** — LLM-driven rule suggestions from business descriptions, powered by DSPy and Databricks Model Serving.
- **Data profiling & rule generation** — automatic statistics collection and quality rule candidate generation from existing data.
- **ML row anomaly detection** — Isolation Forest–based row anomaly detection with SHAP explanations and AI-generated narratives.
- **Data contracts** — generate quality rules from ODCS data contracts, including schema validation.
- **Summary metrics & quality dashboard** — built-in and custom aggregate metrics (input/error/warning/valid row counts, per-check breakdowns) persisted to Delta tables, with a Lakeview quality dashboard for tracking and identifying data quality issues.
- **Actions and alerting** — automatically send Slack, Microsoft Teams, or generic webhook alerts and/or fail the pipeline when summary metrics cross a threshold.
- **Flexible checks storage** — save and load quality rules from YAML/JSON files, Unity Catalog tables, Volumes, or Lakebase (PostgreSQL).
- **DQX Studio** — browser-based no-code UI for authoring, reviewing, running, and monitoring quality rules, deployed as a Databricks App.
- **Data format agnostic & streaming support** — works with PySpark DataFrames and applies checks to both batch and Spark Structured Streaming (including Lakeflow Pipelines / DLT) using the same API.

# 📖 Documentation

The complete documentation is available at: [https://databrickslabs.github.io/dqx/](https://databrickslabs.github.io/dqx/)

# 🛠️ Contribution

Please see the contribution guidance [here](https://databrickslabs.github.io/dqx/docs/dev/contributing/) on how to contribute to the project (build, test, and submit a PR).

# 💬 Project Support

Please note that this project is provided for your exploration only and is not 
formally supported by Databricks with Service Level Agreements (SLAs). They are 
provided AS-IS, and we do not make any guarantees. Please do not 
submit a support ticket relating to any issues arising from the use of this project.

Any issues discovered through the use of this project should be filed as GitHub 
[Issues on this repository](https://github.com/databrickslabs/dqx/issues). 
They will be reviewed as time permits, but no formal SLAs for support exist.
