Metadata-Version: 2.1
Name: dscribe-dq
Version: 0.0.6
Summary: Automatically generated by Nx.
License: Proprietary
Requires-Python: >=3.10,<3.14
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: databricks-sdk (>=0.20.0)
Requires-Dist: databricks-sql-connector (>=4.0.0,<5.0.0)
Requires-Dist: databricks-sqlalchemy (>=2.0.9,<3.0.0)
Requires-Dist: great-expectations (>=1.0.0,<2.0.0)
Requires-Dist: pyodbc (>=5.0.0,<6.0.0)
Requires-Dist: pyspark (>=3.5.0,<5.0.0)
Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
Description-Content-Type: text/markdown

# dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

> For library internals, architecture, and contributing, see [DEVELOPMENT.md](DEVELOPMENT.md).

## Prerequisites

- A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
- Your dScribe API key (Settings → API keys in the dScribe UI)
- The asset UUID you want to validate
- Access to the database the rules target (Databricks or MSSQL)

## Installation

```bash
pip install dscribe-dq
```

## How it works

`run_validation` fetches your rules from dScribe, runs them with Great Expectations, and returns a `DQContext`. You then pipe that context through one or more **post-processors** — small steps that each do one thing (write back to dScribe, upload CSVs, generate reports). This keeps validation and reporting cleanly separated.

```
run_validation() → DQContext → [step1, step2, ...] → done
```

The only post-processor included in the SDK is `write_back_to_dscribe`, which posts pass/fail results back to dScribe so the asset's quality status updates in the UI. Additional post-processors (blob uploads, HTML reports) are available in the separate `postprocessors` package used by the dScribe runner.

## Quickstart

### 1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

```
https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Your API key is under **Settings → API keys**.

### 2. Run validation and write back to dScribe

```python
from dscribe_dq import run_validation, DScribeClient, write_back_to_dscribe

ctx = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
)

client = DScribeClient(api_key="<your-api-key>", base_url="<your-base-url>")

pipeline = [write_back_to_dscribe(client)]

for step in pipeline:
    step(ctx)
```

Each rule in dScribe gets a `lastCheckStatus` (`passed` or `failed`), `lastCheckTimestamp`, and failure metrics added to its `customProperties` after the pipeline runs.

## Databricks notebook

```python
%pip install dscribe-dq
```

```python
from dscribe_dq import run_validation, DScribeClient, write_back_to_dscribe

DSCRIBE_API_KEY = dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY")
DSCRIBE_BASE_URL = "<your-base-url>"
ASSET_ID = "<your-asset-uuid>"

ctx = run_validation(
    dscribe_key=DSCRIBE_API_KEY,
    base_url=DSCRIBE_BASE_URL,
    asset_id=ASSET_ID,
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
)

client = DScribeClient(api_key=DSCRIBE_API_KEY, base_url=DSCRIBE_BASE_URL)

pipeline = [write_back_to_dscribe(client)]

for step in pipeline:
    step(ctx)
```

> The `http_path` can be found in the Databricks UI under **SQL Warehouses → your warehouse → Connection details**.

## Connecting to MSSQL

**SQL Server authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}
```

**Entra ID (service principal) authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}
```

## Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in `source_configs`. Rules are automatically grouped by source and run independently:

```python
source_configs={
    "<mssql-server-id>": { "type": "sqlserver", ... },
    "<databricks-server-id>": { "type": "databricks", ... },
}
```

## Options

| Parameter            | Type   | Default  | Description                                                                 |
|----------------------|--------|----------|-----------------------------------------------------------------------------|
| `dscribe_key`        | str    | —        | dScribe API key (or set `DSCRIBE_API_KEY` env var)                          |
| `asset_id`           | str    | —        | Asset UUID to validate (or set `DSCRIBE_ASSET_ID` env var)                  |
| `base_url`           | str    | —        | dScribe API base URL (or set `DSCRIBE_BASE_URL` env var)                    |
| `source_configs`     | dict   | `{}`     | Per-source connection settings keyed by server ID from the ODCS spec        |
| `connector_config`   | dict   | `{}`     | Default connection settings used when no per-source config is found         |
| `collect_failed_rows`| bool   | `True`   | Fetch the actual failing rows for each failed rule                          |
| `enable_profiling`   | bool   | `False`  | Compute descriptive statistics (row count, null counts, distributions)       |
| `log_level`          | str    | `"INFO"` | Logging verbosity: `DEBUG`, `INFO`, `RESULT`, `WARNING`, `ERROR`            |

## Supported ODCS metrics

| ODCS `metric`     | What it checks                                    |
|-------------------|---------------------------------------------------|
| `rowCount`        | Row count within expected bounds                  |
| `nullValues`      | No NULL values in a column                        |
| `missingValues`   | No missing/empty values in a column               |
| `duplicateValues` | All values in a column (or column set) are unique |
| `invalidValues`   | Values match an allowed list or regex pattern     |

## Environment variable reference

| Variable                   | Description                                    |
|----------------------------|------------------------------------------------|
| `DSCRIBE_API_KEY`          | dScribe API key                                |
| `DSCRIBE_ASSET_ID`         | Asset UUID to validate                         |
| `DSCRIBE_BASE_URL`         | dScribe API base URL                           |
| `DATABRICKS_HOST`          | Databricks workspace hostname                  |
| `DATABRICKS_CLIENT_ID`     | Azure AD service principal client ID           |
| `DATABRICKS_CLIENT_SECRET` | Azure AD service principal client secret       |
| `DATABRICKS_TENANT_ID`     | Azure AD tenant ID                             |
| `DATABRICKS_HTTP_PATH`     | SQL warehouse HTTP path                        |
| `DATABRICKS_WAREHOUSE_ID`  | SQL warehouse ID (alternative to HTTP path)    |
| `DATABRICKS_CATALOG`       | Default Unity Catalog catalog name             |
| `DATABRICKS_SCHEMA`        | Default schema name                            |
| `MSSQL_HOST`               | MSSQL server hostname                          |
| `MSSQL_DATABASE`           | MSSQL database name                            |
| `MSSQL_USER`               | SQL Server username                            |
| `MSSQL_PASSWORD`           | SQL Server password                            |
| `MSSQL_AUTH`               | `SQL Server` or `Entra ID`                     |
| `MSSQL_TENANT_ID`          | Azure tenant ID (Entra ID auth only)           |
| `MSSQL_CLIENT_ID`          | Azure client ID (Entra ID auth only)           |
| `MSSQL_CLIENT_SECRET`      | Azure client secret (Entra ID auth only)       |

