Metadata-Version: 2.1
Name: dscribe-dq
Version: 1.0.2
Summary: Automatically generated by Nx.
License: Proprietary
Requires-Python: >=3.10,<3.14
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: databricks-sdk (>=0.20.0)
Requires-Dist: databricks-sql-connector (>=4.0.0,<5.0.0)
Requires-Dist: databricks-sqlalchemy (>=2.0.9,<3.0.0)
Requires-Dist: great-expectations (>=1.0.0,<2.0.0)
Requires-Dist: pyodbc (>=5.0.0,<6.0.0)
Requires-Dist: pyspark (>=3.5.0,<5.0.0)
Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
Description-Content-Type: text/markdown

# dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

> For library internals, architecture, and contributing, see [DEVELOPMENT.md](DEVELOPMENT.md).

## Prerequisites

- A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
- Your dScribe API key (Settings → API keys in the dScribe UI)
- The asset UUID you want to validate
- Access to the database the rules target (Databricks or MSSQL)

## Installation

```bash
pip install dscribe-dq
```

## How it works

Initialize `DScribeDQ` once with your credentials, then call `run_validation` with a list of **post-processors**. Post-processors are small pipeline steps that run in order after validation — writing results back to dScribe, uploading failed-row CSVs, generating reports, etc.

```
DScribeDQ(credentials) → dq.run_validation(connector_config, postprocessors=[...])
```

The only post-processor built into the SDK is `write_back_to_dscribe`, which posts pass/fail results to dScribe so the asset's quality status updates in the UI.

## Quickstart

### 1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

```
https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Your API key is under **Settings → API keys**.

### 2. Run validation and write back to dScribe

```python
from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key="<your-api-key>",
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
)

ctx = dq.run_validation(
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)
```

Each rule in dScribe gets a `lastCheckStatus` (`passed` or `failed`), `lastCheckTimestamp`, and failure metrics added to its `customProperties` after the pipeline runs.

## Databricks notebook

```python
%pip install dscribe-dq
```

```python
from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key=dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY"),
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="<your-asset-uuid>",
)

ctx = dq.run_validation(
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)
```

> The `http_path` can be found in the Databricks UI under **SQL Warehouses → your warehouse → Connection details**.

## Connecting to MSSQL

**SQL Server authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}
```

**Entra ID (service principal) authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}
```

## Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in `source_configs`. Rules are automatically grouped by source and run independently:

```python
source_configs={
    "<mssql-server-id>": {"type": "sqlserver", ...},
    "<databricks-server-id>": {"type": "databricks", ...},
}
```

## CI/CD usage (env vars)

For automated pipelines, set env vars and call the module-level `run_validation` directly:

```bash
DSCRIBE_API_KEY=...
DSCRIBE_ASSET_ID=...
DSCRIBE_BASE_URL=...
DATABRICKS_HOST=...
DATABRICKS_CLIENT_ID=...
DATABRICKS_CLIENT_SECRET=...
DATABRICKS_TENANT_ID=...
DATABRICKS_HTTP_PATH=...
```

```python
from dscribe_dq import run_validation

ctx = run_validation()  # reads all config from env vars
```

## Options

### `DScribeDQ.__init__`

| Parameter   | Type | Default | Description                                          |
|-------------|------|---------|------------------------------------------------------|
| `api_key`   | str  | —       | dScribe API key (or set `DSCRIBE_API_KEY` env var)   |
| `base_url`  | str  | —       | dScribe API base URL (or set `DSCRIBE_BASE_URL`)     |
| `asset_id`  | str  | —       | Asset UUID to validate (or set `DSCRIBE_ASSET_ID`)   |
| `log_level` | str  | `INFO`  | `DEBUG`, `INFO`, `RESULT`, `WARNING`, `ERROR`        |

### `DScribeDQ.run_validation`

| Parameter             | Type  | Default | Description                                                        |
|-----------------------|-------|---------|--------------------------------------------------------------------|
| `source_configs`      | dict  | `{}`    | Per-source connection settings keyed by server ID from ODCS spec   |
| `connector_config`    | dict  | `{}`    | Default connection settings used when no per-source config found   |
| `collect_failed_rows` | bool  | `True`  | Fetch the actual failing rows for each failed rule                 |
| `enable_profiling`    | bool  | `False` | Compute descriptive statistics (row count, null counts, etc.)      |
| `postprocessors`      | list  | `[]`    | Pipeline steps to run after validation in order                    |

## Supported ODCS metrics

| ODCS `metric`     | What it checks                                    |
|-------------------|---------------------------------------------------|
| `rowCount`        | Row count within expected bounds                  |
| `nullValues`      | No NULL values in a column                        |
| `missingValues`   | No missing/empty values in a column               |
| `duplicateValues` | All values in a column (or column set) are unique |
| `invalidValues`   | Values match an allowed list or regex pattern     |

## Environment variable reference

| Variable                   | Description                                    |
|----------------------------|------------------------------------------------|
| `DSCRIBE_API_KEY`          | dScribe API key                                |
| `DSCRIBE_ASSET_ID`         | Asset UUID to validate                         |
| `DSCRIBE_BASE_URL`         | dScribe API base URL                           |
| `DATABRICKS_HOST`          | Databricks workspace hostname                  |
| `DATABRICKS_CLIENT_ID`     | Azure AD service principal client ID           |
| `DATABRICKS_CLIENT_SECRET` | Azure AD service principal client secret       |
| `DATABRICKS_TENANT_ID`     | Azure AD tenant ID                             |
| `DATABRICKS_HTTP_PATH`     | SQL warehouse HTTP path                        |
| `DATABRICKS_WAREHOUSE_ID`  | SQL warehouse ID (alternative to HTTP path)    |
| `DATABRICKS_CATALOG`       | Default Unity Catalog catalog name             |
| `DATABRICKS_SCHEMA`        | Default schema name                            |
| `MSSQL_HOST`               | MSSQL server hostname                          |
| `MSSQL_DATABASE`           | MSSQL database name                            |
| `MSSQL_USER`               | SQL Server username                            |
| `MSSQL_PASSWORD`           | SQL Server password                            |
| `MSSQL_AUTH`               | `SQL Server` or `Entra ID`                     |
| `MSSQL_TENANT_ID`          | Azure tenant ID (Entra ID auth only)           |
| `MSSQL_CLIENT_ID`          | Azure client ID (Entra ID auth only)           |
| `MSSQL_CLIENT_SECRET`      | Azure client secret (Entra ID auth only)       |

