Metadata-Version: 2.1
Name: dscribe-dq
Version: 0.0.4
Summary: Automatically generated by Nx.
License: Proprietary
Requires-Python: >=3.10,<3.14
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: databricks-sdk (>=0.20.0)
Requires-Dist: databricks-sql-connector (>=4.0.0,<5.0.0)
Requires-Dist: databricks-sqlalchemy (>=2.0.9,<3.0.0)
Requires-Dist: great-expectations (>=1.0.0,<2.0.0)
Requires-Dist: pyodbc (>=5.0.0,<6.0.0)
Requires-Dist: pyspark (>=3.5.0,<5.0.0)
Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
Description-Content-Type: text/markdown

# dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe — all in one function call.

> For library internals, architecture, and contributing, see [DEVELOPMENT.md](DEVELOPMENT.md).

## Prerequisites

- A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
- Your dScribe API key (Settings → API keys in the dScribe UI)
- The asset UUID you want to validate
- Access to the database the rules target (Databricks or MSSQL)

## Installation

```bash
pip install dscribe-dq
```

## Walkthrough

### 1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

```
https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                  this is your asset_id
```

Your API key is under **Settings → API keys**.

### 2. Define your data quality rules in dScribe

Rules are defined in your asset's ODCS spec under `schema[].quality` (dataset-level) or `schema[].properties[].quality` (column-level). Each rule must have a `sourceId` in its `customProperties` that matches a server entry in the `servers` block.

The library maps ODCS metrics to Great Expectations checks automatically. Supported metrics:

| ODCS `metric`       | What it checks                                      |
| ------------------- | --------------------------------------------------- |
| `rowCount`          | Row count within expected bounds                    |
| `nullValues`        | No NULL values in a column                          |
| `missingValues`     | No missing/empty values in a column                 |
| `duplicateValues`   | All values in a column (or column set) are unique   |
| `invalidValues`     | Values match an allowed list or regex pattern       |

### 3. Connect to Databricks

The library authenticates using an **Azure AD service principal** (OAuth M2M). You need:

- The Databricks workspace hostname
- An Azure AD application with client ID and secret
- The tenant ID of your Azure AD directory
- The HTTP path of your SQL warehouse

**Set environment variables** (recommended for CI/CD and notebooks):

```bash
DATABRICKS_HOST=adb-858283489583940.0.azuredatabricks.net
DATABRICKS_CLIENT_ID=f0d8e3e9-b910-402f-bc21-9bbefc5432ef
DATABRICKS_CLIENT_SECRET=<your-secret>
DATABRICKS_TENANT_ID=307d700a-c1eb-4e24-bb12-56f114c2d470
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abc123def456
DATABRICKS_CATALOG=hive_metastore        # optional
DATABRICKS_SCHEMA=default                # optional
DSCRIBE_API_KEY=<your-api-key>
DSCRIBE_ASSET_ID=337eaa9e-47ed-4b37-a124-050d4932a520
```

Then run with no arguments:

```python
from dscribe_dq import run_validation

results = run_validation()
```

**Or pass credentials in code:**

```python
from dscribe_dq import run_validation

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "f0d8e3e9-b910-402f-bc21-9bbefc5432ef",
            "client_secret": "<your-secret>",
            "tenant_id": "307d700a-c1eb-4e24-bb12-56f114c2d470",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",
            "schema": "default",
        }
    },
)
```

> The `http_path` can be found in the Databricks UI under **SQL Warehouses → your warehouse → Connection details**.

### 4. Connect to MSSQL

Two authentication modes are supported: SQL Server (username + password) and Entra ID (service principal).

**SQL Server authentication:**

```python
results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "SQL Server",
            "username": "your-user",
            "password": "your-password",
        }
    },
)
```

**Entra ID (service principal) authentication:**

```python
results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "Entra ID",
            "tenant_id": "<tenant-id>",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
        }
    },
)
```

### 5. Run against multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in `source_configs`:

```python
results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            # MSSQL config ...
        },
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            # Databricks config ...
        },
    },
)
```

The library groups rules by source, runs each connector independently, and merges all results.

### 6. Understand the results

`run_validation` returns a list of dicts, one per rule:

```python
[
    {
        "rule_id": "b9426493-3eb6-4ad5-872a-41a0533adac1",
        "expectation": "expect_column_values_to_not_be_null",
        "column": "ListPrice",
        "success": True,
        "result": { ... }   # full Great Expectations result dict
    },
    ...
]
```

Results are also written back to dScribe automatically so the asset's quality status updates in the UI. Each rule gets a `lastCheckStatus` (`passed` or `failed`) and `lastCheckTimestamp` added to its `customProperties`.

### 7. Optional: dry run and logging

**Dry run** — validate locally without posting results back to dScribe:

```python
results = run_validation(..., dry_run=True)
```

**Log level** — reduce output to results only:

```python
results = run_validation(..., log_level="RESULT")
```

Log levels in order: `DEBUG` → `INFO` → `RESULT` → `WARNING` → `ERROR`. Use `RESULT` for scheduled jobs to see only pass/fail lines and the summary table.

**Save failed rows to CSV** — collect the actual failing rows for each failed rule:

```python
results = run_validation(
    ...,
    failed_rows_mode="csv",
    failed_rows_dir="./reports/failed",
)
```

Files are written to `<failed_rows_dir>/<asset_id>/<dataset>_failed_rows_<timestamp>.csv`.

## Environment variable reference

| Variable                   | Description                                    | Default                                 |
| -------------------------- | ---------------------------------------------- | --------------------------------------- |
| `DSCRIBE_API_KEY`          | dScribe API key                                | —                                       |
| `DSCRIBE_ASSET_ID`         | Asset UUID to validate                         | —                                       |
| `DSCRIBE_BASE_URL`         | dScribe API base URL                           | `https://app.dscribe.cloud/catalog/api` |
| `DQ_DRY_RUN`               | Skip write-back when `true`                    | `false`                                 |
| `DQ_FAILED_ROWS_MODE`      | `none` or `csv`                                | `none`                                  |
| `DQ_FAILED_ROWS_DIR`       | Output directory for failed-rows CSVs          | `./reports/dscribe-dq/failed_rows`      |
| `DATABRICKS_HOST`          | Databricks workspace hostname                  | —                                       |
| `DATABRICKS_CLIENT_ID`     | Azure AD service principal client ID           | —                                       |
| `DATABRICKS_CLIENT_SECRET` | Azure AD service principal client secret       | —                                       |
| `DATABRICKS_TENANT_ID`     | Azure AD tenant ID                             | —                                       |
| `DATABRICKS_HTTP_PATH`     | SQL warehouse HTTP path                        | —                                       |
| `DATABRICKS_WAREHOUSE_ID`  | SQL warehouse ID (alternative to HTTP path)    | —                                       |
| `DATABRICKS_CATALOG`       | Default Unity Catalog catalog name             | —                                       |
| `DATABRICKS_SCHEMA`        | Default schema name                            | —                                       |
| `MSSQL_HOST`               | MSSQL server hostname                          | —                                       |
| `MSSQL_DATABASE`           | MSSQL database name                            | —                                       |
| `MSSQL_USER`               | SQL Server username                            | —                                       |
| `MSSQL_PASSWORD`           | SQL Server password                            | —                                       |
| `MSSQL_AUTH`               | `SQL Server` or `Entra ID`                     | `SQL Server`                            |
| `MSSQL_TENANT_ID`          | Azure tenant ID (Entra ID auth only)           | —                                       |
| `MSSQL_CLIENT_ID`          | Azure client ID (Entra ID auth only)           | —                                       |
| `MSSQL_CLIENT_SECRET`      | Azure client secret (Entra ID auth only)       | —                                       |

