Metadata-Version: 2.4
Name: prizm-adls-scanner
Version: 0.1.0
Summary: Prizm ADLS Gen2 data quality scanner (operational, profile, structural, and business metrics)
Author-email: Prizm Team <team@prizm.com>
Maintainer-email: Prizm Team <team@prizm.com>
License: MIT
Project-URL: Homepage, https://github.com/DQLabs-Inc/prizm-cli
Project-URL: Repository, https://github.com/DQLabs-Inc/prizm-cli.git
Project-URL: Bug Tracker, https://github.com/DQLabs-Inc/prizm-cli/issues
Keywords: prizm,adls,azure,data-quality,metrics,databricks,spark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: pyspark<4.0.0,>=3.0.0
Requires-Dist: pandas<2.0.0,>=1.0.0
Requires-Dist: azure-storage-file-datalake<13.0.0,>=12.9.0
Requires-Dist: azure-identity<2.0.0,>=1.12.0
Requires-Dist: urllib3<2.0.0,>=1.26.0
Requires-Dist: python-dateutil<3.0.0,>=2.8.0
Provides-Extra: dev
Requires-Dist: pre-commit==2.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest==8.2.2; extra == "test"
Requires-Dist: pytest-cov==2.0; extra == "test"
Requires-Dist: pytest-mock==3.0; extra == "test"
Dynamic: license-file

# prizm-adls-scanner

Prizm ADLS Gen2 data quality scanner for collecting operational, profile, structural, and business metrics from files in Azure Data Lake Storage, and sending the results to Prizm.

**Location:** `prizm-cli/prizm-adls-scanner` (under Server root)

**Key Design Principles:**
- ✅ Scans **ADLS Gen2 only** — no S3 support (use a different connector for S3 sources)
- ✅ Works with or without an active Spark session — falls back to Azure SDK-only reads when no Spark session is available
- ✅ Safe to use in CI/CD, cron, GitHub Actions, Jenkins, etc.
- ✅ Explicit flags, no magic, deterministic behavior

**Not included here:** the Databricks notebook path (`databricks/notebook_runner.py`, `notebooks/prizm_scanner.py`) that runs on a Spark cluster notebook to build the Iceberg table and collect metrics — that's a separate deployment and lives outside this CLI package.

## Installation

### From Wheel File

```bash
pip install dist/prizm_adls_scanner-*.whl
```

### From Source (Development)

```bash
pip install -e .
```

For development, you can install the package in editable mode. See the [Development](#development) section for more details.

## Quick Start

### 1. Set Environment Variables (Optional)

You can pass credentials as flags, or export them and reference via your shell:

```bash
export DQLABS_ACCESS_TOKEN=prizm_xxx
```

### 2. Scan a File and Push Metrics to Prizm

```bash
prizm-adls-scanner \
  --access-token "$DQLABS_ACCESS_TOKEN" \
  --mcp-host https://mcp.dqlabs.ai \
  --asset-id "my-asset" \
  --source-id "my-source" \
  --storage-account mystorageaccount \
  --container mycontainer \
  --file-path path/to/file.csv
```

## Command

### `prizm-adls-scanner`

Read a file from ADLS Gen2, compute operational/profile/structural (and optionally business/duplicate) metrics, and push the results to Prizm.

**Required Flags:**
- `--access-token`: Prizm/DQLabs access token (or `--api-key`, deprecated)
- `--asset-id`: Prizm asset ID
- `--source-id`: Prizm source ID
- `--storage-account`: ADLS storage account name
- `--container`: ADLS container name
- `--file-path`: Path to the file within the container

**Optional Flags:**
- `--mcp-host`: MCP host URL (default: `https://mcp.dqlabs.ai`)
- `--access-key`: Storage account access key (alternative: Azure AD service principal via `DQLABS_CLIENT_ID`/`DQLABS_CLIENT_SECRET`/`DQLABS_TENANT_ID` environment variables)
- `--max-columns`: Maximum columns to profile (default: 300)
- `--no-duplicates`: Disable duplicate-row detection
- `--webhook-url`: Custom webhook URL for results
- `--output`: `json` (default) or `pretty`
- `--quiet`: Suppress progress output

**Example:**

```bash
prizm-adls-scanner \
  --access-token "$DQLABS_ACCESS_TOKEN" \
  --asset-id "sales-daily-csv" \
  --source-id "adls-prod" \
  --storage-account mystorageaccount \
  --container raw \
  --file-path sales/2026/08/sales.csv \
  --max-columns 200 \
  --output pretty
```

**Metrics Collected:**
- Operational: row count, column count, schema, file size, freshness
- Profile: completeness, uniqueness, character, space per column
- Structural: per-column technical statistics
- Duplicate detection (full-row or partition-based, unless `--no-duplicates`)
- Business/exception/lookup/behavioral/comparison metrics, when configured server-side for the asset

## CI/CD Integration

### GitHub Actions

```yaml
- name: Push ADLS scan metrics to Prizm
  env:
    DQLABS_ACCESS_TOKEN: ${{ secrets.DQLABS_ACCESS_TOKEN }}
  run: |
    prizm-adls-scanner \
      --access-token "$DQLABS_ACCESS_TOKEN" \
      --asset-id "my-asset" \
      --source-id "my-source" \
      --storage-account mystorageaccount \
      --container mycontainer \
      --file-path path/to/file.csv
```

### Jenkins

```groovy
stage('Scan ADLS file with Prizm') {
    steps {
        sh '''
            prizm-adls-scanner \
              --access-token "${DQLABS_ACCESS_TOKEN}" \
              --asset-id "my-asset" \
              --source-id "my-source" \
              --storage-account mystorageaccount \
              --container mycontainer \
              --file-path path/to/file.csv
        '''
    }
}
```

## Development

### Development Setup

```bash
pip install -e ".[dev,test]"
```

### Running Tests

```bash
make test
```

## Building Wheels

```bash
make build
```

For offline/local environments where build dependencies are already installed:

```bash
make build-local
```

This outputs a wheel to `dist/`.

### Clean Build Artifacts

```bash
make clean
```

This removes `build/` and `dist/` directories, `egg-info`, and Python cache files.

## Security

- Token-based authentication via `--access-token` or environment variable
- Tokens never logged or printed
- TLS enforced for all API calls

## License

MIT License

## Support

For issues and questions, please visit:
https://github.com/DQLabs-Inc/prizm-cli/issues
