Metadata-Version: 2.4
Name: dqlabs-prizm
Version: 1.0.0
Summary: DQLabs Prizm - Data Quality Metrics Collection for Databricks and Cloud Storage
Author-email: DQLabs <support@dqlabs.ai>
Maintainer-email: DQLabs <support@dqlabs.ai>
License-Expression: MIT
Project-URL: Homepage, https://github.com/dqlabs/prizm-python
Project-URL: Documentation, https://github.com/dqlabs/prizm-python#readme
Project-URL: Repository, https://github.com/dqlabs/prizm-python
Project-URL: Issues, https://github.com/dqlabs/prizm-python/issues
Keywords: dqlabs,prizm,data-quality,metrics,databricks,spark,adls,s3
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: pyspark>=3.0.0
Requires-Dist: pandas>=1.0.0
Requires-Dist: azure-storage-file-datalake>=12.9.0
Requires-Dist: boto3>=1.26.0
Requires-Dist: urllib3>=1.26.0
Requires-Dist: python-dateutil>=2.8.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: flake8>=5.0.0; extra == "dev"
Requires-Dist: mypy>=0.990; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Provides-Extra: databricks
Requires-Dist: databricks-sdk>=0.1.0; extra == "databricks"
Dynamic: license-file

# DQLabs Prizm - Data Quality Metrics Library

[![Python Version](https://img.shields.io/badge/python-3.7+-blue.svg)](https://python.org)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Databricks](https://img.shields.io/badge/Databricks-Supported-red.svg)](https://databricks.com)

A comprehensive Python library for collecting operational, profile, and structural metrics from cloud storage (ADLS/S3) using Databricks Spark.

---

## 📋 Table of Contents

- [Features](#-features)
- [Installation](#-installation)
- [Quick Start](#-quick-start)
- [Configuration](#-configuration)
- [Usage Examples](#-usage-examples)
- [Metrics Collected](#-metrics-collected)
- [API Reference](#-api-reference)
- [Error Handling](#-error-handling)
- [Development](#-development)
- [License](#-license)

---

## ✨ Features

- ✅ **Operational Metrics**: Row count, column count, schema, file size, freshness
- ✅ **Profile Metrics**: Completeness, Uniqueness, Character, Space per column
- ✅ **Structural Metrics**: Per-column technical statistics
- ✅ **Duplicate Detection**: Full-row or partition-based duplicate counting
- ✅ **Multiple Storage**: ADLS Gen2 and S3 support
- ✅ **Databricks Native**: Optimized for Databricks Spark (Serverless compatible)
- ✅ **API Integration**: Sends metrics to DQLabs webhook endpoint
- ✅ **Webhook Support**: Custom webhook URL for results
- ✅ **Job Type Support**: OPERATIONAL, PROFILE, STRUCTURAL, FULL, COMPUTE_METRIC

---

## 📦 Installation

### Option 1: Install from PyPI (Recommended)

```bash
pip install dqlabs-prizm
```

### Option 2: Install from Source
```bash
git clone https://github.com/dqlabs/prizm-python.git
cd prizm-python
pip install -e .
```

### Option 3: Build locally
```bash
./build_package.sh
pip install dist/dqlabs_prizm-1.0.0-py3-none-any.whl
```

### Databricks Notebook Installation
```bash
%pip install dqlabs-prizm
dbutils.library.restartPython()
```

### Basic Usage
```bash
from dqlabs_prizm import PrizmScanner, PrizmConfig
from dqlabs_prizm.config.settings import JobType

# Create configuration
config = PrizmConfig(
    access_token="your_access_token",
    mcp_host="https://your-mcp-host.com",
    api_version="v1",
    asset_id="your_asset_id",
    source_id="your_source_id",
    qualified_name="your_file.csv",
    run_id="run_001",
    job_type=JobType.FULL,
    connector="adls",
    storage_account="your_storage_account",
    container="your_container",
    file_path="path/to/file.csv",
    access_key="your_access_key",
    extra_headers={"ngrok-skip-browser-warning": "true"}  # For ngrok tunnels
)

# Initialize scanner with Spark session
scanner = PrizmScanner(config, spark=spark, validate_token=True)

# Run the scan
result = scanner.scan()

# Access results
print(f"Row Count: {result['row_count']:,}")
print(f"Column Count: {result['column_count']}")
print(f"Duplicate Count: {result['duplicate_count']}")
```

### 🤝 Support
For support, please contact DQLabs support or open an issue on GitHub.

### 📝 Changelog
## Version 1.0.0
- Initial release
- Support for ADLS and S3
- Operational, Profile, Structural metrics
- Databricks Spark integration
- COMPUTE_METRIC job type
- Duplicate detection
- Webhook support
- Extra headers support
- Serverless compatible

### Built with ❤️ by DQLabs
