Metadata-Version: 2.4
Name: dqal
Version: 0.1.0
Summary: DQAL — Data-Quality-Aware Learning: A real-time, model-agnostic quality assurance and selective retraining framework for ML pipelines.
Author-email: Pratham Khatri <prathamdkhatri@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Pratham1227-beep/dqal
Project-URL: Repository, https://github.com/Pratham1227-beep/dqal
Project-URL: Bug Tracker, https://github.com/Pratham1227-beep/dqal/issues
Project-URL: Changelog, https://github.com/Pratham1227-beep/dqal/blob/main/CHANGELOG.md
Keywords: machine-learning,data-quality,mlops,drift-detection,model-monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23.0
Requires-Dist: pandas>=1.5.0
Requires-Dist: scipy>=1.9.0
Requires-Dist: scikit-learn>=1.1.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: torch
Requires-Dist: torch>=2.0.0; extra == "torch"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.6.0; extra == "viz"
Requires-Dist: seaborn>=0.12.0; extra == "viz"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Provides-Extra: all
Requires-Dist: torch>=2.0.0; extra == "all"
Requires-Dist: matplotlib>=3.6.0; extra == "all"
Requires-Dist: seaborn>=0.12.0; extra == "all"
Requires-Dist: pytest>=7.0.0; extra == "all"
Dynamic: license-file

# 🛡️ DQAL — Data-Quality-Aware Learning

[![DQAL CI](https://github.com/Pratham1227-beep/dqal/actions/workflows/ci.yml/badge.svg)](https://github.com/Pratham1227-beep/dqal)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

> **Real-time, model-agnostic inference quality scoring, 3-tier gating (`SERVE` / `FLAG` / `ABSTAIN`), and selective retraining orchestration.**

---

## 1. Problem Statement

Deployed machine learning models fail silently. A model trained on clean, stationary data continues generating overconfident predictions when:
- **Upstream features drop out** (missing fields, API breaking changes, sensor timeouts)
- **Input distributions drift** (seasonal shifts, demographic changes, covariate shifts)
- **Extreme noise/outliers slip through** (hardware faults, corrupted data joins)

Traditional monitoring detects these weeks after downstream accuracy degrades. **DQAL** provides a lightweight, inline guardrail layer that inspects each prediction batch in real time, computes an interpretable Quality Score $Q \in [0, 1]$, gates execution, and safely triggers retraining.

---

## 2. System Architecture

```
                    ┌─────────────────────────────────────────┐
   Incoming batch → │           1. QUALITY SCORER             │
   of data          │   • Missingness (Feature null rates)    │
                    │   • Drift (Continuous PSI / Cat JS)     │
                    │   • Outlier (Isolation Forest / Mahala) │
                    └────────────────────┬────────────────────┘
                                         │ Q ∈ [0, 1], Sub-signals
                                         ▼
                    ┌─────────────────────────────────────────┐
                    │            2. TRIGGER LOGIC             │
                    │   • Q > 0.80       →  SERVE             │
                    │   • 0.50 < Q ≤ 0.80 →  FLAG              │
                    │   • Q ≤ 0.50       →  ABSTAIN           │
                    │   • Hysteresis dead-band & confirmation │
                    └────────────────────┬────────────────────┘
                                         │ Decision
                    ┌────────────────────┴────────────────────┐
                    ▼                                         ▼
          ┌───────────────────┐                     ┌───────────────────┐
          │  3. WRAPPED MODEL │                     │ 4. SQLITE LOGGER  │
          │ (Sklearn/PyTorch) │                     │ (Privacy-safe: no │
          │  Serve/Abstain    │                     │  raw PII by def)  │
          └───────────────────┘                     └─────────┬─────────┘
                                                              │
                                                              ▼
                                                    ┌───────────────────┐
                                                    │ 5. ORCHESTRATOR   │
                                                    │ • Volume checks   │
                                                    │ • Label gating    │
                                                    │ • Val. promotion  │
                                                    └─────────┬─────────┘
                                                              │
                                                              ▼
                                                    ┌───────────────────┐
                                                    │ 6. TERMINAL LOGS  │
                                                    │ (Real-time stdout │
                                                    │  telemetry & Q)   │
                                                    └───────────────────┘
```

---

## 3. Key Highlights & Features

- **⚡ Model-Agnostic Adapter**: Wraps any `scikit-learn` estimator, `PyTorch nn.Module`, or custom callable without modifying underlying model code.
- **📊 Interpretable Quality Score $Q$**: Combines Missingness, Population Stability Index (PSI), and Isolation Forest anomaly scores into a single metric $Q \in [0, 1]$.
- **🚦 3-Tier State Machine with Hysteresis**: Prevents state flapping near threshold boundaries with dead-bands and consecutive confirmation counts.
- **🔒 Privacy-Preserving Telemetry**: Logs statistical metrics to SQLite; disables raw feature logging by default to prevent PII exposure.
- **🔄 Validated Retraining & Rollback**: Enforces minimum data volume and verified label availability before retraining. Only promotes candidate models if validation accuracy beats active models.
- **💻 Terminal-First Telemetry & Inspection**: Real-time command-line logging of $Q$, gating decisions (`SERVE`/`FLAG`/`ABSTAIN`), sub-signal diagnostics, and model transition events directly in terminal output.

---

## 4. Installation & Setup

```bash
# Clone repository
git clone https://github.com/Pratham1227-beep/dqal.git
cd dqal

# Install in editable mode with all optional dependencies (visualization, PyTorch, dev)
pip install -e ".[all]"
```

---

## 5. Quickstart Example

### Wrapping a Scikit-Learn Model

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from dqal import DQAL, DQALConfig

# 1. Train your baseline model
data = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3, random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42).fit(X_train, y_train)

# 2. Wrap with DQAL and fit baseline distributions
# CRITICAL: Baseline distributions must be captured from clean training data!
dqal = DQAL(model=model, model_version="v1.0.0")
dqal.fit_baseline(X_train)

# 3. Predict on incoming batches
result = dqal.predict(X_test.iloc[:50])

print(f"Quality Score Q: {result.Q:.3f}")
print(f"Gating Decision: {result.decision}")  # SERVE, FLAG, or ABSTAIN
print(f"Sub-signals:     {result.signals}")
print(f"Predictions:     {result.predictions[:5]}")
```

### Wrapping a PyTorch Model

```python
import torch
import torch.nn as nn
from dqal import DQAL

torch_model = nn.Sequential(
    nn.Linear(30, 16),
    nn.ReLU(),
    nn.Linear(16, 2)
)

dqal = DQAL(model=torch_model, model_version="v1.0.0-torch")
dqal.fit_baseline(X_train)

result = dqal.predict(X_test.iloc[:20])
```

---

## 6. Configuration (`configs/default_config.yaml`)

```yaml
weights:
  missingness: 0.30
  drift: 0.40
  outlier: 0.30

thresholds:
  abstain_below: 0.50
  flag_below: 0.80

hysteresis:
  deadband: 0.03
  consecutive_batches: 2

logging:
  db_path: "dqal_telemetry.db"
  log_raw_features: false  # Opt-in only

retrain:
  min_flagged_batches: 5
  min_samples: 100
  require_labels: true
  min_improvement_delta: 0.01
```

---

## 7. Empirical Validation & Benchmarks

We evaluated DQAL across a progressive 60-batch degradation simulation (`benchmarks/run_benchmark.py`):
1. **Pristine Baseline** (Batches 0–11)
2. **Mild Covariate Shift** (Batches 12–23)
3. **Missing Fields Spike** (Batches 24–35)
4. **Severe Noise & Outliers** (Batches 36–47)
5. **Catastrophic Drift** (Batches 48–59)

### Benchmark Results

| Metric | Target | DQAL Result | Status |
| :--- | :--- | :--- | :--- |
| **Pearson Correlation ($r$)** | $r < -0.70$ or $> 0.70$ | **$r = 0.8502$** ($p = 8.45 \times 10^{-18}$) | Passed |
| **Inference Overhead** | $< 50$ ms / batch | **$49.79$ ms / batch** | Passed |
| **Early Warning Lead Time** | $> 0$ batches | **$9$ batches lead time** | Passed |
| **Label Safety Enforced** | $100\%$ | **Blocked unlabelled retraining** | Passed |

![DQAL Benchmark Results](benchmark_results.png)

---

## 8. Inspecting Telemetry in Terminal

DQAL logs structured, privacy-safe metrics per batch to SQLite (`dqal_telemetry.db`) and displays real-time gating decisions and sub-signals in your terminal:

```python
from dqal.logger import TelemetryLogger

# Inspect logged telemetry in terminal
logger = TelemetryLogger("dqal_telemetry.db")
df = logger.get_telemetry_df()
print(df[["batch_id", "Q", "decision", "missing_signal", "drift_signal", "outlier_signal"]].tail(10))
```

Or query directly using the SQLite CLI:

```bash
sqlite3 dqal_telemetry.db "SELECT batch_id, Q, decision, model_version FROM predictions_telemetry ORDER BY id DESC LIMIT 10;"
```

---

## 9. Running Tests

```bash
pytest -v
```

---

## 10. Known Limitations

- **Row vs. Batch Granularity**: Scoring single isolated rows has higher variance due to sample statistics. Batch mode ($\ge 30$ rows) provides robust distribution estimation.
- **Concept Drift vs. Anomaly**: DQAL detects statistical covariate and quality shift. Distinguishing permanent regime shift from temporary noise is surfaced to operators via the `FLAG` state rather than automated blind retraining.
