Metadata-Version: 2.3
Name: datadoc-cli
Version: 0.6.0
Summary: Leakage-safe, local-first preparation pipelines for tabular machine learning.
Project-URL: Homepage, https://github.com/narain-karti/DATADOC
Project-URL: Repository, https://github.com/narain-karti/DATADOC
Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
Author: narain-karti
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: plotext>=5.2.8
Requires-Dist: polars>=0.20.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.7.0
Requires-Dist: tomli>=2.0.0; python_version < '3.11'
Requires-Dist: typer>=0.12.0
Provides-Extra: ai
Requires-Dist: litellm>=1.40.0; extra == 'ai'
Provides-Extra: all
Requires-Dist: fastapi>=0.115.0; extra == 'all'
Requires-Dist: litellm>=1.40.0; extra == 'all'
Requires-Dist: python-multipart>=0.0.20; extra == 'all'
Requires-Dist: scikit-learn>=1.7.2; extra == 'all'
Requires-Dist: uvicorn>=0.30.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == 'dev'
Requires-Dist: hatchling>=1.25.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: scikit-learn>=1.7.2; extra == 'dev'
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.7.2; extra == 'ml'
Provides-Extra: ui
Requires-Dist: fastapi>=0.115.0; extra == 'ui'
Requires-Dist: python-multipart>=0.0.20; extra == 'ui'
Requires-Dist: uvicorn>=0.30.0; extra == 'ui'
Description-Content-Type: text/markdown

<p align="center">
  <pre align="center">
 ____    _  _____  _    ____   ___   ____
|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
| | | |/ _ \ | | / _ \ | | | | | | | |
| |_| / ___ \| |/ ___ \| |_| | |_| | |___
|____/_/   \_\_/_/   \_\____/ \___/ \____|
  </pre>
</p>

<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>

<p align="center">
  <a href="https://narain-karti.github.io/DATADOC/"><b>&#x1F4D6; View Official Documentation Website</b></a>
</p>

<p align="center">
  <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
  <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
  <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
</p>

<p align="center">
  <a href="#installation"><b>Install</b></a> &bull;
  <a href="#why-datadoc"><b>Why DATADOC?</b></a> &bull;
  <a href="#quick-start"><b>Quick Start</b></a> &bull;
  <a href="#cli-commands"><b>CLI Commands</b></a> &bull;
  <a href="#architecture--plugins"><b>Architecture</b></a>
</p>

<hr>

## 🚀 What is DATADOC?

**DATADOC** is a local-first CLI and Python library for preparing tabular data for machine learning. It profiles dataset risks, creates explainable transformation plans, and saves fitted pipelines that apply the same training-derived rules to validation, test, and inference data.

Powered by **Polars**, DATADOC reads CSV and Parquet files, diagnoses missing values, identifiers, schema issues, duplicates, constants, and unsafe feature types. It does not promise model improvement: optional evaluation reports the observed result against a baseline under a reproducible split.

**DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It profiles data quality, lets you review a plan, fits transformations from training data, and hands you a portable artifact and Python wrapper for reuse.

### ⚡ The Impact: Why Industry Professionals Use DATADOC

Data Scientists and ML Engineers repeatedly rebuild the same preparation steps across projects.
DATADOC turns those steps into a reviewable, reusable pipeline.

- **Save boilerplate:** Review recommendations for imputing nulls, encoding categories, and optional scaling or clipping before applying them.
- **Explainable by default:** The deterministic core records roles, findings, operations, protected columns, and fitted statistics in an inspectable artifact.
- **Local-first:** The core package works offline. Optional ML, UI, and AI features are separate extras.
- **Optional AI planning:** AI can help explain or rank a constrained plan; it is never allowed to execute arbitrary generated code.
- **Leakage-safe workflows:** Fitted statistics for imputation, categorical vocabularies, clipping, and scaling are learned from training data and saved as an artifact.

### 📊 Empirical Proof: Does DATADOC Actually Improve Models?

Yes. We benchmarked naive manual preparation vs. DATADOC automated preprocessing on Kaggle's Titanic dataset under **5-fold Stratified Cross-Validation**:

| Model | Baseline (Naive Prep) | DATADOC Cleaned | Accuracy Δ | Relative Lift |
|:---|:---:|:---:|:---:|:---:|
| **Logistic Regression** | 78.90% ± 0.99% | **79.91% ± 1.90%** | **+1.01%** | **+1.28%** |
| **Random Forest** | 82.15% ± 2.45% | **82.82% ± 2.40%** | **+0.67%** | **+0.82%** |

*Why?* DATADOC extracts informative missingness indicators (`Age__missing`, `Cabin__missing`), frequency-encodes high-cardinality features (`Ticket`, `Cabin`), applies standard scaling, and isolates all statistics strictly to training splits to eliminate target leakage.

---

## 📦 Installation

DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:

```bash
pip install datadoc-cli
```

*(Requires Python 3.10+)*

---

## 🛠️ Quick Start (CLI)

You don't need to write a single line of Python to clean your data. Just use the CLI.
New here? Run the guided wizard — it asks for target + preset and runs everything:

```bash
datadoc wizard train.csv
```

Or run the one-shot happy path (profile → plan → fit → transform + manifest):

```bash
datadoc run train.csv --target churn --preset balanced --evaluate
```

Full step-by-step (auditable) workflow:

```bash
# 0. Optional: save repeatable settings (target, preset, scaling, ...)
datadoc init --preset balanced  # writes datadoc.toml

# 1. Inspect data-quality findings and column roles
datadoc profile raw_data.csv --target churn --explain --output profile.json

# 2. Review the proposed transformations before applying them
datadoc plan raw_data.csv --target churn --explain --output plan.json

# 3. Fit only on a training dataset, then save a reusable artifact
datadoc fit train.csv --target churn --preset balanced --rare-frequency 0.02 --output artifacts/churn-pipeline.json

# 4. Apply the fitted artifact to validation, test, or new data
datadoc transform validation.csv --pipeline artifacts/churn-pipeline.json --output validation-features.parquet --validate

# 5. Optionally benchmark a safe candidate pipeline against a baseline
pip install "datadoc-cli[ml]"
datadoc evaluate train.csv --target churn --task classification --ablation

# 6. Export a small executable wrapper around the fitted artifact
datadoc export --pipeline artifacts/churn-pipeline.json --output pipeline.py
# or: datadoc export --pipeline artifacts/churn-pipeline.json --format joblib --output pipeline.joblib

# 7. Generate a standalone, shareable HTML audit report
datadoc report train.csv --target churn --output report.html

# 8. Visually compare raw vs transformed datasets side-by-side
datadoc compare train.csv validation-features.parquet --target churn --html compare.html

# 9. Lint for leakage risks / diff two plans
datadoc lint train.csv --target churn
datadoc diff plan-v1.json plan-v2.json
```

### 🖥️ Web dashboard (same pipeline, visual)

```bash
pip install "datadoc-cli[ui]"
datadoc ui train.csv --port 8000
```

The local dashboard calls the same `DataDocPipeline` behind the CLI: profile findings and roles, preparation settings (target, scaling, identifiers, dedup, clipping, cyclical datetime, rare frequency), reviewable plan, fit with output-schema preview, lineage/provenance panel, transformed-CSV download, and an executable Python export. Press `Ctrl+K`/`Cmd+K` for the command palette. Full guide: [docs/ui.html](https://narain-karti.github.io/DATADOC/ui.html).

---

## 🐍 Python SDK (Library Usage)

DATADOC is also a Python library. The stable workflow is `profile → plan → fit → transform`; the same fitted artifact can be used in notebooks, services, and batch jobs:

```python
from datadoc import DataDocPipeline, PipelineConfig
import polars as pl

# Fit only on the training split. The target is protected from feature transforms.
train_df = pl.read_csv("train.csv")
pipeline = DataDocPipeline(PipelineConfig(target="churn")).fit(train_df)
pipeline.save("artifacts/churn-pipeline.json")

# Transform data that was never used to fit statistics.
validation_df = pl.read_csv("validation.csv")
validation_features = pipeline.transform(validation_df)
```

For an observed model comparison, install the optional ML extra and call `pipeline.evaluate(train_df)` or `datadoc evaluate`. Evaluation is evidence for the declared task and split strategy; it is not a promise that cleaning always improves a model.

---

## 💻 CLI Commands Reference

| Command | Description |
|---------|-------------|
| `datadoc wizard <file>` | Guided TUI: asks target/preset/scaling, writes `datadoc.toml`, runs pipeline |
| `datadoc init` | Writes a starter `datadoc.toml` (or `pyproject.toml [tool.datadoc]`) config |
| `datadoc profile <file>` | Data-quality report + roles (`--explain`, `--compare profile2.json`) |
| `datadoc plan <file>` | Explainable plan (`--explain`, `--diff plan2.json`) |
| `datadoc fit <train>` | Learns pipeline on train only (`--preset`, `--deduplicate`, `--rare-frequency`, `--cyclical`, `--no-hour`, repeatable `--identifier-column` / `--ignore-column`) |
| `datadoc transform <file>` | Applies saved artifact (`--validate` for schema + drift checks) |
| `datadoc evaluate <file>` | Candidate vs baseline (`--ablation` for per-component deltas) |
| `datadoc export` | Wrapper for artifact (`--format python\|joblib`) |
| `datadoc run <file>` | One-shot profile→plan→fit→transform + `manifest.json` (+ `--evaluate --ablation`) |
| `datadoc report <file>` | Generates a standalone, shareable HTML data health and preparation audit report |
| `datadoc compare <raw> <trans>` | Visual side-by-side dataset comparison (terminal table + HTML export) |
| `datadoc lint <file>` | Leakage/pitfall lint (target duplication, nulls, infinities, duplicates) |
| `datadoc diff <a.json> <b.json>` | Diff profile/plan/pipeline artifacts |
| `datadoc plugins list` | Lists 9 registered plugins (priorities, entry-points) |
| `datadoc ui <file>` | Local FastAPI dashboard (Ctrl+K palette, lineage panel, HTML report export) |

Short aliases: `-t/--target`, `-o/--output`, `-p/--pipeline`, `-f/--format`.
Presets: `--preset quick|balanced|linear|tree|time|robust`. Shell completion: `datadoc --install-completion`.



---

## 🧩 Architecture & Plugins

DATADOC operates as a fitted pipeline. Every transformation learns state only from training data, saves that state to JSON, and reuses it unchanged for later datasets.

| Priority | Plugin | Action Performed |
|----------|--------|-------------|
| 5 | **DuplicateRemoverPlugin** | Detects duplicate rows; `deduplicate=True` drops them at fit (train-only) |
| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
| 20 | **OutlierPlugin** | Offers optional IQR clipping; clipping is not forced by default |
| 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week (+hour when time present, optional cyclical sin/cos) |
| 40 | **CategoricalEncoderPlugin** | Encodes categories using training vocabularies (threshold 20) and handles unseen values |
| 41 | **TargetEncoderPlugin** | Empirical Bayes smoothed target encoding: `(n * cat_mean + m * global_mean) / (n + m)` |
| 42 | **RareCategoryPlugin** | Groups rare categories (< `rare_category_min_frequency`) into `__RARE__` |
| 44 | **PolynomialFeaturesPlugin** | Generates degree-2 interaction terms (`x1 * x2`) and squared terms (`x^2`) |
| 45 | **ScalingPlugin** | Applies configured standard or robust scaling, fit on training data only |

The fitted pipeline is the production source of truth. Plugin work should follow the lifecycle `analyze → recommend → apply`, with fitted state (`median`, `clip`, `vocabularies`, `rare maps`, `hour flags`, `center/spread`) serializable in `pipeline.json` (artifact v2 with `provenance`). External plugins auto-register via `datadoc.plugins` entry-points.

Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!

---

## 🗺️ Roadmap

- [x] Core Engine with plugin orchestration
- [x] 9 Built-in deterministic plugins (duplicate, missing, outlier, datetime, encoder, target encoder, rare, polynomial, scaling)
- [x] Stunning Rich Terminal UI (wizard, presets, `datadoc.toml`, completion)
- [x] Pipeline export capability (`python` + `joblib`)
- [x] Polars backend and local-first pipeline artifacts (v2 + provenance)
- [x] PyPI Release (`pip install datadoc-cli`)
- [x] Constrained optional AI planning path
- [x] Session-scoped local FastAPI dashboard (Ctrl+K palette, lineage, drift)
- [x] Addictive loop: `profile --compare`, `plan --explain/--diff`, `transform --validate`, `evaluate --ablation`
- [x] Standalone HTML audit reports (`datadoc report`)
- [x] Visual dataset comparison engine (`datadoc compare`)
- [x] Notebook widgets (`profile_to_html`, `_repr_html_`)
- [ ] Export targets for `dbt` and Apache Airflow
- [x] Local FastAPI dashboard/API companion

---

## ⚖️ License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

## 🤝 Contributing

We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).

See [CHANGELOG.md](CHANGELOG.md) for the 0.6.0 release notes.
