Metadata-Version: 2.4
Name: dd-parser-cleaner
Version: 0.4.4
Summary: A private, local LLM-powered data dictionary parser and entity mapper with automated cleaning.
Project-URL: Homepage, https://github.com/user/dd-parser-cleaner
Project-URL: Repository, https://github.com/user/dd-parser-cleaner
Author-email: Rajiv <rajiv@example.com>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27.0
Requires-Dist: ollama>=0.2.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: pypdf>=4.1.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: requests>=2.34.2
Requires-Dist: rich>=13.7.0
Requires-Dist: tabulate>=0.10.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Provides-Extra: notebook
Requires-Dist: ipykernel>=6.29.0; extra == 'notebook'
Requires-Dist: jupyter>=1.1.1; extra == 'notebook'
Requires-Dist: notebook>=7.5.6; extra == 'notebook'
Description-Content-Type: text/markdown

# dd-parser-cleaner: AI-Driven Data Preparation & Documentation

A specialized framework for automating data preparation documentation and preparing datasets for machine learning through AI-driven metadata discovery.

## 🌍 The Big Picture
KMDS is an initiative focused on developing documented, maintainable data science and ML projects using open-source tools and knowledge graphs. The `kmds-data-helper` was the first step in this journey—automatically building knowledge graphs from standard repository structures.

`dd-parser-cleaner` is a further refinement of this ecosystem. It targets **data preparation**, historically the most difficult and detail-loaded segment of any data science project. This tool provides:
1.  An **AI-driven framework** to generate comprehensive cleaning documentation.
2.  An **agent-driven interface** for developing datasets for ML featurization and analytics.

By capturing metadata during the cleaning phase, we create the foundation upon which future featurization logic is built.

## 🚀 Status & Roadmap
*   **Current State:** `dd-parser-cleaner` is feature-complete (v0.4.2).
*   **Short Term:** We will be releasing public examples of dataset migrations shortly.
*   **Future:** Development of specialized featurization modules for ML and analytics projects will begin once the migration examples are public.

## 📑 Documentation Strategy (Agent-First)
This project uses a **Markdown-Native documentation architecture** rather than traditional external sites.

*   **Why?** Keeping technical guides and design contracts as Markdown within the repo allows AI Agents (like your Migration Assistant) to "read" the documentation and provide better code suggestions.
*   **Where to look:** Human users should consult the `documents/` directory for methodology, and `USER_GUIDE.md` for quick-start instructions.

## 🛠️ Technical Constraints
*   **Offline First:** Optimized for batch processing without external streaming dependencies.
*   **Deterministic:** Ensures that running the same config on the same data yields the same results.
*   **Privacy-Centric:** All processing and LLM grounding (via local models) stay within your local environment.

## Core Capability Matrix

| Capability | Operational Impact |
| :--- | :--- |
| **AI Recommendations** | **Saves Hours:** Replaces manual data profiling with LLM-generated `cleaning_recommendations.md`. |
| **Clean Bucket Policy** | **De-risks Models:** Prevents "ghost" data and undocumented noise from leaking into ML training sets. |
| **Handshake Protocol** | **Audit-Ready:** Creates a formal, documented bridge between Raw Data and Logic Implementation. |
| **Agent Interface** | **AI-Native:** Designed for AI Assistants to autonomously implement complex, vectorized domain logic. |
| **Metadata Discovery API** | **Faster Featurization:** Programmatic access to semantic tags (Geographic, Risk, Financial) for ML pipelines. |

## ⚙️ Installation

### Standard Installation (CLI Only)
```bash
pip install dd-parser-cleaner
```

### Installation with Notebook Support (Migration Assistant)
```bash
pip install "dd-parser-cleaner[notebook]"
```

## 🚀 Quick Start

### 1. Classification (The Handshake)
Synchronize metadata and execute semantic classification:
```bash
classify-entities
```

### 2. Cleaning (The Pipeline)
Run the cleaner to apply types, filters, and transformations grounded in the parser's metadata:
```bash
uv run clean-dataset --action full --workspace ./tests
```

---
*For detailed documentation and custom logic implementation, see the `documents/` directory and `USER_GUIDE.md`.*