Metadata-Version: 2.4
Name: dd-parser-cleaner
Version: 0.4.2
Summary: A private, local LLM-powered data dictionary parser and entity mapper with automated cleaning.
Project-URL: Homepage, https://github.com/user/dd-parser-cleaner
Project-URL: Repository, https://github.com/user/dd-parser-cleaner
Author-email: Rajiv <rajiv@example.com>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27.0
Requires-Dist: ollama>=0.2.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: pypdf>=4.1.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: requests>=2.34.2
Requires-Dist: rich>=13.7.0
Requires-Dist: tabulate>=0.10.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# dd-parser-cleaner: Enterprise Pipeline Governance Engine

An offline metadata parsing and pipeline governance engine that enforces data provenance and automated schema serialization at the ingestion boundary.

## Technical & Architectural Constraints
This system is built under strict architectural constraints to ensure stability in production enterprise environments:
* **Zero Streaming Footprint:** Exclusively optimized for offline, design-time, and batch processing pipelines.
* **Deterministic Execution:** Operates as a stateless execution wrapper over data ingestion blocks.
* **No Telemetry Leakage:** All metadata parsing, validation, and serialization occur entirely within your closed local or cloud perimeter.

## Executive Summary
dd-parser-cleaner eliminates pipeline technical debt by intercepting batch data transfers and programmatically locking down data state, lineage, and structural metadata. It converts runtime data execution into audit-ready JSON/Markdown documentation, guaranteeing absolute reproducibility for downstream batch optimization matrices. This architecture provides **significant time savings for Data Science and ML teams** by automating the most fragile link in the analytical chain: data preparation and semantic alignment.

## Core Capability Matrix

| Capability | Operational Impact |
| :--- | :--- |
| **Deterministic State Capture** | Automatically serializes dataset shapes, cryptographic hashes, data types, and ingestion timestamps to prevent downstream model drift. |
| **Zero-Overhead Schema Extraction** | Generates machine-readable JSON metadata payloads directly from batch dataframes, decoupling physical schema properties from pipeline code. |
| **Automated Pipeline Lineage** | Compiles runtime execution state into standardized, human-readable Markdown asset logs for enterprise compliance reviews. |
| **Strict Schema Integrity** | Enforces a "Clean Bucket" policy via Integrity Sync, purging undocumented columns to ensure 1:1 semantic mapping. |
| **Metadata Discovery API** | Provides a programmatic interface for notebooks to query semantic tags, enabling seamless integration with ML pipelines. |

## 🚀 Quick Start

### 1. Classification (The Handshake)
Synchronize metadata and execute semantic classification:
```bash
classify-entities
```

### 2. Cleaning (The Pipeline)
Run the cleaner to apply types, filters, and transformations grounded in the parser's metadata:
```bash
uv run clean-dataset --action full --workspace ./tests
```

---
*For detailed documentation and custom logic implementation, see the `documents/` directory and `USER_GUIDE.md`.*