Metadata-Version: 2.4
Name: dd-parser-cleaner
Version: 0.9.0
Summary: A private, local LLM-powered data dictionary parser and entity mapper with automated cleaning.
Project-URL: Homepage, https://github.com/user/dd-parser-cleaner
Project-URL: Repository, https://github.com/user/dd-parser-cleaner
Author-email: Rajiv <rajiv.sambasivan@r2-ds.com>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27.0
Requires-Dist: ollama>=0.2.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: pypdf>=4.1.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: requests>=2.34.2
Requires-Dist: rich>=13.7.0
Requires-Dist: tabulate>=0.10.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Provides-Extra: notebook
Requires-Dist: ipykernel>=6.29.0; extra == 'notebook'
Requires-Dist: jupyter>=1.1.1; extra == 'notebook'
Requires-Dist: notebook>=7.5.6; extra == 'notebook'
Description-Content-Type: text/markdown

# dd-parser-cleaner

**One-line summary**  
`dd-parser-cleaner` inspects incoming datasets, emits validated manifests describing structure and modalities, runs deterministic integrity checks, and writes a handshake file that downstream featurizers must read before transforming data.

## Purpose

This package provides discovery and validation for enterprise datasets. It detects dataset type (cross-sectional, event-log, panel, homogeneous/bipartite/heterogeneous graph), tags attributes with roles and modalities, validates keys and joins, and produces actionable diagnostics and remediation hints. The canonical outputs are **dataset manifest**, **attribute manifest**, and **handshake.json**.

## Quick start (workflow)

1. **Discover package capabilities**

```python
from dd_parser_cleaner import get_package_info
info = get_package_info()
```

2. **Run the parser** (CLI or API) to produce:

- `manifests/<dataset_id>.json` (dataset manifest)
- `attributes/<dataset_id>_attributes.json` (attribute manifest)

3. **Run the cleaner** to validate manifests and produce:

- `manifests/handshake.json`

4. **Featurizer** must read `manifests/handshake.json` and proceed only if `status == "ready"`.

## Key capabilities

- **Dataset discovery**: auto-detects `dataset_type` and primary/time keys.
- **Attribute tagging**: emits `role`, `time_dependency`, `granularity`, `modality`, `suggested_checks`, `generated_key_flag`.
- **Graph support**: homogeneous, bipartite, heterogeneous graphs with entity/relationship maps.
- **Longitudinal support**: event-log vs panel; static vs dynamic attributes.
- **Manifest emission**: canonical JSON manifests for downstream deterministic featurization.
- **Cleaner validations**: monotonicity, lag consistency, cycle detection, relation consistency, URL/geo sanity checks.
- **Handshake contract**: `handshake.json` with `status` (`ready` | `blocked` | `warnings`).
- **Config driven**: behavior controlled by `config.yaml` flags.

## Example artifacts

### Example dataset manifest (snippet)

```json
{
  "dataset_id": "orders_2026",
  "dataset_type": "event_log",
  "primary_key_spec": ["order_id"],
  "time_key_spec": "event_time",
  "entity_files": [],
  "relation_files": [],
  "panel_variable_map": null,
  "notes": "Order events from e-commerce pipeline",
  "validation_errors": []
}
```

### Example attribute manifest entry

```json
{
  "attribute_name": "order_id",
  "role": "subject_key",
  "time_dependency": "none",
  "granularity": null,
  "modality": "categorical",
  "suggested_checks": ["null_profile"],
  "generated_key_flag": false
}
```

### Example handshake.json

```json
{
  "status": "ready",
  "manifest_path": "manifests/orders_2026.json",
  "blocking_reasons": []
}
```

## Where to find schemas and examples

- **JSON Schema files** (manifest validation): `schemas/dataset_manifest.json`, `schemas/attribute_manifest.json`, `schemas/handshake.json`
- **Sample manifests and fixtures**: `tests/fixtures/manifests/` and `tests/fixtures/csvs/`
- **Docs and design**: `USER_GUIDE.md`, `documents/`, and `docs/manifest.md`

## Important config flags (defaults)

Add or review these in `config.yaml` under a `manifest` section:

```yaml
manifest:
  require_manifest_before_featurize: true
  use_case_questions_enabled: false
  graph_entity_limit: 5
  generate_surrogate_keys: true
  url_sample_size: 10
```

## Handshake contract (featurizer requirements)

- Featurizer **must** read `manifests/handshake.json` before any transformation.
- If `status == "blocked"`, the featurizer must refuse to proceed.
- If `status == "warnings"`, the featurizer may proceed only after acknowledging and recording the warnings.

## Migration and compatibility

- New manifest fields are **additive** and optional. Existing cross-sectional outputs remain unchanged during phased rollout.
- Recommended phased rollout:

1. Emit manifests and handshake while preserving legacy outputs.
2. Enable cleaner validators and handshake enforcement behind config flags.
3. Deprecate legacy outputs after one release cycle.

## Troubleshooting (common validation failures)

- **Missing primary key**: parser will generate a surrogate key and set `generated_key_flag`; prefer providing explicit keys.
- **Time key absent for longitudinal data**: set `time_key_spec` or mark dataset as `cross_sectional`.
- **Relation file join mismatch**: ensure `entity_key_spec` matches keys referenced in relation files.
- **Heterogeneous graph cycle detected**: convert to acyclic tree or correct relationship files.
- **Invalid URLs or geo addresses**: check `modality` tags and sample rows flagged in diagnostics.

Each validation error includes `severity`, `remediation`, and `sample_rows` in the cleaner report.

## How clients and agents should use `get_package_info()`

Use `get_package_info()` to discover:

- CLI commands and entry points
- `manifest_schema_paths` for validation
- `handshake_spec` and allowed `status` values
- `supported_dataset_types` and important `config_flags`

Treat `get_package_info()` as the canonical programmatic discovery endpoint.

## Support and contribution

- **Issue tracker**: add issues at the repository issue tracker (link in `get_package_info()` output).
- **Contributing**: follow repository CONTRIBUTING.md for tests, fixtures, and schema updates.
- **Contact**: open an issue for integration questions or schema clarifications.

## One-line blurb for top-level README

`dd-parser-cleaner` inspects datasets, emits validated manifests and a handshake file describing keys, time semantics, modalities, and graph structure, and provides deterministic diagnostics so downstream featurizers can safely and reproducibly transform data.

## Existing quick links

- `USER_GUIDE.md` for usage details
- `documents/` for methodology and internal design notes
- `tests/notebooks/` for example notebook workflows
