Metadata-Version: 2.4
Name: pydqkit
Version: 0.0.1
Summary: A developer-first Python toolkit for data quality profiling, validation, and interactive HTML reports.
Project-URL: Documentation, https://github.com/Niki9001/pydqkit#readme
Project-URL: Issues, https://github.com/Niki9001/pydqkit/issues
Project-URL: Source, https://github.com/Niki9001/pydqkit
Author-email: Niki Zheng <nikizheng90@gmail.com>, Luqun Li <success@shnu.edu.cn>
License: MIT License
        
        Copyright (c) 2026-present Niki Zheng <nikizheng90@gmail.com>
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE
Keywords: data engineering,data profiling,data quality,data validation,data-engineering,data-profiling,data-quality,data-validation,etl,pandas,sanity check,sanity-check
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Requires-Dist: pandas>=1.5
Description-Content-Type: text/markdown

# pydqkit

**pydqkit** is a lightweight, developer-first Python toolkit for **data quality profiling, data validation, sanity check and interactive HTML visualization**.

It helps data engineers and analysts quickly understand the structure, completeness, and patterns of tabular datasets, without requiring any external platforms or services.

---

## Features

- Column-level data profiling
  - Null and non-null statistics
  - Distinct and duplicate counts
  - Length analysis (minimum, maximum, average)
- Pattern discovery for string, datetime, and boolean columns
- Type inference (numeric, string, boolean, datetime)
- Interactive, self-contained HTML profiling reports
- Designed for exploratory analysis and debugging workflows

---

## Installation

```bash
pip install pydqkit
```

---

## Quick Start

```python
import pandas as pd
from pydqkit.profiling import profile_dataframe
from pydqkit.viz import profile_to_html

df = pd.DataFrame({
    "id": ["AB123456", "CD654321", None, "EF000001"],
    "age": [25, 30, None, 40],
    "score": [88.5, 92.0, 79.5, 85.0],
    "date": ["2025-01-01", "2025-01-02", None, "2025-01-04"],
    "flag": [True, True, False, True],
})

profile = profile_dataframe(df, dataset_name="demo")

html = profile_to_html(profile)
with open("profile_report.html", "w", encoding="utf-8") as f:
    f.write(html)
```

Open `profile_report.html` in your browser to explore the interactive profiling report.

---

## What the Profiling Report Shows

For each column, the report includes:

### Completeness
- Proportion of non-null and null values (visual bar and percentages)

### Value Statistics
- Distinct count and duplicate count

### Type Information
- Inferred profile type and suggested logical type

### Length Metrics
- Minimum length (reported as 0 if missing values exist)
- Maximum and average length

### Pattern Summary
- Common structural patterns inferred from values

### Top Values
- Most frequent values with counts and percentages

The report is designed to be readable at a glance while still exposing enough detail for debugging and validation.

---

## Design Philosophy

pydqkit is intentionally:

### Developer-first
Optimized for notebooks, scripts, and local inspection.

### Platform-independent
No cloud services, no accounts, no metadata dependencies.

### Explainable
Metrics and visual elements are designed to be interpretable rather than opaque.

The project follows conventions commonly used in enterprise data quality tools, but is implemented as a standalone Python library.

---

## License

This project is released under the **MIT License**.  
See the [LICENSE](LICENSE) file for details.

The author reserves the right to relicense future versions.

---

## Disclaimer

This project is an independent open-source toolkit and is **not affiliated with, endorsed by, or associated with Informatica or any other commercial data quality platform**.

---

## Contributing

Contributions are welcome.

By submitting a pull request, you agree that your contributions will be licensed under the same license as this project.

See [CONTRIBUTING.md](CONTRIBUTING.md) for details.

---

## Roadmap (Tentative)

- Rule-based data quality checks
- Column-level and cell-level validation
- Support for multiple rule definitions:
  - Regular expressions
  - SQL-based rules
  - Plain-language rule specifications
- Data engineering quality checks (schema, type, and pipeline sanity)
- Machine learning data sanity checks (distribution drift, outliers)
- Column comparison across datasets
- Cell level data quality check
- Export profiling reports to multiple formats (HTML, PDF, Excel)
- Configurable thresholds and validation summaries

---

## Author

**Niki Zheng,** **Luqun Li**
