Metadata-Version: 2.4
Name: dqengine-py
Version: 0.1.1
Summary: A Python engine for profiling and analyzing data quality
Author-email: Rekik Samson <rekiksamson5@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/rcodes-ix/dqengine
Project-URL: Repository, https://github.com/rcodes-ix/dqengine
Project-URL: Issues, https://github.com/rcodes-ix/dqengine/issues
Keywords: data-quality,data-profiling,data-analysis,polars,data-validation
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: polars>=1.0
Dynamic: license-file

# dqengine-py

A Python engine for profiling and analyzing data quality.

## Overview

`dqengine-py` is a lightweight Python package for exploring datasets and evaluating their quality through automated profiling and reporting.

It is designed to make it easier to understand the structure and quality of a dataset before using it for analysis, machine learning, or other data-driven applications.

Built with Polars for fast DataFrame operations and designed to be extensible as the project grows.

## Features

- Dataset profiling with Polars
- Automatic data quality analysis
- Detection and analysis of missing data
- Column type analysis
- Numerical and categorical column analysis
- Constant column detection
- Duplicate row detection
- Duplicate column detection
- Inconsistent categorical value detection
- Date column detection
- Invalid date detection
- Outlier detection using the IQR method
- Correlation analysis
- Overall data quality scoring
- HTML quality reports
- JSON quality reports
- Simple Python API
- Designed to be extensible as the project grows

## Installation

Install the latest release from PyPI:

```bash
pip install dqengine-py
````

## Usage

Import `DatasetProfiler` from `dqengine`:

```python
from dqengine import DatasetProfiler
```

Create a profiler using the path to your CSV dataset:

```python
profiler = DatasetProfiler("data.csv")
```

Load and profile the dataset:

```python
profiler.load_csv()
results = profiler.profile()
```

The profiler analyzes the dataset and returns the results as a Python dictionary.

## Reports

`dqengine-py` supports generating data quality reports in different formats:

* Terminal output for quick inspection
* HTML for human-readable reports
* JSON for programmatic use

Example:

```python
from dqengine import DatasetProfiler
from dqengine.report import QualityReport


profiler = DatasetProfiler("data.csv")

profiler.load_csv()
results = profiler.profile()

report = QualityReport(results)

report.generate()
report.to_json("quality_report.json")
report.to_html("quality_report.html")
```

The generated files are:

```text
quality_report.html
quality_report.json
```

The HTML report provides a visual overview of the dataset's quality, while the JSON report is useful when the results need to be processed by another program.

## Example Workflow

A typical workflow looks like this:

```text
CSV Dataset
    |
    v
DatasetProfiler
    |
    v
Polars DataFrame
    |
    v
Data Quality Analysis
    |
    +------> Missing Data Analysis
    |
    +------> Column Analysis
    |
    +------> Duplicate Detection
    |
    +------> Category Analysis
    |
    +------> Date Analysis
    |
    +------> Outlier Detection
    |
    +------> Correlation Analysis
    |
    +------> Quality Score
    |
    v
QualityReport
    |
    +------> Terminal
    |
    +------> HTML
    |
    +------> JSON
```

## Analysis

The profiler currently analyzes several aspects of dataset quality.

### Missing Values

Detects missing values in each column and calculates the percentage of missing values.

### Duplicate Rows

Detects duplicate rows in the dataset and reports the total number found.

### Constant Columns

Identifies columns containing only one unique value.

### Inconsistent Categories

Detects categorical values that differ in formatting but represent the same normalized value.

For example:

```text
Addis Ababa
addis ababa
ADDIS ABABA
```

### Duplicate Columns

Detects columns containing identical data.

### Date Analysis

Attempts to identify date columns and detects invalid date values.

### Outliers

Detects numerical outliers using the Interquartile Range (IQR) method.

### Correlations

Identifies strongly correlated numerical column pairs.

### Quality Score

Calculates an overall data quality score based on detected issues.

The score is represented on a scale from:

```text
0 - 100
```

## Requirements

* Python 3.12+
* Polars

## Project Status

`dqengine-py` is currently in the early development stage.

The current `0.1.0` release focuses on establishing the core profiling and reporting functionality.

Future versions will expand the analysis capabilities, improve the reporting system, and introduce additional data quality checks.

## Development

Clone the repository:

```bash
git clone https://github.com/rcodes-ix/dqengine.git
cd dqengine
```

Create a virtual environment:

```bash
python -m venv .venv
```

Activate it on Linux/macOS:

```bash
source .venv/bin/activate
```

Install the project in editable mode:

```bash
pip install -e .
```

## Project Structure

```text
dqengine/
├── src/
│   └── dqengine/
│       ├── __init__.py
│       ├── profiler.py
│       └── report.py
├── .github/
│   └── workflows/
│       └── release.yml
├── .gitignore
├── LICENSE
├── README.md
└── pyproject.toml
```

## License

This project is licensed under the MIT License.

See the `LICENSE` file for more information.


