Metadata-Version: 2.4
Name: hashprep
Version: 0.1.0
Summary: A library for dataset quality checks, preprocessing, and report generation
Author-email: "Aftaab Siddiqui (MaskedSyntax)" <aftaab@aftaab.xyz>
License: MIT License
        
        Copyright (c) 2025 HashPrep
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/cachevector/hashprep
Project-URL: Repository, https://github.com/cachevector/hashprep
Project-URL: Documentation, https://github.com/cachevector/hashprep
Project-URL: Issues, https://github.com/cachevector/hashprep/issues
Keywords: data-science,machine-learning,eda,data-cleaning,profiling
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.3.0
Requires-Dist: fuzzybunny
Requires-Dist: jinja2>=3.1.6
Requires-Dist: matplotlib>=3.10.6
Requires-Dist: numpy>=2.2.6
Requires-Dist: pandas>=2.3.2
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: scipy>=1.15.3
Requires-Dist: seaborn>=0.13.2
Requires-Dist: fonttools>=4.60.2
Requires-Dist: tabulate>=0.9.0
Requires-Dist: weasyprint>=68.0
Dynamic: license-file

<div align="center">
  <picture>
    <img src="https://raw.githubusercontent.com/cachevector/hashprep/refs/heads/main/docs/assets/hashprep-wobg.svg" width="80">
  </picture>

  <h1>HashPrep</h1>
  <p>
    <b> Dataset Profiler & Debugger for Machine Learning </b>
  </p>

  <p align="center">
    <!-- Distribution -->
    <img src="https://img.shields.io/pypi/v/hashprep?color=blue&label=PyPI" />
    <!-- License -->
    <img src="https://img.shields.io/badge/License-MIT-green" />
    <img src="https://img.shields.io/badge/CLI-Supported-orange" />
    <!-- CI -->
    <a href="https://github.com/cachevector/hashprep/actions/workflows/ci.yml"><img src="https://github.com/cachevector/hashprep/actions/workflows/ci.yml/badge.svg" /></a>
  </p>
  <p>
    <!-- Features -->
    <img src="https://img.shields.io/badge/Feature-Dataset%20Quality%20Assurance-critical" />
    <img src="https://img.shields.io/badge/Feature-Preprocessing%20%2B%20Profiling-blueviolet" />
    <img src="https://img.shields.io/badge/Feature-Report%20Generation-3f4f75" />
    <img src="https://img.shields.io/badge/Feature-Quick%20Fixes-success" />
  </p>
</div>

> [!NOTE]
> HashPrep is now in its first stable release (v0.1.0). Core features are fully tested with CI.

## Overview

**HashPrep** is a Python library for intelligent dataset profiling and debugging that acts as a comprehensive pre-training quality assurance tool for machine learning projects.
Think of it as **"Pandas Profiling + PyLint for datasets"**, designed specifically for machine learning workflows.

It catches critical dataset issues before they derail your ML pipeline, explains the problems, and suggests context-aware fixes.  
If you want, HashPrep can even apply those fixes for you automatically.


---

## Features

Key features include:

- **Intelligent Profiling**: Detect missing values, skewed distributions, outliers, and data type inconsistencies.
- **ML-Specific Checks**: Identify data leakage, dataset drift, class imbalance, and high-cardinality features.
- **Automated Preparation**: Get suggestions for encoding, imputation, scaling, and transformations.
- **Rich Reporting**: Generate statistical summaries and exportable reports (HTML/PDF/Markdown/JSON) with embedded visualizations.
- **Production-Ready Pipelines**: Output reproducible cleaning and preprocessing code (`fixes.py`) that integrates seamlessly with ML workflows.
- **Modern Themes**: Choose between "Minimal" (professional) and "Neubrutalism" (bold) report styles.

---

## Installation

### Using pip
```bash
pip install hashprep
```

### Using uv (recommended)
```bash
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install hashprep
uv pip install hashprep

# Or for development from source
git clone https://github.com/cachevector/hashprep.git
cd hashprep
uv sync
```

After installation, the `hashprep` command will be available directly in your terminal.

---

## Usage

HashPrep can be used both as a **command-line tool** and as a **Python library**.

### CLI Usage

#### 1. Quick Scan
Get a quick summary of critical issues in your terminal.
```bash
hashprep scan dataset.csv
```

**Options:**
- `--critical-only`: Show only critical issues
- `--quiet`: Minimal output (counts only)
- `--json`: Output in JSON format
- `--target COLUMN`: Specify target column for ML-specific checks
- `--checks CHECKS`: Run specific checks (comma-separated)
- `--comparison FILE`: Compare with another dataset for drift detection
- `--sample-size N`: Limit analysis to N rows
- `--no-sample`: Disable automatic sampling
- `--config FILE`: Load thresholds from a YAML/TOML/JSON config file

**Example:**
```bash
# Scan with target column and specific checks
hashprep scan train.csv --target Survived --checks outliers,high_missing_values,class_imbalance

# Quick scan with JSON output
hashprep scan dataset.csv --json --quiet
```

#### 2. Detailed Analysis
Get comprehensive details about all detected issues.
```bash
hashprep details dataset.csv
```

**Options:** Same as `scan` command (including `--config`)

**Example:**
```bash
hashprep details train.csv --target Survived
```

#### 3. Generate Reports
Generate comprehensive reports in multiple formats with visualizations.
```bash
hashprep report dataset.csv --format html --theme minimal
```

**Options:**
- `--output PATH`, `-o PATH`: Custom output file path
- `--format {md,json,html,pdf}`: Report format (default: md)
- `--theme {minimal,neubrutalism}`: HTML report theme (default: minimal)
- `--with-code`: Generate Python scripts for fixes and pipelines
- `--full` / `--no-full`: Include/exclude full summaries (default: True)
- `--visualizations` / `--no-visualizations`: Include/exclude plots (default: True)
- `--target COLUMN`: Specify target column
- `--checks CHECKS`: Run specific checks
- `--comparison FILE`: Compare with another dataset for drift detection
- `--sample-size N`: Limit analysis to N rows
- `--no-sample`: Disable automatic sampling
- `--config FILE`: Load thresholds from a YAML/TOML/JSON config file

**Examples:**
```bash
# Generate HTML report with minimal theme
hashprep report dataset.csv --format html --theme minimal --full

# Generate PDF report without visualizations (faster)
hashprep report dataset.csv --format pdf --no-visualizations

# Generate report with automatic fix scripts
hashprep report dataset.csv --with-code

# Generate report with custom output path
hashprep report dataset.csv --format html --output my_reports/analysis.html

# This creates:
# - dataset_hashprep_report.md (or .html/.pdf/.json)
# - dataset_hashprep_report_fixes.py (pandas script)
# - dataset_hashprep_report_pipeline.py (sklearn pipeline)

# Compare two datasets for drift detection
hashprep report train.csv --comparison test.csv --format html
```

#### 4. List Available Checks
Discover all data quality checks that HashPrep can perform.
```bash
hashprep checks
```

#### 5. Version
Check HashPrep version.
```bash
hashprep version
```

#### Available Checks
- `outliers` - Detect outliers using z-score
- `duplicates` - Find duplicate rows
- `high_missing_values` - Columns with high missing data
- `empty_columns` - Completely empty columns
- `dataset_missingness` - Overall missing data patterns
- `missing_patterns` - Correlated missing value patterns
- `high_cardinality` - Categorical columns with too many unique values
- `single_value_columns` - Constant columns with no variance
- `mixed_data_types` - Columns with mixed data types
- `class_imbalance` - Imbalanced target variable (requires --target)
- `feature_correlation` - Highly correlated numeric features
- `categorical_correlation` - Highly associated categorical features
- `mixed_correlation` - Numeric-categorical associations
- `data_leakage` - Columns identical to target
- `target_leakage_patterns` - Features that may leak target information
- `dataset_drift` - Distribution drift between datasets (requires --comparison)
- `uniform_distribution` - Uniformly distributed numeric columns
- `unique_values` - Columns where >95% values are unique
- `high_zero_counts` - Columns with excessive zero values
- `skewness` - Highly skewed numeric distributions
- `infinite_values` - Columns containing infinite values
- `constant_length` - String columns with constant character length
- `extreme_text_lengths` - Text columns with extreme value lengths
- `datetime_skew` - Datetime columns concentrated in one period
- `datetime_future_dates` - Datetime columns with values in the future
- `datetime_gaps` - Anomalous gaps in datetime sequences
- `datetime_monotonicity` - Non-monotonic datetime columns
- `normality` - Non-normal numeric distributions (Shapiro-Wilk / D'Agostino-Pearson)
- `variance_homogeneity` - Unequal variances across target groups (Levene's test, requires --target)
- `low_mutual_information` - Features with near-zero mutual information with the target (requires --target)
- `empty_dataset` - Empty or all-missing datasets

---

### Python Library Usage

#### Basic Analysis
```python
import pandas as pd
from hashprep import DatasetAnalyzer

# Load your dataset
df = pd.read_csv("dataset.csv")

# Create analyzer
analyzer = DatasetAnalyzer(df)

# Run analysis
summary = analyzer.analyze()

# Access results
print(f"Critical issues: {summary['critical_count']}")
print(f"Warnings: {summary['warning_count']}")

# Iterate through issues
for issue in summary['issues']:
    print(f"{issue['severity']}: {issue['description']}")
```

#### Analysis with Target Column
```python
# Specify target for ML-specific checks
analyzer = DatasetAnalyzer(
    df,
    target_col='target_column'
)
summary = analyzer.analyze()
```

#### Run Specific Checks
```python
# Only run specific checks
analyzer = DatasetAnalyzer(
    df,
    selected_checks=['outliers', 'high_missing_values', 'class_imbalance']
)
summary = analyzer.analyze()
```

#### Include Visualizations
```python
# Generate analysis with plots
analyzer = DatasetAnalyzer(df, include_plots=True)
summary = analyzer.analyze()

# Plots are stored in summary['summaries']['plots']
```

#### Drift Detection
```python
# Compare two datasets
train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")

analyzer = DatasetAnalyzer(
    train_df,
    comparison_df=test_df,
    selected_checks=['dataset_drift']
)
summary = analyzer.analyze()
```

#### Generate Reports Programmatically
```python
from hashprep.reports import generate_report

# Analyze dataset
analyzer = DatasetAnalyzer(df, include_plots=True)
summary = analyzer.analyze()

# Generate HTML report
generate_report(
    summary,
    format='html',
    full=True,
    output_file='report.html',
    theme='minimal'
)

# Generate PDF report
generate_report(
    summary,
    format='pdf',
    full=True,
    output_file='report.pdf'
)

# Generate JSON report
generate_report(
    summary,
    format='json',
    full=True,
    output_file='report.json'
)

# Generate Markdown report
generate_report(
    summary,
    format='md',
    full=True,
    output_file='report.md'
)
```

#### Generate Fix Scripts
```python
from hashprep.checks.core import Issue
from hashprep.preparers.codegen import CodeGenerator
from hashprep.preparers.pipeline_builder import PipelineBuilder
from hashprep.preparers.suggestions import SuggestionProvider

# After running analysis
analyzer = DatasetAnalyzer(df, target_col='target')
summary = analyzer.analyze()

# Convert issues to proper format
issues = [Issue(**i) for i in summary['issues']]
column_types = summary.get('column_types', {})

# Get suggestions
provider = SuggestionProvider(
    issues=issues,
    column_types=column_types,
    target_col='target'
)
suggestions = provider.get_suggestions()

# Generate pandas fix script
codegen = CodeGenerator(suggestions)
fixes_code = codegen.generate_pandas_script()
with open('fixes.py', 'w') as f:
    f.write(fixes_code)

# Generate sklearn pipeline
builder = PipelineBuilder(suggestions)
pipeline_code = builder.generate_pipeline_code()
with open('pipeline.py', 'w') as f:
    f.write(pipeline_code)
```

#### Load Config from File
```python
from hashprep.utils.config_loader import load_config
from hashprep import DatasetAnalyzer

# Load thresholds from YAML, TOML, or JSON
config = load_config("hashprep.yaml")  # or .toml / .json

analyzer = DatasetAnalyzer(df, config=config)
summary = analyzer.analyze()
```

Example `hashprep.yaml`:
```yaml
missing_values:
  warning: 0.3
  critical: 0.6
outliers:
  z_score: 3.5
statistical_tests:
  normality_p_value: 0.01
```

Only the keys you specify are overridden; all others fall back to defaults.

#### Custom Sampling
```python
from hashprep.utils.sampling import SamplingConfig

# Configure sampling for large datasets
sampling_config = SamplingConfig(max_rows=10000)

analyzer = DatasetAnalyzer(
    df,
    sampling_config=sampling_config,
    auto_sample=True
)
summary = analyzer.analyze()

# Check if sampling occurred
if 'sampling_info' in summary:
    info = summary['sampling_info']
    print(f"Sampled: {info['sample_fraction']*100:.1f}%")
```

---

## License

This project is licensed under the [**MIT License**](./LICENSE).

---

## Contributing

We welcome contributions from the community to make HashPrep better!

Before you get started, please:

- Review our [CONTRIBUTING.md](./CONTRIBUTING.md) for detailed guidelines and setup instructions
- Write clean, well-documented code
- Follow best practices for the stack or component you’re working on
- Open a pull request (PR) with a clear description of your changes and motivation
