Metadata-Version: 2.4
Name: datacleanr
Version: 0.1.1
Summary: Clean messy real-world datasets with safe, explainable defaults before ML.
Author: Ishwar Soni
License: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: numpy
Dynamic: license-file

# datacleaner

Clean messy real-world datasets with safe, explainable defaults before ML.

## Overview

datacleaner is a practical data-cleaning library for tabular datasets in pandas.
It is designed for ML practitioners, students, and data engineers who need a reliable baseline cleaning pipeline.
The library emphasizes safety, transparent decisions, and clear diagnostics so you can inspect what changed and why.

## Key Features

- Missing value handling with smart fill and high-missing column dropping.
- Duplicate removal for exact duplicate rows.
- Datatype correction with numeric and datetime detection.
- Text standardization (lowercase, trim, and common value normalization).
- Outlier handling with `cap` or `remove` modes, plus capping diagnostics.
- Column selection for constant columns and likely ID-like columns.
- Correlation reduction for highly correlated numeric features.
- Safety mode to prevent excessive row or column loss.
- Structured diagnostics reporting for explainable cleaning.

## Quick Start

Minimal usage:

```python
from datacleaner import clean

df = clean(df)
```

Advanced usage with report and verbose logs:

```python
from datacleaner import clean

df, report = clean(df, return_report=True, verbose=True)
```

## Example Output

```text
Missing values: dropped 1 columns, filled 99 values.
Duplicates: removed 5 rows.
Outliers (cap): capped 12 values.
Column selection: dropped 1 low-information columns.

Diagnostics:
- numeric_candidate_columns: ['mixed_numeric']
- max_correlation: 0.48
- high_outlier_capping_columns: []
```

## Diagnostics and Safety

- `numeric_candidate` detection highlights text columns that are mostly numeric-like.
- Outlier diagnostics flag columns with high capping rates to avoid silent over-correction.
- Correlation diagnostics report max observed correlation and number of pairs checked.
- `safe_mode=True` warns when data loss is high and can roll back aggressive steps.

## When to Use / When Not to Use

Use datacleaner when:

- You are working with messy real-world tabular datasets.
- You need a quick, safe baseline cleaning pass before modeling.

Do not use datacleaner when:

- You need full ML preprocessing (for example scaling, encoding, or feature engineering).
- Your workflow requires domain-specific cleaning rules that cannot be generalized.

## Installation

```bash
pip install datacleaner
```

## Testing

The library has been validated using:

- synthetic stress tests
- sklearn datasets
- edge-case scenarios

Run tests:

```bash
python tests/test_sklearn_datasets.py
```

## Project Structure

```text
datacleaner/
	src/datacleaner/
		core.py
		analysis.py
		missing_values.py
		duplicates.py
		datatypes.py
		text_standardization.py
		outliers.py
		column_selection.py
		correlation_reduction.py
		reporting.py
	tests/
	examples/
```

## Contributing

Contributions are welcome.
Please open an issue for bug reports or design proposals, and submit a pull request for fixes and improvements.
