Metadata-Version: 2.4
Name: smartclean-ai
Version: 1.2.0
Summary: An intelligent Python library for automated structured-data cleaning, profiling, outlier handling, and reporting.
Author-email: Yash Talati <yashtalati07@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/yashtalati10/SmartClean
Project-URL: Repository, https://github.com/yashtalati10/SmartClean
Project-URL: Issues, https://github.com/yashtalati10/SmartClean/issues
Keywords: data-cleaning,automated-data-cleaning,data-quality,data-profiling,exploratory-data-analysis,eda,missing-values,outlier-detection,outlier-handling,pandas,numpy,machine-learning,preprocessing,csv,excel,python
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.0.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: scikit-learn>=1.3.0
Dynamic: license-file

# SmartClean-AI

**Automated Data Cleaning, Profiling, and Reporting for Python**

SmartClean-AI is a Python library designed to simplify structured-data cleaning and exploratory data analysis (EDA). It helps identify and handle common data-quality problems in CSV and Excel datasets, including missing values, columns with excessive missing data, and outliers.

The library provides a configurable cleaning workflow, a cleaned dataset, and reports that help users understand the changes made during preprocessing.

- **PyPI:** https://pypi.org/project/smartclean-ai/
- **GitHub:** https://github.com/yashtalati10/SmartClean
- **Package name:** `smartclean-ai`
- **Python import:** `smartclean`
- **Documented version:** `1.2.0` — update this to match the latest published release.

---

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [How It Works](#how-it-works)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Command-Line Workflow](#command-line-workflow)
- [Configuration](#configuration)
- [Output Files](#output-files)
- [Dashboard Preview](#dashboard-preview)
- [Example JSON Report](#example-json-report)
- [Example TXT Report](#example-txt-report)
- [Using the Python API](#using-the-python-api)
- [Project Structure](#project-structure)
- [Dependencies](#dependencies)
- [Limitations and Notes](#limitations-and-notes)
- [Contributing](#contributing)
- [License](#license)

---

## Overview

Data preparation is an important step in data analysis and machine learning. Real-world datasets can contain missing values, redundant records, inconsistent data, and extreme values that affect downstream analysis.

SmartClean-AI aims to make this process easier by providing reusable cleaning components and reporting capabilities.

The project focuses on:

- Detecting and handling missing values.
- Identifying columns with excessive missing data.
- Applying outlier-handling techniques.
- Saving cleaned datasets.
- Producing structured JSON and human-readable TXT reports.
- Generating a cleaning dashboard for reviewing results.

## Features

### Data Cleaning

- Automated missing-value handling.
- Configurable threshold for dropping columns with excessive missing data.
- Outlier handling with configurable methods supported by the implementation.
- CSV and Excel input support through the data-loading component.
- Reusable cleaning components.

### Reporting and Analysis

- Before-and-after dataset dimensions.
- Missing-value statistics.
- Number of columns dropped.
- Outlier-handling statistics.
- Detailed dropped-column information in reports.
- JSON report for programmatic access.
- TXT report for human-readable results.
- Dashboard image for visual inspection.
- Concise terminal summary.

### Developer Experience

- Python-based workflow.
- Modular package design.
- Configurable cleaning parameters.
- Suitable for data preparation and preprocessing workflows.

> **Note:** Duplicate removal, specific imputation strategies, and individual outlier-removal modes should only be advertised as supported options when they are implemented and verified in the published package version.

---

## How It Works

The structured-data cleaning workflow follows these general stages:

```text
             Input Dataset
                   |
                   v
             Data Loading
                   |
                   v
          Initial Data Inspection
                   |
                   v
          Missing-Value Handling
                   |
                   v
        Excess-Missing Column Check
                   |
                   v
           Outlier Handling
                   |
                   v
         Save Cleaned Dataset
                   |
          +--------+--------+
          |        |        |
          v        v        v
        JSON      TXT    Dashboard
        Report   Report    Image
                   |
                   v
           Cleaning Summary
```

The precise behavior depends on the selected configuration and the cleaning components available in the installed version.

---

## Installation

Install the published package from PyPI:

```bash
python -m pip install --upgrade smartclean-ai
```

Verify that the package can be imported:

```bash
python -c "import smartclean; print('SmartClean-AI import successful')"
```

### Install from GitHub

```bash
git clone https://github.com/yashtalati10/SmartClean.git
cd SmartClean
python -m pip install .
```

For development work, install the project in editable mode if the repository's build configuration supports it:

```bash
python -m pip install -e .
```

---

## Quick Start

The following example demonstrates the intended workflow for users who already have access to a compatible `Cleaner` API.

```python
import pandas as pd
from smartclean import Cleaner

# Load a dataset.
df = pd.read_csv("enter_file_path.csv")

# Initialize the cleaner.
cleaner = Cleaner()

# Run the cleaning workflow.
result = cleaner.clean(df)

# Access the cleaned DataFrame.
cleaned_df = result.cleaned_df

# Save the result.
cleaned_df.to_csv("cleaned_data.csv", index=False)

# Compare the dataset dimensions.
print("Original shape:", df.shape)
print("Cleaned shape:", cleaned_df.shape)

# Inspect the returned report.
print(result.report)
```

**API compatibility:** This example follows the API documented in the previous README. Confirm that the published version you install exposes `Cleaner()` and `cleaner.clean(df)` before using it. The command-line workflow below is documented separately.

---

## Command-Line Workflow

The project also contains a module-based cleaning workflow.

### 1. Prepare the input dataset

For the default project configuration, place the input CSV at:

```text
data/
└── raw/
    └── data.csv
```

### 2. Run the cleaning module

From the project root, execute:

```bash
python -m smartclean.main
```

The workflow uses the configured input path, applies missing-value handling and outlier handling, and writes its outputs to the cleaned-data directory.

### 3. Review the results

The default output directory discussed for this workflow is:

```text
data/
└── cleaned/
    ├── cleaned_data.csv
    ├── cleaning_report.json
    ├── cleaning_report.txt
    └── cleaning_dashboard.png
```

Your actual paths may differ if you configure the input or output directories.

---

## Configuration

The module-based workflow has parameters for the input file, outlier-handling method, and missing-value threshold.

The following values reflect the discussed `main()` function signature:

| Parameter | Purpose | Example |
|---|---|---|
| `file_path` | Optional input dataset path | `data/raw/data.csv` |
| `outlier_method` | Outlier-handling method | `"cap"` |
| `drop_threshold` | Missing-value percentage threshold for dropping columns | `90` |

For example, the function has been structured to accept:

```python
main(
    file_path="data/raw/data.csv",
    outlier_method="cap",
    drop_threshold=90,
)
```

This example assumes the module's `main()` function is imported and that the specified path exists.

### Missing-value threshold

A threshold of `90` means that columns exceeding the configured missing-value percentage are candidates for removal, subject to the implementation's comparison rule.

For example, a column with 96% missing values exceeds a 90% threshold.

### Outlier handling

The example configuration uses `"cap"` for outlier handling. Use only method names accepted by the installed `OutlierHandler` implementation.

---

## Output Files

The module-based workflow is designed to generate the following artifacts.

| File | Purpose |
|---|---|
| `cleaned_data.csv` | Dataset after the cleaning workflow |
| `cleaning_report.json` | Structured report suitable for scripts and further analysis |
| `cleaning_report.txt` | Human-readable summary with detailed dropped-column information |
| `cleaning_dashboard.png` | Visual summary of cleaning results |

### Terminal Summary

The terminal output is intended to show concise statistics rather than print every dropped column.

A representative summary might look like this:

```text
========== SmartClean-AI Summary ==========

Original rows:              11,933
Cleaned rows:               11,933
Original columns:           1,112
Cleaned columns:             930

Missing values before:  7,853,567
Missing values filled:  5,740,609
Missing values remaining:       0

Columns dropped:                182
Outlier values capped:     1,741,216

Cleaning completed.
Reports and dashboard saved.
```

**Illustrative output:** The figures above come from a previously discussed run and should not be interpreted as guaranteed results. Actual values depend on the input dataset and configuration.

The detailed dropped-column list belongs in the generated reports rather than the concise terminal summary.

---

## Dashboard Preview

SmartClean-AI's workflow is intended to generate a dashboard image at:

```text
data/cleaned/cleaning_dashboard.png
```

To display it in this README, add a copy of the actual generated image to the repository. A convenient location is:

```text
docs/images/cleaning_dashboard.png
```

Then use this Markdown:

```markdown
![SmartClean-AI Cleaning Dashboard](docs/images/cleaning_dashboard.png)
```

The rendered image should appear here after you commit the image file to GitHub.

> **Important:** No real dashboard image was supplied with this README content, so the image reference is a placeholder until you add your actual generated dashboard.

---

## Example JSON Report

The JSON report is intended for structured access to dataset statistics, cleaning results, and detailed column information.

The following is an **illustrative schema example**, not a guaranteed copy of the generated report. Keep the actual field names from your `build_full_report()` implementation when documenting a real report.

```json
{
  "original_shape": [11933, 1112],
  "cleaned_shape": [11933, 930],
  "missing_report": {
    "missing_values_before": 7853567,
    "missing_values_filled": 5740609,
    "missing_values_remaining": 0,
    "columns_dropped": 182,
    "drop_threshold": 90.0
  },
  "dropped_columns": [
    {
      "column": "example_column",
      "missing_percentage": 96.84
    }
  ],
  "outlier_report": {
    "outlier_values_capped": 1741216
  },
  "outputs": {
    "cleaned_dataset": "cleaned_data.csv",
    "text_report": "cleaning_report.txt",
    "dashboard": "cleaning_dashboard.png"
  }
}
```

The column name shown above is a placeholder. In the real report, the dropped-column section should contain the actual column names and statistics produced by the cleaning run.

### Reading the JSON report

```python
import json
from pathlib import Path

report_path = Path("data/cleaned/cleaning_report.json")

with report_path.open("r", encoding="utf-8") as file:
    report = json.load(file)

print(json.dumps(report, indent=2))
```

The report can be used for automated validation, experiment records, or integration into other data-processing workflows.

---

## Example TXT Report

The text report is intended to provide a readable summary of the cleaning process, including the detailed dropped-column list.

Illustrative example:

```text
========================================
       SMARTCLEAN-AI CLEANING REPORT
========================================

Original dataset shape: (11933, 1112)
Cleaned dataset shape:  (11933, 930)

MISSING-VALUE HANDLING
----------------------
Missing values before: 7,853,567
Missing values filled: 5,740,609
Missing values remaining: 0
Columns dropped: 182
Drop threshold: 90.00%

DROPPED COLUMNS
---------------
- example_column: 96.84% missing
- another_example_column: 94.20% missing

OUTLIER HANDLING
----------------
Outlier values capped: 1,741,216

OUTPUT FILES
------------
cleaned_data.csv
cleaning_report.json
cleaning_report.txt
cleaning_dashboard.png

========================================
```

The example column names are placeholders, and the report layout may vary from the actual generated TXT file. A real report should preserve the full list of columns dropped during cleaning.

---

## Using the Python API

The previous documented API exposes the cleaned DataFrame and report information through the returned result object.

```python
result = cleaner.clean(df)

# Cleaned dataset
cleaned_df = result.cleaned_df

# Dataset profile before cleaning
before_profile = result.before_profile

# Dataset profile after cleaning
after_profile = result.after_profile

# Cleaning report
report = result.report
```

Save the cleaned data:

```python
result.cleaned_df.to_csv("cleaned_data.csv", index=False)
```

The names and availability of these result attributes should be checked against the installed release. Refer to the repository source code for the authoritative API.

---

## Project Structure

The following is a conceptual structure for the documented workflow. It is not intended to guarantee that every listed directory exists in every repository version.

```text
SmartClean/
├── smartclean/
│   ├── __init__.py
│   ├── main.py
│   ├── cleaner.py
│   └── ...
├── data/
│   ├── raw/
│   └── cleaned/
├── docs/
│   └── images/
│       └── cleaning_dashboard.png
├── examples/
├── README.md
├── pyproject.toml
└── requirements.txt
```

The package's actual structure should be used when documenting specific modules or classes.

---

## Dependencies

The previously documented runtime dependencies include:

- NumPy
- pandas
- Matplotlib
- scikit-learn

The package may use additional dependencies depending on the modules included in the release.

Install SmartClean-AI through PyPI to allow its declared runtime dependencies to be installed automatically. The authoritative dependency list is the package's published metadata and `pyproject.toml`.

---

## Limitations and Notes

- Cleaning outcomes depend on the input dataset, configuration, and selected methods.
- Dropping a column with excessive missing values can remove useful information; review the report before using the cleaned dataset for analysis.
- Outlier capping changes extreme values and may not be appropriate for every dataset.
- A dataset with zero missing values after cleaning is not necessarily free of all data-quality problems.
- Review the generated reports before using the cleaned output in production or research.
- Examples in this README are illustrative unless explicitly identified as verified output.

---

## Contributing

Contributions, bug reports, and feature suggestions are welcome.

- **Repository:** https://github.com/yashtalati10/SmartClean
- **Issue tracker:** https://github.com/yashtalati10/SmartClean/issues

When reporting a bug, include:

1. SmartClean-AI version.
2. Python version.
3. Operating system.
4. A minimal reproducible example.
5. The full error message, with sensitive data removed.

For code contributions, test the affected cleaning workflow and update the documentation when behavior or public APIs change.

---

## License

The previous README identifies SmartClean-AI as distributed under the MIT License. Confirm that the repository contains the corresponding `LICENSE` file before publishing this statement.

---

## Links

- **PyPI package:** https://pypi.org/project/smartclean-ai/
- **GitHub repository:** https://github.com/yashtalati10/SmartClean
- **GitHub issues:** https://github.com/yashtalati10/SmartClean/issues

**SmartClean-AI — making structured-data cleaning and reporting easier for Python developers.**
