Metadata-Version: 2.4
Name: aidatapilot
Version: 1.0.0
Summary: Automated data validation and cleaning package
Author-email: Rooben RS <rooben.rs@dhsit.co.uk>
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: scikit-learn>=1.2.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: xgboost>=1.7.0
Requires-Dist: openpyxl>=3.1.0

# AIDataPilot 🚀

**AIDataPilot** is an intelligent, automated data cleaning and profiling library designed to transform messy datasets into analysis-ready data with minimal human intervention.

## Key Features

- **🧠 Intelligent Decision Engine**: Automatically determines the best cleaning strategies based on dataset characteristics.
- **🧹 Structural Cleaning**: Standardizes column names, handles date formats (DD-MM-YYYY), and removes unnecessary white spaces.
- **💎 Duplicate Handling**: Identifies and resolves exact and near-duplicates using normalization techniques.
- **📉 Outlier Management**: Detects and caps outliers using IQR (Interquartile Range) to preserve data distribution without losing rows.
- **⚙️ Type Conversion**: Automatically promotes columns to their most appropriate types (numeric, datetime, boolean, etc.).
- **❓ Missing Value Handling**: Implements smart defaults and handles null values consistently across different data types.
- **📊 Quality Profiling**: Generates comprehensive reports on data quality, structure, and the actions taken during cleaning.

## Installation

```bash
pip install .
```

*Note: Requires Python 3.8+ and dependencies: pandas, numpy, scikit-learn, scipy.*

## Quick Start

### 🚀 Full Intelligence Engine (Auto-Pilot)
Run the full suite (Profile → Validate → Clean → ML → Text → Report) in one command:
```python
import aidatapilot

# Ingest any messy dataset
results = aidatapilot.auto_pilot("messy_data.csv", output_dir="results")

# results contains:
# - .data: The cleaned DataFrame
# - .report: Executive summary and quality scores
```

### 🧩 Individual Pipelines
You can also run specific pipelines independently:
- `auto_clean()`: Deep semantic cleaning and outlier handling.
- `auto_validate()`: Diagnostic data health reporting.
- `auto_ml()`: Feature engineering and model recommendations.
- `auto_text()`: Natural language narrative generation.
- `auto_profile()`: Detailed statistical distribution profiling.

## Demos
Check the `examples/` directory for standalone scripts demonstrating each pipeline:
- `demo_clean.py`, `demo_ml.py`, `demo_report.py`, etc.

## Project Structure

- `aidatapilot/core/`: Core logic and shared context.
- `aidatapilot/features/`: Modular cleaning features (missing values, outliers, duplicates, etc.).
- `aidatapilot/pipelines/`: Orchestration layers for automated workflows.
- `aidatapilot/utils/`: Internal helper functions.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the LICENSE file for details.
