Metadata-Version: 2.4
Name: data-detective-toolkit
Version: 0.3.1
Summary: Automated data-quality profiling: CLI, web app, and REST API for CSV intelligence reports.
License-Expression: MIT
Project-URL: Homepage, https://github.com/murs5l/data-detective
Project-URL: Issues, https://github.com/murs5l/data-detective/issues
Keywords: data-quality,profiling,csv,data-profiling,eda,outlier-detection
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas<4.0,>=2.0
Requires-Dist: numpy<3.0,>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == "api"
Requires-Dist: uvicorn[standard]>=0.29; extra == "api"
Requires-Dist: python-multipart>=0.0.9; extra == "api"
Dynamic: license-file

# 🕵️ Data Detective

[![Tests](https://github.com/murs5l/data-detective/actions/workflows/tests.yml/badge.svg)](https://github.com/murs5l/data-detective/actions/workflows/tests.yml)

**Automated data quality profiling in seconds.** Point Data Detective at a CSV and get back an intelligent report: shape, data types, missing values, duplicates, outliers, correlations, distribution skew, ID-like columns, and actionable insights. Choose your interface: fast CLI for scripts and CI, browser-based web app for exploration, or REST API for integration.

### Why Data Detective?

- **Fast**: profiles 1000+ rows in under a second
- **Thorough**: detects 20+ data quality issues (outliers, duplicates, high cardinality, mixed types, skew, etc.)
- **Private**: your data never leaves your machine; everything runs locally
- **Flexible**: use as a CLI tool, web app, Python library, or REST API
- **Lightweight**: no external ML services or cloud dependencies

### Two ways to use it

1. **CLI**: `data-detective analyze myfile.csv --html` for scripts and CI pipelines.
2. **Web app**: `docker compose up` for interactive exploration with live charts.

Both interfaces use the same profiling engine and produce identical reports.

---

## Quick start

### Web app (interactive, no command line needed)

The easiest way to explore a CSV:

```bash
git clone https://github.com/murs5l/data-detective.git
cd data-detective
docker compose up --build
```

Then open **http://localhost:8000**, drag and drop a CSV, and explore. The report includes:
- Data shape and types
- Missing value heatmap
- Column Explorer: drill down into any numeric column to see histogram, boxplot, and 5-number summary
- Outlier detection (IQR and MAD methods)
- Correlation heatmap for numeric column pairs
- Insights highlighted in plain English

The browser-based UI is designed for non-technical and technical users alike: actionable insights appear first, and granular technical details are tucked behind a "Full technical report" toggle.

**Running without Docker:** if you have Python and FastAPI installed:
```bash
pip install -e ".[api]"
uvicorn backend.app.main:app --reload
```
Then visit `http://localhost:8000`.

### CLI (scripting and CI/CD)

Install locally for use in scripts and automated pipelines:

```bash
pip install data-detective-toolkit
data-detective analyze myfile.csv
```

> The PyPI package is `data-detective-toolkit`; the installed command is `data-detective`.

See [CLI docs](#cli) below for all flags and output formats.

## CLI reference

### Installation

```bash
pip install data-detective-toolkit
```

### Basic usage

```bash
# Print a text summary to stdout
data-detective analyze myfile.csv

# Generate an interactive HTML report
data-detective analyze myfile.csv --html

# Output JSON for programmatic use
data-detective analyze myfile.csv --json | jq '.insights'

# Write to specific files
data-detective analyze myfile.csv --output-html reports/q1_data.html
```

### Options

| Flag               | Description                                      | Example |
|--------------------|-------------------------------------------------|---------|
| `--json`           | Print full report as JSON to stdout              | `data-detective analyze data.csv --json` |
| `--html`           | Generate `report.html` in current directory      | `data-detective analyze data.csv --html` |
| `--output-json`    | Write JSON report to a specific file path         | `--output-json reports/profile.json` |
| `--output-html`    | Write HTML report to a specific file path         | `--output-html reports/profile.html` |
| `--outlier-method` | Choose outlier detection: `iqr` or `mad` (default) | `--outlier-method iqr` |
| `--quiet`          | Suppress non-error progress messages              | `--quiet` |

### Examples

**CI/CD pipeline**: fail if data quality issues exceed a threshold
```bash
data-detective analyze incoming.csv --json | jq '.outliers_mad | length' | xargs -I {} bash -c 'exit {}'
```

**Batch processing**: profile all CSVs in a directory
```bash
for f in data/*.csv; do
  data-detective analyze "$f" --output-html "reports/$(basename $f .csv).html"
done
```

**Data validation**: check for specific issues before ETL
```bash
data-detective analyze input.csv --quiet --json | jq '.high_cardinality_columns'
```

## REST API (for integration)

When running the web app or backend API server, you can integrate Data Detective into your own applications.

### Endpoints

**POST /api/analyze** – Analyze a CSV file, return JSON report
```bash
curl -F "file=@data.csv" "http://localhost:8000/api/analyze?outlier_method=mad"
```

Response: JSON object with all profiling data (see example below).

**POST /api/analyze/html** – Analyze a CSV file, return standalone HTML report
```bash
curl -F "file=@data.csv" "http://localhost:8000/api/analyze/html" > report.html
```

Response: A self-contained HTML file (no external dependencies).

**GET /api/health** – Health check
```bash
curl http://localhost:8000/api/health
```

Response: `{"status": "ok", "version": "0.3.0"}`

### Python example

```python
import requests

with open("mydata.csv", "rb") as f:
    response = requests.post(
        "http://localhost:8000/api/analyze",
        files={"file": f},
        params={"outlier_method": "mad"}
    )

report = response.json()
print(f"Shape: {report['shape']}")
print(f"Insights: {report['insights']}")
print(f"Outliers detected: {report['outliers_mad']}")
```

### JavaScript example

```javascript
const formData = new FormData();
formData.append("file", csvFile); // File object from input[type=file]

const response = await fetch("/api/analyze?outlier_method=mad", {
  method: "POST",
  body: formData
});

const report = await response.json();
console.log(report.insights);
```

Full API documentation is available at `/docs` (Swagger UI) when the backend is running.

## What it detects

Data Detective automatically flags 20+ data quality issues:

**Structure & Completeness**
- Shape (rows × columns) and column data types
- Missing values (count and percentage per column)
- Duplicate rows
- Duplicate columns (identical data, different names)
- Constant columns (no variation; e.g., all 1s)

**Statistical Issues**
- Outliers: IQR method (classic box-and-whisker) or MAD (robust for skewed data)
- Highly correlated numeric pairs
- Skewed distributions (heavy left/right tails)
- Kurtosis (fat tails or sharp peaks)

**Data Type & Value Issues**
- High-cardinality columns (likely IDs or keys)
- Mixed-type columns (e.g., numbers and strings in the same column)
- Unexpected negative values (e.g., negative `age`, `price`, `quantity`)
- Date-like values not parsed as datetime

**Text Columns**
- Length statistics (min, max, average)
- Blank or whitespace-only values

**Aggregates**
- Full numeric correlation matrix
- Histograms with bin edges and counts for numeric columns
- Five-number summary (min, Q1, median, Q3, max) per numeric column
- Per-column memory footprint (KB)

**Output**
- Plain-English actionable insights highlighting the most important issues

## Development

```bash
# core engine + CLI
pip install -e ".[dev]"
pytest tests

# backend API
pip install -e ".[api]"
pip install -r backend/requirements-dev.txt
pytest backend/tests

# fastscan (Go speed layer)
cd tools/fastscan && go test ./...
```

CI runs all three (`.github/workflows/tests.yml`) on every push and PR.

## Roadmap

- GitHub Action to run Data Detective as a data-quality gate in CI
- Support for additional sources (Parquet, JSON, database connections)
- Scheduled/hosted profiling for pipelines beyond ad-hoc uploads

## License

See [LICENSE](LICENSE).
