Metadata-Version: 2.4
Name: dataprocessing-ai
Version: 0.1.1
Summary: AI-native data processing library — REST API, MCP server, and Python package
Author: oluwagbemilesi
License: MIT
Project-URL: Homepage, https://github.com/Techlon/dataprocessing-ai
Project-URL: Repository, https://github.com/Techlon/dataprocessing-ai
Project-URL: Issues, https://github.com/Techlon/dataprocessing-ai/issues
Keywords: data,processing,AI,REST,MCP,pandas
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.2.0
Requires-Dist: numpy>=1.26.0
Provides-Extra: api
Requires-Dist: fastapi>=0.110.0; extra == "api"
Requires-Dist: uvicorn>=0.27.0; extra == "api"
Requires-Dist: pydantic>=2.6.0; extra == "api"
Requires-Dist: python-multipart>=0.0.9; extra == "api"
Provides-Extra: mcp
Requires-Dist: mcp>=0.9.0; extra == "mcp"
Provides-Extra: formats
Requires-Dist: openpyxl>=3.1.0; extra == "formats"
Requires-Dist: pyarrow>=15.0.0; extra == "formats"
Provides-Extra: all
Requires-Dist: dataprocessing-ai[api,formats,mcp]; extra == "all"
Provides-Extra: dev
Requires-Dist: dataprocessing-ai[all]; extra == "dev"
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: httpx>=0.27.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Dynamic: license-file

# DataProcessing AI

An AI-native Python library for data processing and transformation.
Built to be called by any AI agent via REST API, MCP server, or direct Python import.

## What it does

- **Ingest** - read CSV, JSON, Excel, Parquet files into a standard format
- **Clean** - remove nulls, duplicates, outliers, standardise column names
- **Transform** - filter, sort, group, pivot, merge, reshape datasets
- **Analyse** - generate statistics, correlations, distributions, outlier reports
- **Visualise** - produce Vega-Lite chart specs (histogram, bar, scatter, line, correlation heatmap)

## Three ways to use it

### 1. REST API (any AI, any language)

Install with the API extra, then start the server:
```bash
pip install "dataprocessing-ai[api]"
uvicorn dataprocessing.api:app --host 0.0.0.0 --port 8000
```

Call it:
```bash
curl -X POST http://localhost:8000/ingest -F "file=@data.csv"
curl -X POST http://localhost:8000/clean -H "Content-Type: application/json" -d '{"data": [...]}'
curl -X POST http://localhost:8000/transform -H "Content-Type: application/json" -d '{"data": [...], "operation": "filter_rows", "params": {"column": "age", "operator": "gt", "value": 25}}'
curl -X POST http://localhost:8000/analyse -H "Content-Type: application/json" -d '{"data": [...]}'
curl -X POST http://localhost:8000/visualise -H "Content-Type: application/json" -d '{"data": [...], "chart": "histogram", "params": {"column": "age", "bins": 10}}'
```

### 2. MCP Server (Claude native tools)

Install with the MCP extra:
```bash
pip install "dataprocessing-ai[mcp]"
```

Then add to your `claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "dataprocessing": {
      "command": "dataprocessing-mcp"
    }
  }
}
```

No paths, no `PYTHONPATH` — the installed package provides the
`dataprocessing-mcp` command directly.

### 3. Python package (direct import)

```python
from dataprocessing.ingest import read_file
from dataprocessing.clean import clean_all
from dataprocessing.transform import filter_rows
from dataprocessing.analyse import full_report
from dataprocessing.visualise import histogram, correlation_heatmap

df = read_file("data.csv")
df = clean_all(df)
report = full_report(df)

# Chart functions return Vega-Lite spec dicts (JSON-serialisable)
hist_spec = histogram(df, column="age", bins=10)
heatmap_spec = correlation_heatmap(df)
```

## Installation

```bash
pip install dataprocessing-ai
```

That gives you the core library (`ingest`, `clean`, `transform`, `analyse`,
`visualise`) with a minimal dependency footprint.

Optional extras add the interfaces and file formats you need:

```bash
pip install "dataprocessing-ai[mcp]"      # MCP server (for Claude and other agents)
pip install "dataprocessing-ai[api]"      # REST API server
pip install "dataprocessing-ai[formats]"  # Excel (.xlsx) and Parquet support
pip install "dataprocessing-ai[all]"      # everything
```

### Installing from source

```bash
git clone https://github.com/Techlon/dataprocessing-ai.git
cd dataprocessing-ai
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

## API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | /health | API status |
| POST | /ingest | Upload and read a data file |
| POST | /clean | Clean a dataset |
| POST | /transform | Transform a dataset |
| POST | /analyse | Analyse a dataset |
| POST | /visualise | Produce a Vega-Lite chart spec from a dataset |

## Charts

The `/visualise` endpoint (and the `visualise_data` MCP tool) accept a `chart`
name and a `params` object. Each returns a [Vega-Lite](https://vega.github.io/vega-lite/)
spec dict that any Vega-Lite renderer can display.

| Chart | Params | Notes |
|-------|--------|-------|
| `histogram` | `column` (numeric), `bins` (int, default 10) | Distribution of a numeric column |
| `bar_chart` | `column` (any) | Counts per category |
| `scatter` | `x` (numeric), `y` (numeric) | Relationship between two numeric columns |
| `line_chart` | `x` (any), `y` (numeric) | `x` may be a date or category; `y` must be numeric |
| `correlation_heatmap` | `columns` (list, optional) | Correlations across numeric columns; defaults to all numeric columns |

A bad chart name, a missing column, or a non-numeric column where a numeric one
is required returns HTTP 400 (or raises `ValueError` when called directly).

### Example response

A call with `{"chart": "histogram", "params": {"column": "age", "bins": 3}}`
returns a Vega-Lite spec like:

```json
{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "Histogram of age",
  "data": {
    "values": [
      {"bin_start": 23.0, "bin_end": 29.33, "count": 4},
      {"bin_start": 29.33, "bin_end": 35.67, "count": 3},
      {"bin_start": 35.67, "bin_end": 42.0, "count": 1}
    ]
  },
  "mark": "bar",
  "encoding": {
    "x": {"field": "bin_start", "type": "quantitative"},
    "x2": {"field": "bin_end", "type": "quantitative"},
    "y": {"field": "count", "type": "quantitative"}
  }
}
```

Paste the returned spec into the [Vega-Lite editor](https://vega.github.io/editor/)
or render it with any Vega-Lite-compatible frontend.

## MCP Tools

The MCP server exposes each capability as a Claude-native tool:

| Tool | Arguments | Description |
|------|-----------|-------------|
| `ingest_file` | `file_path` | Read a data file (CSV, JSON, Excel, Parquet) from disk |
| `clean_data` | `data`, `drop_null_threshold`, `remove_dupes`, `standardise_cols` | Remove nulls, duplicates, and standardise column names |
| `transform_data` | `data`, `operation`, `params` | Apply a named transform (filter, select, rename, sort, group, add column) |
| `analyse_data` | `data` | Full statistical report (summary, correlations, missing, outliers) |
| `visualise_data` | `data`, `chart`, `params` | Produce a Vega-Lite chart spec (see [Charts](#charts) for names and params) |

The `visualise_data` tool uses the same chart names and params as the `/visualise`
endpoint and returns a Vega-Lite spec dict.

## Supported formats
CSV, JSON, Excel (.xlsx), Parquet

## License
MIT
