Metadata-Version: 2.4
Name: ai-data-platform
Version: 0.3.0
Summary: AI Data Platform: metadata catalog, profiling, synthetic data generation, semantic models, NL-to-SQL, and MCP server — local-first.
Project-URL: Homepage, https://github.com/Yogi776/data-generation-sdk
Project-URL: Documentation, https://github.com/Yogi776/data-generation-sdk#readme
Project-URL: Issues, https://github.com/Yogi776/data-generation-sdk/issues
Project-URL: Repository, https://github.com/Yogi776/data-generation-sdk
Project-URL: Source, https://github.com/Yogi776/data-generation-sdk
Author-email: Yogesh Khangode <yogesh.khangode@tmdc.io>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-catalog,data-quality,duckdb,mcp,metadata,polars,profiling,semantic-layer,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: duckdb>=1.0
Requires-Dist: fastapi>=0.111
Requires-Dist: httpx>=0.27
Requires-Dist: numpy>=1.26
Requires-Dist: polars>=1.0
Requires-Dist: pyarrow>=16.0
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn>=0.30
Provides-Extra: all
Requires-Dist: mcp>=1.2; extra == 'all'
Requires-Dist: openai>=1.30; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1; extra == 'all'
Requires-Dist: pymysql>=1.1; extra == 'all'
Provides-Extra: bigquery
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: llm
Requires-Dist: openai>=1.30; extra == 'llm'
Provides-Extra: mcp
Requires-Dist: mcp>=1.2; extra == 'mcp'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Provides-Extra: snowflake
Provides-Extra: trino
Provides-Extra: ui
Description-Content-Type: text/markdown

# ai-data-platform

## Why it matters

Data is the prerequisite for every analytics dashboard, ML model, pipeline test, and stakeholder demo — yet most teams spend weeks waiting for it. Production exports require compliance approval. Hand-crafted test files miss relationships and break at scale. POCs stall while access is negotiated.

**ai-data-platform** is a local-first tool that turns a schema declaration or a small sample into FK-safe, realistic synthetic data in minutes — with automated quality scoring, semantic models, SQL analytics, and AI-driven workflows. No cloud account. No PII exposure. Works in retail, healthcare, finance, and any other domain without custom code.

| Without ADP | With ADP |
|---|---|
| Wait 2–6 weeks for production data access | Generate 50k rows in under a minute |
| Hand-write fake CSVs that break FK integrity | FK-safe, zero-orphan data at any volume |
| Rebuild test data after every schema change | Re-run `apply-spec` or `scan` → regenerate |
| Demo dashboards on empty tables | Realistic distributions from day one |
| Share production exports (PII risk) | Synthetic data — no real PII leaves your machine |

### Acceleration timeline

| Phase | Traditional | With ai-data-platform |
|---|---|---|
| Get representative data | 2–6 weeks | 5–10 minutes |
| Validate data quality | Manual spot-checks | Automated score + report |
| Build semantic layer | Weeks of hand-written YAML | Auto-detected facts/dims/measures |
| Demo to stakeholders | Empty charts or risky prod samples | Realistic KPIs on synthetic data |
| Onboard a new team member | "Ask someone for the test dump" | `adp init && adp apply-spec && adp generate-data` |

### Works in any domain — zero customization

There is no `if domain == "healthcare"` in the codebase. Retail orders, patient admissions, bank transactions, factory work orders, and telecom CDRs are all just tables, keys, categories, amounts, and dates.

| Domain | What you build faster | Validated example |
|---|---|---|
| **Retail / E-commerce** | Sales dashboards, fraud models, recommendation engines | [retail-ecommerce](examples/retail-ecommerce/) — 4 tables, 100/100 |
| **Healthcare** | Patient analytics, admission workflows, compliance testing | [healthcare-claims](https://github.com/Yogi776/data-generation-sdk/tree/main/healthcare-claims) — 5+ tables, temporal/hierarchy |
| **Finance / Banking** | Risk scoring, transaction monitoring, regulatory reporting | FK-safe account → transaction chains |
| **SaaS / CRM** | Customer 360, churn models, sales pipelines | [customer-transaction](examples/customer-transaction/) — 98 columns, 100/100 |
| **Manufacturing / IoT** | Supply chain dashboards, work-order tracking | Parent-child BOM relationships |
| **Any new vertical** | POC before production access is granted | Cold-start from `spec.yaml` — no sample data needed |

---

## End-to-end — what you get

One project, one flow, five tangible deliverables:

```
  YOUR INPUT                    ADP PIPELINE                      YOUR OUTPUT
  ──────────                    ────────────                      ───────────

  spec.yaml          ──▶  apply-spec  ──▶  catalog + profiles
  or CSV/DB sample   ──▶  scan/profile ──▶  learned distributions
                              │
                              ▼
                         generate-data
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
         output/         quality.md      semantic model
    (parquet/csv/      (score +          (Cube.js YAML
     duckdb files)      evidence)         for BI tools)
              │
              ▼
         explore / ui / sql
    (query, dashboard KPIs,
     natural-language analytics)
```

| Deliverable | What it is | Who uses it |
|---|---|---|
| **`output/`** | FK-safe synthetic tables (Parquet, CSV, DuckDB, or SQL) | Engineers, QA, ML pipelines |
| **`quality.md`** | Weighted score (0–100) with per-check pass/fail evidence | QA, data engineers, compliance |
| **Semantic model** | Auto-detected facts, dimensions, measures → Cube.js YAML | BI developers, analytics teams |
| **Data dictionary** | Markdown documentation of every table and column | Onboarding, documentation, audits |
| **SQL analytics** | Query generated data in DuckDB; NL-to-SQL for business questions | Analysts, product managers, demos |

**A complete example:** A product manager describes a retail dataset → agent applies a spec → 50k rows generated → quality score 100/100 → revenue-by-city query returns realistic KPIs → stakeholder demo ready in one session. No production data touched.

Full walkthrough: [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md)

---

## Who it's for

| Persona | Your problem | What you do | What you get (in minutes) |
|---|---|---|---|
| **Data engineer** | Weeks waiting for masked prod exports | `connect` → `scan` → `profile` → `generate-data` | FK-safe test data at any volume, CI-reproducible with `--seed 42` |
| **Product / solution architect** | Can't demo until prod access is approved | `apply-spec spec.yaml` → `generate-data` | Working dataset from a design doc — no sample files needed |
| **Analytics / BI developer** | Dashboards built on empty or fake data | `generate-data` → `semantic-model` → `explore sql` | Realistic data + Cube.js layer + queryable KPIs |
| **QA / test engineer** | Hand-crafted CSVs break FK integrity | `generate-data` → `quality-check` | Scored report with 0 orphans guaranteed |
| **ML engineer** | Not enough labeled data to train or evaluate | `profile` (learn shapes) → `generate-data --rows 1000000` | 1M rows preserving statistical distributions |
| **AI agent user** | Agent has nothing to work with in the IDE | MCP: "generate 10k rows and run quality check" | Full pipeline driven by natural language in Cursor/Claude |
| **Consultant / SI** | Every client engagement starts from zero | Same tool, swap `spec.yaml` per domain | Retail Monday, healthcare Tuesday — no custom code |

**Not sure where to start?**

| If you are… | Start here | Time |
|---|---|---|
| Non-technical / business user | Ask your AI agent (Path C) or read [USE-CASES.md](docs/USE-CASES.md) | 3 min |
| Have a schema but no data | [Path A](#path-a--no-data-needed-5-min) — `apply-spec` | 5 min |
| Have CSV or database samples | [Path B](#path-b--learn-from-sample-data-10-min) — `connect` → `scan` | 10 min |
| Want the full tutorial | [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md) | 15 min |

---

## How it works

Three ways in. Same pipeline. Same validated output.

```mermaid
flowchart TD
  subgraph paths [Choose your path]
    Q1{Have sample data?}
    Q1 -->|No| PathA["Path A: spec.yaml\napply-spec → generate"]
    Q1 -->|Yes| PathB["Path B: learn from data\nconnect → scan → profile → generate"]
    Q1 -->|AI IDE| PathC["Path C: MCP\nCursor / Claude / Windsurf"]
  end

  PathA --> Pipeline
  PathB --> Pipeline
  PathC --> Pipeline

  subgraph Pipeline [Generation pipeline]
    Gen[FK-safe generator]
    QC[quality-check]
    Gen --> QC
  end

  QC --> Out[output/ files]

  Out --> Next[semantic-model · explore sql · ui · docs]
```

**Five steps, plain language:**

| Step | What happens | You see |
|---|---|---|
| 1. **Define** | Declare your schema (`spec.yaml`) or connect your sample data | Tables, columns, and relationships in the catalog |
| 2. **Learn** | Engine profiles distributions, detects keys, flags PII | Statistics per column — means, categories, null ratios |
| 3. **Generate** | FK-safe synthetic data at any volume, seeded for reproducibility | Files in `output/` — one per table |
| 4. **Validate** | Auto-derived quality checks run against your metadata | Score out of 100 with per-check evidence |
| 5. **Use** | Query with SQL, build semantic models, browse in UI, or drive from AI | KPIs, dashboards, data dictionaries, insights |

Full internals: [docs/USER-FLOW.md](docs/USER-FLOW.md) · Architecture: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)

---

## Get started

### Install

```bash
pip install 'ai-data-platform[all]'      # everything (recommended)
pip install 'ai-data-platform[mcp]'       # MCP only (add to existing install)
pip install 'ai-data-platform[postgres]'  # PostgreSQL connector
```

Requires Python 3.11+.

### Path A — no data needed (~5 min)

You have a design doc or schema in mind. No sample files required.

```bash
mkdir demo && cd demo
adp init --name demo
adp apply-spec path/to/spec.yaml       # see examples/customer-transaction, healthcare-claims (GitHub)
adp generate-data --rows 50000 --output parquet
adp quality-check --report quality.md
```

**You get:** `output/*.parquet` + `quality.md` with score 100/100.

### Path B — learn from sample data (~10 min)

You have CSV, Parquet, DuckDB, PostgreSQL, or MySQL samples.

```bash
mkdir demo && cd demo
adp init --name demo
adp connect --name my-db --type csv --path ./data
adp scan && adp profile
adp generate-data --rows 50000 --output parquet
adp quality-check
```

**You get:** Data that mirrors your sample's distributions, scaled to 50k rows.

### Path C — drive from Cursor / Claude (~3 min)

```bash
pip install 'ai-data-platform[mcp]'
cd my-project && adp init && adp setup-agent
```

MCP works with any client (Claude, Cursor, Windsurf, VS Code). Cursor users also get auto-installed agent skills via `adp init`.

Add `.cursor/mcp.json` next to your `adp.yaml` (or use `adp init` to auto-write):

```json
{"mcpServers": {"adp": {"command": "adp", "args": ["mcp-server"]}}}
```

Open the project folder as your workspace, reload MCP, then ask:

> "Apply the healthcare spec and generate 10k rows. Run a quality check and show me the first 10 rows."

**You get:** Agent reports quality score, shows sample rows, files land in `output/`.

**Runnable walkthrough with expected output at every step:** [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md)

**All 16 use cases:** [docs/USE-CASES.md](docs/USE-CASES.md)

---

## Key capabilities

| Capability | CLI | MCP tool | What it does |
|---|---|---|---|
| Declarative generation | `adp apply-spec` | `apply_spec` | Generate from `spec.yaml` — no sample data |
| Learn from samples | `adp scan` + `adp profile` | `scan_sources` + `profile_source` | Discover schema and distributions |
| FK-safe generation | `adp generate-data` | `generate_synthetic_data` | Seeded, deterministic, zero orphans |
| Quality validation | `adp quality-check` | `run_quality_check` | Weighted score with per-check evidence |
| SQL analytics | `adp explore sql` | `execute_sql` | Query generated data in DuckDB |
| Semantic models | `adp semantic-model` | `create_semantic_model` | Auto-detect facts, dims, measures → Cube.js YAML |
| Natural language SQL | `adp sql` | `generate_sql` | NL → read-only SELECT (PII-safe) |
| Data dictionary | `adp docs` | `generate_docs` | Markdown catalog documentation |
| Web console | `adp ui` | — | Browse at http://127.0.0.1:8765 |
| AI spec drafting | — | `propose_spec` | LLM drafts `spec.yaml` from plain language |

One backend (`ADPClient`), four interfaces: **CLI · SDK · Web UI · MCP**. Same logic everywhere.

```python
from ai_data_platform import ADPClient

client = ADPClient(".")
client.apply_spec("spec.yaml")
result = client.generate_data(rows=50_000, output_format="parquet")
print(client.quality_check()["quality_score"])   # → 100.0
```

---

## Examples

| Example | Path | Tables | Best for |
|---|---|---|---|
| [customer-transaction](examples/customer-transaction/) | A or B | 2 (+ KYC) | Full 98-column e-commerce model |
| [retail-ecommerce](examples/retail-ecommerce/) | B — CSV | 4 | Retail star schema, 32/32 checks |
| [healthcare-claims](https://github.com/Yogi776/data-generation-sdk/tree/main/healthcare-claims) | A — spec | 5+ | Temporal rules, hierarchies |

---

## Configuration

**Data sources** (`adp.yaml`):

```yaml
sources:
  - name: csv_files
    type: csv
    path: ./data
  - name: postgres_prod
    type: postgres
    dsn: "postgresql+psycopg://user:${PGPASSWORD}@host:5432/shop"
    schema: public
```

Secrets use `${ENV_VAR}` interpolation — plaintext passwords are rejected.

**AI provider** (for NL-to-SQL and `propose_spec`):

```yaml
model_provider:
  provider: minimax          # minimax | openai | anthropic | gemini | local
  api_key_env: MINIMAX_API_KEY
```

`provider: local` runs fully offline without LLM calls.

**Declarative spec** (`spec.yaml`) — see [docs/SPEC-REFERENCE.md](docs/SPEC-REFERENCE.md):

```yaml
version: 1
tables:
  - name: dim_customer
    columns:
      - {name: customer_id, type: uuid, primary_key: true}
      - {name: gender, type: string, values: {Male: 48, Female: 50, Other: 2}}
      - {name: age, type: int, min: 18, max: 85}
      - {name: signup_date, type: date, start: 2020-01-01, end: 2026-01-01}
```

---

## Documentation

| Guide | What it covers |
|---|---|
| [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md) | Runnable E2E walkthrough — Paths A, B, C, D with expected outputs |
| [docs/USE-CASES.md](docs/USE-CASES.md) | All 16 use cases — goal, commands, example, expected outcome |
| [docs/SPEC-REFERENCE.md](docs/SPEC-REFERENCE.md) | Complete `spec.yaml` language reference |
| [docs/MCP-GUIDE.md](docs/MCP-GUIDE.md) | All 25 MCP tools, agent flows, IDE setup |
| [docs/AGENT-FLOW.md](docs/AGENT-FLOW.md) | Guided agent workflows (intake → KPI validation) |
| [docs/USER-FLOW.md](docs/USER-FLOW.md) | Step-by-step internals for each CLI stage |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design principles, pipeline detail, extension points |
| [LOCAL-SETUP.md](LOCAL-SETUP.md) | macOS dev install and MCP setup |

---

## Development

```bash
git clone git@github.com:Yogi776/data-generation-sdk.git
cd data-generation-sdk
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,all]"

pytest && ruff check . && mypy src
```

---

## License

Apache-2.0 — see [LICENSE](./LICENSE).
