Metadata-Version: 2.5
Name: datamcp-ai
Version: 0.1.5
Summary: AI Data Engineer MCP — Postgres + dlt + dbt + Dagster
License: Apache-2.0
License-File: LICENSE
Keywords: dagster,data-engineering,dbt,dlt,mcp,postgres
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.11
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: asyncpg>=0.31.0
Requires-Dist: dagster-dbt>=0.29.25
Requires-Dist: dagster-webserver>=1.13.25
Requires-Dist: dagster>=1.13.25
Requires-Dist: dbt-core>=1.12.5
Requires-Dist: dbt-duckdb>=1.11.0
Requires-Dist: dbt-postgres>=1.11.0
Requires-Dist: dlt>=1.30.0
Requires-Dist: duckdb>=1.5.6
Requires-Dist: google-genai>=2.27.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: jinja2>=3.1.0
Requires-Dist: keyring>=25.7.0
Requires-Dist: mcp>=2.2.0
Requires-Dist: openai>=3.23.0
Requires-Dist: pandas>=3.0.6
Requires-Dist: plotly>=7.1.0
Requires-Dist: psycopg2-binary>=2.9.13
Requires-Dist: pydantic-settings>=2.15.0
Requires-Dist: pydantic>=2.13.5
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sqlite-vec>=0.1.9
Requires-Dist: streamlit>=1.64.0
Requires-Dist: typer>=0.27.2
Provides-Extra: dev
Requires-Dist: mypy>=2.4; extra == 'dev'
Requires-Dist: pre-commit>=4.6.2; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.4.0; extra == 'dev'
Requires-Dist: pytest-cov>=7.1.0; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: ruff>=0.16; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
Requires-Dist: mkdocs>=1.6.0; extra == 'docs'
Provides-Extra: kafka
Requires-Dist: kafka-python>=2.0.0; extra == 'kafka'
Provides-Extra: typesafe
Requires-Dist: typesafe-sdk>=0.7.2; extra == 'typesafe'
Description-Content-Type: text/markdown

# DataMCP — AI Data Engineer

> Connect your Postgres database. Get a production dbt project in under 10 minutes.

DataMCP is an MCP server that acts as an AI data engineer inside Claude Code, Cursor, and Codex. It connects to your Postgres database, reads and maps the schema, infers the business domain, and generates a complete, production-quality dbt project — automatically.

**Full documentation**: build and view locally with `pip install -e '.[docs]' && mkdocs serve`, then open `http://127.0.0.1:8000`. Covers installation, a full pipeline walkthrough with real output, every tool, AI provider setup, security, and troubleshooting.

## Quickstart

```bash
pip install datamcp-ai

datamcp init
datamcp configure   # stores credentials in OS keychain, never on disk

# Open Claude Code and type:
# "Set up analytics for my project"
```

That's it. DataMCP calls the 10 core pipeline tools in sequence (of 30 MCP tools total — see below) and produces:
- Staging models (one per source table, DuckDB-validated)
- Fact and dimension mart models
- 5–11 domain-specific KPI definitions
- A dlt ingestion pipeline
- A Dagster project with a daily schedule
- 30+ data quality tests — all passing

**Total time: under 10 minutes.**

## Installation

```bash
pip install datamcp-ai
```

Requires Python 3.11+. Bring your own Gemini API key (OpenAI optional).
The PyPI distribution is named `datamcp-ai` (the plain `datamcp` name was
already taken by an unrelated package) — the installed CLI command and
Python import are still `datamcp`.

## Configuration

```bash
datamcp configure
# Prompts for: DB host, port, database, username, password, Gemini key
# (OpenAI key optional — only needed if DATAMCP_AI_PROVIDER=openai)
# All credentials stored in OS keychain — never on disk

datamcp install --editor claude-code  # or cursor, codex
```

## MCP Tools (30)

**Core pipeline** (steps 1–10 of the standard flow)

| Tool | Description |
|---|---|
| `connect_database` | Establish and validate a Postgres connection |
| `introspect_schema` | Read the complete Postgres schema: tables, columns, data types, PKs, FKs |
| `infer_domain` | Classify the business domain from schema metadata only |
| `generate_dbt_project` | Scaffold a complete dbt project with Postgres and DuckDB profiles |
| `generate_staging_models` | Generate one staging model (stg_*.sql) per source table |
| `generate_mart_models` | Generate fact and dimension mart models from the relationship graph |
| `generate_kpis` | Generate domain-specific KPI definitions as dbt metrics |
| `run_dbt` | Execute a dbt command (run, test, compile, build) |
| `check_freshness` | Check when each source table last received new rows |
| `get_pipeline_status` | Get an aggregated health dashboard across all pipeline components |

**Explore & query**

| Tool | Description |
|---|---|
| `query_knowledge_base` | Semantic search over the project knowledge base |
| `explain_model` | Get a plain-English explanation of any dbt model |
| `query_data` | Ask a natural language question about the data |
| `execute_sql` | Execute a raw SELECT query against the connected database |

**Schema analysis & scaffolding**

| Tool | Description |
|---|---|
| `get_project_id` | Look up the internal project UUID for a project_name, creating it on first use |
| `generate_er_diagram` | Generate a Mermaid erDiagram from the current schema snapshot |
| `infer_model_candidates` | Classify tables as fact, dimension, bridge, lookup, or scd_snapshot candidates |
| `generate_dbt_macros` | Generate dbt macros derived from schema column patterns |
| `scaffold_dagster` | Generate a complete Dagster project with dbt asset definitions and a daily schedule |
| `check_dlt_alignment` | Compare the dlt source schema against dbt staging models for drift |
| `configure_postgres_sink` | Generate a Kafka Connect connector config for Postgres |
| `scaffold_dlt` | Generate a complete dlt ingestion pipeline for the connected Postgres database |
| `run_dlt` | Execute a generated dlt pipeline.py and persist the run result |

**Orchestration, BI & streaming**

| Tool | Description |
|---|---|
| `run_dagster` | Start, stop, check status of, or trigger a run on a generated Dagster dev server |
| `connect_metabase` | Authenticate to Metabase, register the connected database, and trigger an initial sync |
| `sync_metabase` | Re-sync Metabase's metadata and flag charts referencing changed dbt models |
| `create_metabase_dashboard` | Create one Metabase question per project KPI and group them into a dashboard |
| `create_metabase_chart` | Turn a natural-language question into a real Metabase chart |
| `provision_kafka_topic` | Create a Kafka topic for streaming ingestion |
| `check_consumer_lag` | Check Kafka consumer group lag per partition |

## Security

- Database credentials: OS keychain only (Mac Keychain / Linux Secret Service / Windows Credential Manager)
- AI inference: schema metadata only — table names, column names, types, FK relationships. Never actual data values, credentials, or PII.
- Knowledge base: schema metadata and model definitions only. No row data.

See [SECURITY.md](docs/SECURITY.md) and [DATA_PRIVACY.md](docs/DATA_PRIVACY.md).

## What gets generated

```
datamcp_output/
├── dbt/
│   └── my_project/
│       ├── dbt_project.yml
│       ├── profiles.yml          # Postgres target
│       ├── profiles_duckdb.yml   # DuckDB local dev target
│       └── models/
│           ├── staging/
│           │   ├── sources.yml
│           │   ├── stg_users.sql
│           │   ├── stg_orders.sql
│           │   └── ...
│           └── marts/
│               ├── fct_orders.sql
│               ├── dim_customers.sql
│               └── _metrics.yml
├── dlt/
│   ├── pipeline.py
│   └── sources/my_project.py
└── dagster_datamcp/
    ├── assets/dbt_assets.py
    ├── schedules/daily_schedule.py
    └── definitions.py
```

## Running locally

```bash
# Run dbt against DuckDB (no Postgres needed)
cd datamcp_output/dbt/my_project
dbt run --target duckdb
dbt test --target duckdb

# Or run the full Dagster pipeline
cd datamcp_output/dagster_datamcp
dagster dev
```

## Contributing

See [CONTRIBUTING.md](docs/CONTRIBUTING.md).

## License

Apache 2.0
