Metadata-Version: 2.5
Name: md-ddl
Version: 0.9.2
Summary: Markdown Data Definition Language — an AI-native standard for domain ontology and semantic data models, with the agents that drive it
Project-URL: Homepage, https://github.com/Semprini/md-ddl
Project-URL: Repository, https://github.com/Semprini/md-ddl
Project-URL: Specification, https://github.com/Semprini/md-ddl/blob/main/md-ddl-specification/1-Foundation.md
Project-URL: Issues, https://github.com/Semprini/md-ddl/issues
Author: Paul Atkinson
License-Expression: CC-BY-4.0
License-File: LICENSE
Keywords: ai-agents,data-governance,data-mesh,data-model,data-modelling,data-products,ddl,markdown,ontology,semantic-model
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Documentation
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.9
Requires-Dist: pyyaml>=5.1
Description-Content-Type: text/markdown

# Markdown Data Definition Language (MD-DDL)

[![CC BY 4.0][cc-by-shield]][cc-by]

> **Version 0.9.2**

MD-DDL is a simple, standard way to organise domain ontology & semantic data models. Data modelling historically uses complex tools which have equally complex underlying data stores. These components are not very 'accessible' for both AI and humans.

md-ddl is: **AI‑native · Human‑friendly · Version‑controlled · Semantically rich · Ready for automation**

**Read the spec**: [1-Foundation.md](./md-ddl-specification/1-Foundation.md) or [MD-DDL-Complete.md](./md-ddl-specification/MD-DDL-Complete.md) for single-file AI context

---

## What MD-DDL covers

- **Domain layer** — domains, entities, enums, relationships, events, and constraints
- **Source layer** — source system declarations and column-level transformation rules (direct, derived, conditional, lookup, reconciliation, aggregation)
- **Data products** — source-aligned, domain-aligned, and consumer-aligned products declaring scope, shape, consumers, SLA, governance, and masking — driving automated artifact generation
- **Governance** — classification, PII, retention, regulatory scope, access roles, and masking strategies living with the model, not in a separate system
- **Physical artifacts** — dimensional star schemas, normalized 3NF DDL, wide-column schemas, knowledge graph (Cypher), JSON Schema, Parquet contracts
- **Synthetic data** — Python Faker factory classes generated straight from your entity definitions; source, canonical, or destination scope; safe-mode PII for shared environments, realistic mode for local dev

---

## Quick Start

**Install from PyPI** — the standard, its agents, and the linter ship as a Python package, so it can be pulled through a corporate artifactory like any other dependency.

```bash
pip install md-ddl
md-ddl init
```

`md-ddl init` unpacks the standard into `.md-ddl/` and installs the agent wrappers for Claude Code and GitHub Copilot, with their paths rewritten to resolve against the unpacked copy. Run it in an existing project or in an empty directory.

```text
md-ddl init [DIR] --ai claude|copilot|both   which wrappers to install (default: both)
                  --name NAME                project name for the generated instructions
                  --force                    overwrite existing wrappers and instructions
                  --no-instructions          skip CLAUDE.md / copilot-instructions.md
                  --track                    commit .md-ddl/ instead of ignoring it
```

By default `.md-ddl/` ignores itself, so the unpacked standard stays out of your history — upgrade it with `pip install --upgrade md-ddl && md-ddl init`.

`init` finishes by verifying that every `{{INCLUDE: ...}}` directive in the installed agent prompts resolves, so a prompt that cannot reach its spec section is reported at setup time instead of silently loading truncated. Re-run that check at any time with `md-ddl check`. The package also installs the pre-flight linter as `md-ddl lint <domain-folder>`.

The PyPI package carries the specification, agents, guides, examples, and the architecture references. It omits `references/industry_standards/` — the 63 MB of raw BIAN, FHIR and TM Forum snapshots — which stays in this repository; the distilled standards markdown the agents actually load ships with the agents.

---

**Or start a new project** using the bootstrap script — it sets up git, adds MD-DDL as a submodule, and installs the agent wrappers for your AI tool in one step.

Bash (macOS / Linux / WSL):

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/Semprini/md-ddl/main/scripts/start-project.sh)
```

PowerShell (Windows):

```powershell
Invoke-Expression (Invoke-WebRequest https://raw.githubusercontent.com/Semprini/md-ddl/main/scripts/start-project.ps1).Content
```

Or download [start-project.sh](./scripts/start-project.sh) / [start-project.ps1](./scripts/start-project.ps1) and run them locally.

---

**Learn by conversation**: MD-DDL includes **Agent Guide** an AI learning companion available from the repo via Claude or CoPilot in VS Code. It adapts to your role and goals, teaches through discussion rather than documentation, and routes you to the right specialist agent when you're ready to work.

Example prompts (Claude AI uses `/agent-guide`, CoPilot uses `@agent-guide`):

```text
/agent-guide I'm new to MD-DDL — walk me through the key concepts and help me get started.
```

```text
@agent-guide I'm a data architect at a retail bank. We have 15+ legacy source systems and no canonical data model. Give me an overview of MD-DDL and help me decide where to start.
```

```text
/agent-guide I need to model a Customer domain. We track individuals and business accounts. Walk me through the MD-DDL approach.
```

---

## Workflow

**md-ddl** is not rigid or dogmatic. A typical flow is:

1. **Position** — discuss the architectural approach with Agent Architect: compare to alternatives, prepare material for governance councils or CIOs
2. **Discover** — scope the domain with Agent Ontology: identify entities, relationships, events, and governance posture
3. **Model** — write domain.md, entity files, enums, and events
4. **Map sources** — declare source systems and column-level transforms
5. **Publish** — declare data products with scope, shape, SLA, and masking
6. **Generate** — produce physical artifacts and synthetic test data with Agent Artifact
7. **Govern** — audit standards conformance and regulatory posture with Agent Governance

Agent Guide helps you navigate between these stages and explains any concept along the way.

---

## Using MD-DDL in your project

Your model files live in your own repository; MD-DDL provides the specification, agents, and examples. Bring it in either as a pip dependency (`pip install md-ddl && md-ddl init`, above) or as a git submodule.

### Manual setup

If you prefer not to use the scripts and set up manually:

```bash
mkdir myproject
cd myproject
git init
git submodule add https://github.com/Semprini/md-ddl .md-ddl
git submodule update --init
```

Then copy the agent wrappers for your AI tool:

- **Copilot**: `.md-ddl/.github/agents/*.agent.md` → `.github/agents/`
- **Claude**: `.md-ddl/.claude/commands/*.md` → `.claude/commands/`

If you use Claude, you need to update `./claude/commands/*.md` files. The `agents/` path needs to be `.md-ddl/agents`

Next, create your `copilot-instructions.md` or `CLAUDE.md`. See the start project scripts for examples.

Update MD-DDL to a new release later:

```bash
git submodule update --remote .md-ddl
```

### Suggested project layout

```text
your-project/
  .md-ddl/                   ← submodule (this repo)
  .github/agents/            ← Copilot agent wrappers  (Copilot users)
  .claude/commands/          ← Claude slash commands    (Claude users)
  domains/
    customer/
      domain.md
      entities/
      products/
  sources/
    salesforce-crm/
      source.md
      transforms/
  generated/                 ← Agent Artifact output (commit alongside the model)
    customer/
      ddl/                   ← SQL DDL, JSON Schema, Parquet contracts, Cypher
      synthetic/
        factories.py         ← Faker factory classes
        test_factories.py    ← integrity and consistency tests
```

---

## Examples

Five reference domains at increasing complexity:

Example | Focus | Complexity
--- | --- | ---
[Simple Customer](examples/Simple%20Customer/domain.md) | Minimal — one domain, three entities, one event | Starter
[Financial Crime](examples/Financial%20Crime/domain.md) | AML/KYC/CTF — BIAN alignment, 15+ entities, sources, products, generated artifacts | Intermediate
[Healthcare](examples/Healthcare/domain.md) | FHIR R4 — HIPAA governance, source transforms, knowledge-graph product | Intermediate
[Telecom](examples/Telecom/domain.md) | TM Forum ODA — PCI-DSS, associative entities, new relationship types, dimensional product | Advanced
[Retail Sales + Retail Service](examples/Retail%20Sales/domain.md) | Bounded Context — two greenfield domains defining Customer differently, cross-domain Customer 360 | Advanced

The [feature coverage matrix](examples/README.md) maps every spec feature to the example that demonstrates it.

---

## Repository layout

```text
md-ddl-specification/        Normative standard
  1-Foundation.md            Start here to understand the model
  2-Domains.md … 10-Adoption.md
  MD-DDL-Complete.md         Single-file version for AI context windows

guides/                      Non-normative companion guides
  adoption-playbook.md       Brownfield adoption methodology
  diagram-style.md           Mermaid diagram conventions
  lifecycle-versioning.md    Change management and versioning process
  validation-tooling.md      Validation levels, pre-flight checks, tool interface

agents/                      Canonical agent prompts and skills
  agent-guide/               Learning companion and navigator
  agent-ontology/            Domain modelling and source mapping
  agent-artifact/            Physical schema generation
  agent-architect/           Architecture philosophy, data product design, ODPS
  agent-governance/          Standards conformance and compliance auditing

examples/                    Reference examples
  Simple Customer/
  Financial Crime/
  Healthcare/
  Telecom/
  Retail Sales/
  Retail Service/

references/                  Architecture and industry reference data
  industry_standards/        BIAN, FHIR, TM Forum reference datasets
  architecture/              Data Autonomy blog series, external references, Mermaid diagrams

src/md_ddl/                  The `md-ddl` PyPI package
  cli.py                     `md-ddl init` / `md-ddl lint` / `md-ddl path`
  lint.py                    Pre-flight linter (also `python scripts/md_ddl_lint.py`)
```

---

This work is licensed under a
[Creative Commons Attribution 4.0 International License][cc-by].

[![CC BY 4.0][cc-by-image]][cc-by]

[cc-by]: http://creativecommons.org/licenses/by/4.0/
[cc-by-image]: https://i.creativecommons.org/l/by/4.0/88x31.png
[cc-by-shield]: https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg
