Metadata-Version: 2.4
Name: dbt-conceptual
Version: 0.4.0
Summary: Bridge the gap between conceptual models and your lakehouse
Project-URL: Homepage, https://github.com/feriksen-personal/dbt-conceptual
Project-URL: Documentation, https://dbt-conceptual.readthedocs.io/
Project-URL: Repository, https://github.com/feriksen-personal/dbt-conceptual
Project-URL: Issues, https://github.com/feriksen-personal/dbt-conceptual/issues
Project-URL: Changelog, https://github.com/feriksen-personal/dbt-conceptual/blob/main/CHANGELOG.md
Author-email: Fridthjof-Greger Eriksen <fridthjof.eriksen@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: conceptual-model,data-architecture,data-modeling,databricks,dbt,lakehouse,medallion,snowflake
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Documentation
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: click>=8.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: all
Requires-Dist: black>=24.0; extra == 'all'
Requires-Dist: flask>=3.0; extra == 'all'
Requires-Dist: mkdocs-material>=9.0; extra == 'all'
Requires-Dist: mkdocs>=1.5; extra == 'all'
Requires-Dist: mypy>=1.0; extra == 'all'
Requires-Dist: pillow>=10.0; extra == 'all'
Requires-Dist: pytest-cov>=4.0; extra == 'all'
Requires-Dist: pytest>=7.0; extra == 'all'
Requires-Dist: ruff>=0.1.0; extra == 'all'
Requires-Dist: types-pyyaml; extra == 'all'
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0; extra == 'docs'
Requires-Dist: mkdocs>=1.5; extra == 'docs'
Provides-Extra: png
Requires-Dist: pillow>=10.0; extra == 'png'
Provides-Extra: serve
Requires-Dist: flask>=3.0; extra == 'serve'
Description-Content-Type: text/markdown

# dbt-conceptual

[![PyPI version](https://img.shields.io/pypi/v/dbt-conceptual.svg)](https://pypi.org/project/dbt-conceptual/)
[![Python versions](https://img.shields.io/pypi/pyversions/dbt-conceptual.svg)](https://pypi.org/project/dbt-conceptual/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/feriksen-personal/dbt-conceptual/actions/workflows/ci.yml/badge.svg)](https://github.com/feriksen-personal/dbt-conceptual/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/feriksen-personal/dbt-conceptual/branch/main/graph/badge.svg)](https://codecov.io/gh/feriksen-personal/dbt-conceptual)
[![Downloads](https://img.shields.io/pypi/dm/dbt-conceptual.svg)](https://pypi.org/project/dbt-conceptual/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**Bridge the gap between conceptual models and your lakehouse.**

---

## ⚠️ Opinionated by Design

This tool is built for a specific architecture. It expects:

- **dbt** — Your transformation layer
- **Medallion architecture** — Silver (cleaned/conformed) → Gold (business/integrated)
- **Dimensional modeling** — dims, facts, bridges in gold

**What we're opinionated about:**
- Naming conventions (`dim_`, `fact_`, `bridge_` prefixes)
- Layer structure (silver/gold concepts)
- Conceptual model in a single `conceptual.yml` file

**What we're flexible about:**
- Whether you use one `schema.yml` per model or shared schema files
- Your specific silver/gold path names (configurable)
- Whether you use dbt groups, tags, or other organizational features

If that's not your stack, this tool is not for you. No judgment — just not the target audience.

---

## The Problem

Every data team has a conceptual model — *"we have Customers, Orders, Products"* — but it lives in PowerPoint, Visio, ERD tools, or people's heads. Disconnected from code. Diverging from reality. Rotting.

```
CONCEPTUAL MODEL                          LAKEHOUSE
(slides, whiteboards, heads)              (dbt, Databricks, Snowflake)

  Customer ── Order ── Product    →→→     dim_customer_shopify, dim_customer_crm,
                                          dim_product, fact_order_lines,
        🤷 GAP 🤷                          bridge_customer_segment...
```

When someone asks *"Do we have Customer data?"* — you grep through 200 models hoping to find the answer.

## The Solution

**dbt-conceptual** tracks which business concepts your dbt models implement:

```yaml
# models/conceptual/conceptual.yml
concepts:
  customer:
    domain: party
    owner: customer_team
    definition: "A person or company that purchases products"

  order:
    domain: transaction
    owner: orders_team
    definition: "A confirmed purchase by a customer"

relationships:
  - name: places
    from: customer
    to: order

  - name: contains
    from: order
    to: product
```

```yaml
# models/gold/dim_customer/dim_customer.yml
models:
  - name: dim_customer
    meta:
      concept: customer

# models/gold/fact_order_lines/fact_order_lines.yml
models:
  - name: fact_order_lines
    meta:
      realizes:
        - customer:places:order
        - order:contains:product
```

Then visualize, validate, and export:

```
┌──────────────────────┐         ┌──────────────────────┐         ┌──────────────────────┐
│      Customer        │         │        Order         │         │       Product        │
│                      │         │                      │         │                      │
│   [S:●●] [G:●]       │━places━▶│   [S:●]  [G:●]       │━contains▶│   [S:●]  [G:●]      │
└──────────────────────┘         └──────────────────────┘         └──────────────────────┘
```

---

## Installation

```bash
# No signup required. No telemetry. No "please star this repo" popups.
pip install dbt-conceptual
```

## Quick Start

Try it with the [jaffle-shop](https://github.com/dbt-labs/jaffle-shop) demo project:

```bash
# Clone jaffle-shop
git clone https://github.com/dbt-labs/jaffle-shop.git
cd jaffle-shop

# Install dbt-conceptual
pip install dbt-conceptual[serve]

# Initialize conceptual model
dbt-conceptual init
# This creates models/conceptual/conceptual.yml

# Define your business concepts (see example below)
# Edit models/conceptual/conceptual.yml

# View coverage
dbt-conceptual status

# Launch interactive UI
dbt-conceptual serve

# Validate in CI
dbt-conceptual validate

# Export diagrams
dbt-conceptual export --format excalidraw -o diagram.excalidraw
```

**Example conceptual.yml for jaffle-shop:**
```yaml
version: 1

domains:
  party:
    name: Party
    color: "#E3F2FD"
  transaction:
    name: Transaction
    color: "#FFF3E0"

concepts:
  customer:
    name: Customer
    domain: party
    definition: "A person or entity that places orders"
    status: complete

  order:
    name: Order
    domain: transaction
    definition: "A purchase transaction made by a customer"
    status: complete

relationships:
  - name: places
    from: customer
    to: order
    cardinality: "1:N"
```

Then tag your dbt models:
```yaml
# models/customers.yml
models:
  - name: customers
    meta:
      concept: customer

# models/orders.yml
models:
  - name: orders
    meta:
      realizes:
        - customer:places:order
```

---

## Features

### 📊 Coverage Tracking

See which concepts are implemented at each layer:

```
$ dbt-conceptual status

Concepts by Domain
==================

party (1 concept)
  ✓ customer [complete]     [S:●●] [G:●]

transaction (2 concepts)
  ✓ order [complete]        [S:●]  [G:●]
  ⚠ shipment [stub]         [S:○]  [G:○]

catalog (1 concept)
  ✓ product [complete]      [S:●]  [G:●]
```

### 🎨 Interactive Web UI

Launch a visual editor for your conceptual model with real-time editing:

```bash
$ dbt-conceptual serve
Starting dbt-conceptual UI server...
Open your browser to: http://127.0.0.1:5000
```

**Features:**
- **Graph Editor** — Interactive force-directed graph visualization with D3.js
  - Drag and position concepts
  - Click concepts/relationships to edit
  - Real-time visual updates
  - Domain-based coloring
  - Zoom and pan canvas
- **Direct Editing** — Changes save directly to `conceptual.yml`
  - Edit concept name, description, definition, domain, owner, status
  - Edit relationship name, from/to concepts, cardinality, description
  - No sync needed - what you see is what gets saved
- **Integrated Reports** — View coverage and bus matrix in tabs
  - Coverage Report tab shows concept completion and implementations
  - Bus Matrix tab shows which fact tables realize which relationships
  - Switch between Editor, Coverage, and Bus Matrix views

**Installation:**
```bash
# UI requires Flask
pip install dbt-conceptual[serve]

# Or install Flask separately
pip install flask
```

### ✅ CI Validation

Catch drift before it ships:

```bash
$ dbt-conceptual validate

ERROR: dim_customer_legacy references concept 'client' which does not exist
       Did you mean: 'customer'?

ERROR: fact_returns realizes 'customer:returns:order' 
       but relationship 'returns' does not exist
```

### 📤 Export Formats

```bash
# Excalidraw — editable diagrams
dbt-conceptual export --format excalidraw

# PNG — static diagram image
dbt-conceptual export --format png -o diagram.png

# Mermaid — for docs and GitHub
dbt-conceptual export --format mermaid

# Coverage report — HTML dashboard
dbt-conceptual export --format coverage

# Bus matrix — dimensions vs facts
dbt-conceptual export --format bus-matrix
```

**PNG Export Note:** Requires Pillow. Install with `pip install dbt-conceptual[png]`

---

## How It Works

### 1. Define Concepts

Create `models/conceptual/conceptual.yml`:

```yaml
version: 1

domains:
  party:
    name: "Party"
    color: "#E3F2FD"
  transaction:
    name: "Transaction"
    color: "#FFF3E0"
  catalog:
    name: "Catalog"
    color: "#F1F8E9"

concepts:
  customer:
    name: "Customer"
    domain: party
    owner: customer_team
    definition: "A person or company that purchases products"
    status: complete

  order:
    name: "Order"
    domain: transaction
    owner: orders_team
    definition: "A confirmed purchase by a customer"
    status: complete

  product:
    name: "Product"
    domain: catalog
    owner: catalog_team
    definition: "An item available for purchase"
    status: complete

relationships:
  - name: places
    from: customer
    to: order
    cardinality: "1:N"

  - name: contains
    from: order
    to: product
    cardinality: "N:M"
```

### 2. Tag dbt Models

Add `meta.concept` to dimensions:

```yaml
# models/gold/dim_customer/dim_customer.yml
version: 2
models:
  - name: dim_customer
    description: "Customer dimension"
    meta:
      concept: customer
```

Add `meta.realizes` to facts and bridges:

```yaml
# models/gold/fact_order_lines/fact_order_lines.yml
version: 2
models:
  - name: fact_order_lines
    description: "Order line items fact"
    meta:
      realizes:
        - customer:places:order
        - order:contains:product
```

### 3. Validate & Visualize

```bash
# Check everything links up
dbt-conceptual validate

# See coverage
dbt-conceptual status

# Open visual editor
dbt-conceptual serve
```

---

## Bottom-Up Adoption

Already have a dbt project? Start tagging models, then generate the conceptual model:

```bash
# Add tags to your existing schema.yml files
# Then run:
dbt-conceptual sync --create-stubs

# Output:
# Created 12 concept stubs
# Created 8 relationship stubs
# 
# Run 'dbt-conceptual status' to see what needs enrichment
```

The tags ARE the adoption. The tool just makes them visible.

---

## Configuration

Works out of the box. Override in `dbt_project.yml` if needed:

```yaml
vars:
  dbt_conceptual:
    conceptual_path: models/conceptual    # default
    silver_paths:
      - models/silver                     # default
      - models/staging                    # add custom paths
    gold_paths:
      - models/gold                       # default
      - models/marts                      # add custom paths
```

Or via CLI:

```bash
dbt-conceptual validate --gold-paths models/marts
```

---

## CI/CD Integration

```yaml
# .github/workflows/ci.yml
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install dbt-core dbt-conceptual

      - name: Validate conceptual model
        run: dbt-conceptual validate --format github
```

Use `--format github` for native GitHub Actions annotations. Errors and warnings will appear inline in PR diffs.

### Validation Rules Configuration

Configure which validation rules are errors, warnings, or ignored in `dbt_project.yml`:

```yaml
vars:
  dbt_conceptual:
    validation:
      orphan_models: warn          # Models not linked to any concept
      unimplemented_concepts: warn # Concepts with no implementing models
      unrealized_relationships: warn # Relationships with no realizing models
      missing_definitions: ignore  # Concepts without definitions
      domain_mismatch: warn        # Models with meta.domain != concept domain
```

Severity options: `error`, `warn`, `ignore`

**Note:** Unknown references (e.g., `meta.concept: nonexistent`) are always errors and cannot be configured.

---

## Documentation

- [Full Documentation](https://dbt-conceptual.readthedocs.io/)
- [Configuration Reference](docs/configuration.md)
- [CLI Reference](docs/cli.md)
- [Export Formats](docs/exports.md)

---

## Contributing

Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

PRs that work > PRs with extensive documentation about why they might work.

```bash
# Development setup
git clone https://github.com/feriksen-personal/dbt-conceptual.git
cd dbt-conceptual
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check .
black --check .
```

---

## License

MIT License. See [LICENSE](LICENSE) for details.

---

## Acknowledgments

Built with frustration from years of conceptual models rotting in PowerPoint, Visio, and expensive ERD software.

No mass emails were sent to coordinate this project. No penguins were harmed in its making.

---

<p align="center">
  <sub>Works on my machine. Might work on yours.</sub>
</p>
