Metadata-Version: 2.3
Name: dbt-col-lineage
Version: 0.10.0
Summary: Impact analysis for dbt projects using column-level data lineage
License: MIT
Author: Fszta
Requires-Python: >=3.9
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: click (>=8.1.0,<9.0.0)
Requires-Dist: fastapi (>=0.115.12,<0.116.0)
Requires-Dist: graphviz (>=0.20.3,<0.21.0)
Requires-Dist: pydantic (>=2.10.6,<3.0.0)
Requires-Dist: sqlglot (>=26.8.0,<27.0.0)
Requires-Dist: uvicorn (>=0.34.0,<0.35.0)
Description-Content-Type: text/markdown

# DBT Column Lineage

[![Tests](https://github.com/Fszta/dbt-column-lineage/actions/workflows/test.yml/badge.svg)](https://github.com/Fszta/dbt-column-lineage/actions/workflows/test.yml)
[![PyPI Downloads](https://img.shields.io/pypi/dm/dbt-col-lineage?style=flat-square&logo=pypi)](https://pypi.org/project/dbt-col-lineage/)
[![Python Version](https://img.shields.io/pypi/pyversions/dbt-col-lineage?style=flat-square&logo=python)](https://pypi.org/project/dbt-col-lineage/)
[![License](https://img.shields.io/github/license/Fszta/dbt-column-lineage?style=flat-square)](LICENSE)

📖 **[Documentation](https://fszta.github.io/dbt-column-lineage/)** | 🚀 [Live Demo](https://dbt-column-lineage.onrender.com) | 🐛 [Report Bug](https://github.com/Fszta/dbt-column-lineage/issues) | 💡 [Request Feature](https://github.com/Fszta/dbt-column-lineage/issues)

## Overview

**Understand the impact of column changes before you make them.** DBT Column Lineage provides impact analysis & column lineage to help you answer critical questions: What happens if I change this column? Which models, transformations, and exposures will be affected? Does it use complex business logic, or is this column only projected?

The core idea behind this project: be more comfortable when making changes on a large project by knowing the impact.

**Key Features:**
- **Impact Analysis**: Understand downstream effects of column changes, including affected models, transformations, and exposures
- **Interactive Explorer**: A local web server with an intuitive UI to explore model and column lineage visually, including associated transformations



![DBT Column Lineage Demo - Concept](assets/demo_lineage.gif)


## Installation

```bash
pip install dbt-col-lineage
```

## Usage

First, ensure your dbt project is compiled and you have generated the catalog:

```bash
dbt compile
dbt docs generate
```

### Interactive Explorer

Start the interactive web interface to explore lineage and perform impact analysis:

```bash
dbt-col-lineage --explore \
    --manifest path/to/manifest.json \
    --catalog path/to/catalog.json \
    --port 8080  # Optional port selection
```

This starts a server (defaulting to port 8000). Open your web browser to the specified address (e.g., `http://127.0.0.1:8080`).

**Workflow:**
1. Select a model and column from the sidebar to visualize its lineage
2. Click **"Analyze Impact"** to see which models, transformations, and exposures will be affected if you change that column
3. Review the impact analysis panel showing:
   - Columns that require review (with transformations that may break)
   - Pass-through columns (direct references)
   - Affected models in the dependency chain
   - Affected exposures

### Machine-readable output (AI / automation)

Emit column lineage and downstream impact analysis as a single JSON document —
ideal for feeding AI agents, CI checks, or other tooling:

```bash
dbt-col-lineage --select stg_accounts.account_id --format json \
    --manifest path/to/manifest.json --catalog path/to/catalog.json
```

The document contains the column's `upstream` and `downstream` lineage (each split
into `models`, `sources`, `direct_refs`, `exposures`) and, when downstream lineage
is requested, an `impact` block summarising affected models, columns and exposures.

> Note: the tool works even when your `manifest.json` doesn't embed `compiled_code`
> (e.g. produced by `dbt parse`), as long as `target/compiled/**` exists — it falls
> back to the compiled SQL on disk.

### Diff-driven impact (`impact`)

The unit of work in a PR is a *set* of changed columns, not a single one. The
`impact` command derives that changeset and reports one consolidated blast radius.

Two-manifest diff (the reliable, dbt-native signal — compares the base branch's
artifacts against the current ones for added / removed / retyped columns and
changed compiled SQL):

```bash
dbt-col-lineage impact \
    --manifest target/manifest.json --catalog target/catalog.json \
    --base-manifest base/manifest.json --base-catalog base/catalog.json
```

Git-diff fallback (when only one manifest is available — diffs changed `.sql`
model files against a git ref and treats touched models as logic changes):

```bash
dbt-col-lineage impact --git-base main
```

Output defaults to a human-readable Markdown summary (exposures first, then a
blast-radius table). Use `--format json` for a machine-readable report — a
superset of the single-column `impact` block plus a top-level `changeset` and a
per-change `by_change` breakdown.

### CI check + PR comment (`impact --ci`)

`--ci` turns the report into a pull-request check: it posts a **sticky Markdown
comment** (found-or-updated by a hidden marker, so re-runs edit one comment
instead of spamming the thread) and applies a **severity gate** as the process
exit code.

```bash
dbt-col-lineage impact --ci \
    --manifest target/manifest.json --catalog target/catalog.json \
    --base-manifest base/manifest.json --base-catalog base/catalog.json \
    --fail-on exposures
```

- `--fail-on`: gate policy — `none` (default: warn only, never block),
  `exposures` (fail when a business-facing exposure is affected), `critical`
  (fail when a downstream column recomputes derived logic), or `any`.
- The PR context (token, repo, PR number) is resolved from the standard GitHub
  Actions environment (`GITHUB_TOKEN`, `GITHUB_REPOSITORY`, `GITHUB_EVENT_PATH`);
  override with `--github-token` / `--repo` / `--pr-number`. When no context is
  available the comment is skipped and only the gate runs.

**GitHub Action.** A ready-to-use composite action ships in this repo. Pin the
floating **major** tag to get updates within a major version (the usual
convention, like `actions/checkout@v4`); the tool is currently `0.x`, so that
tag is `@v0`:

```yaml
- uses: Fszta/dbt-column-lineage@v0
  with:
    manifest: artifacts/head/manifest.json
    catalog: artifacts/head/catalog.json
    base-manifest: artifacts/base/manifest.json
    base-catalog: artifacts/base/catalog.json
    fail-on: none # start non-blocking; flip to exposures|critical once trusted
```

For reproducible builds, pin an exact release instead — `@v0.8.0` (the same tag
as the pip package). The action installs the source bundled at whichever ref you
pin, so the CLI always matches it.

The workflow needs `permissions: pull-requests: write` to post the comment. A
complete example workflow (building base- and head-branch artifacts) lives at
[`docs/examples/impact-pr-check.yml`](docs/examples/impact-pr-check.yml).

### Options

- `--explore`: Starts the interactive web server for exploring lineage and impact analysis
- `--catalog`: Path to the dbt catalog file (default: `target/catalog.json`)
- `--manifest`: Path to the dbt manifest file (default: `target/manifest.json`)
- `--port`, `-p`: Port for the interactive web server (default: `8000`)
- `--adapter`: Override the SQL dialect used by the parser (sqlglot dialect name, e.g., `tsql`, `snowflake`, `bigquery`). When provided, this overrides the adapter detected from the dbt manifest.
- `--format`, `-f`: Output format for `--select`: `text` (default), `dot`, or `json`.

For the `impact` subcommand:

- `--base-manifest` / `--base-catalog`: base-branch artifacts for the two-manifest diff.
- `--git-base`: git ref to diff changed `.sql` files against (git-diff fallback).
- `--scope-git`: with a two-manifest diff, restrict the report to models changed in `git diff <ref>...HEAD` — drops changes on models the branch didn't touch (useful when the base artifacts may be staler than the base ref).
- `--ci`: post a sticky PR comment and apply the `--fail-on` severity gate as an exit code.
- `--fail-on`: gate policy — `none` (default), `exposures`, `critical`, or `any`.
- `--github-token` / `--repo` / `--pr-number`: override the GitHub context (defaults from the Actions env).

## Limitations
- Doesn't support python models
- Some functions/syntax cannot be parsed properly, leading to models being skipped

## Compatibility

The tool has been tested with the following dbt adapters:
- Snowflake
- SQLite
- DuckDB
- MS SQLServer / TSQL


## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

