Metadata-Version: 2.4
Name: lineagemap
Version: 0.1.0
Summary: Column-level data lineage for dbt projects. One command, no catalog required.
Project-URL: Homepage, https://github.com/christianquebral/lineagemap
Project-URL: Repository, https://github.com/christianquebral/lineagemap
Project-URL: Issues, https://github.com/christianquebral/lineagemap/issues
Author-email: Christian Quebral <christianquebral@gmail.com>
License: Business Source License 1.1
        
        Parameters
        
        Licensor:             Christian Quebral
        Licensed Work:        LineageMap
                              The Licensed Work is (c) 2026 Christian Quebral
        Additional Use Grant: You may make production use of the Licensed Work, provided
                              such use does not include offering the Licensed Work to
                              third parties on a hosted or embedded basis in order to
                              compete with Licensor's paid version of the Licensed Work.
        Change Date:          2030-06-26
        Change License:       Apache License, Version 2.0
        
        -----------------------------------------------------------------------------
        
        Business Source License 1.1
        
        License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
        "Business Source License" is a trademark of MariaDB Corporation Ab.
        
        -----------------------------------------------------------------------------
        
        Terms
        
        The Licensor hereby grants you the right to copy, modify, create derivative
        works, redistribute, and make non-production use of the Licensed Work. The
        Licensor may make an Additional Use Grant, above, permitting limited production
        use.
        
        Effective on the Change Date, or the fourth anniversary of the first publicly
        available distribution of a specific version of the Licensed Work under this
        License, whichever comes first, the Licensor hereby grants you rights under
        the terms of the Change License, and the rights granted in the paragraph
        above terminate.
        
        If your use of the Licensed Work does not comply with the requirements
        currently in effect as described in this License, you must purchase a
        commercial license from the Licensor, its affiliated entities, or authorized
        resellers, or you must refrain from using the Licensed Work.
        
        All copies of the original and modified Licensed Work, and derivative works
        of the Licensed Work, are subject to this License. This License applies
        separately for each version of the Licensed Work and the Change Date may vary
        for each version of the Licensed Work released by Licensor.
        
        You must conspicuously display this License on each original or modified copy
        of the Licensed Work. If you receive the Licensed Work in original or modified
        form from a third party, the terms and conditions set forth in this License
        apply to your use of that work.
        
        Any use of the Licensed Work in violation of this License will automatically
        terminate your rights under this License for the current and all other versions
        of the Licensed Work.
        
        This License does not grant you any right in any trademark or logo of Licensor
        or its affiliates (provided that you may use a trademark or logo of Licensor
        as expressly required by this License).
        
        TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
        AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
        EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
        MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
        TITLE.
        
        MariaDB hereby grants you permission to use this License's text to license
        your works, and to refer to it using the trademark "Business Source License",
        as long as you comply with the Covenants of Licensor below.
        
        Covenants of Licensor
        
        In consideration of the right to use this License's text and the "Business
        Source License" name and trademark, Licensor covenants to MariaDB, and to all
        recipients of the Licensed Work to be provided by Licensor:
        
        1. To specify as the Change License the GPL Version 2.0 or any later version,
           or a license that is compatible with GPL Version 2.0 or a later version,
           where "compatible" means that software provided under the Change License can
           be included in a program with software provided under GPL Version 2.0 or a
           later version. Licensor may specify additional Change Licenses without
           limitation.
        
        2. To either: (a) specify an additional grant of rights to use that does not
           impose any additional restriction on the right granted in this License, as
           the Additional Use Grant; or (b) insert the text "None".
        
        3. To specify a Change Date.
        
        4. Not to modify this License in any other way.
License-File: LICENSE
Keywords: analytics engineering,data engineering,data lineage,dbt,sql
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: click>=8.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sqlglot>=20.0.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Provides-Extra: server
Requires-Dist: fastapi>=0.110.0; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.29.0; extra == 'server'
Description-Content-Type: text/markdown

# LineageMap

Column-level data lineage for dbt projects. One command, no catalog required.

![LineageMap demo](docs/demo.gif)

```
$ lineagemap trace --column revenue

revenue (fct_orders)
└── revenue (stg_orders)
    └── amount (raw.orders)
```

## Install

```bash
pip install lineagemap
```

For the web UI:

```bash
pip install "lineagemap[server]"
```

## Usage

Run `dbt compile` first, then:

```bash
# Interactive web UI at http://localhost:3000
lineagemap serve

# Trace a column in the terminal
lineagemap trace --column revenue

# Scope to a specific model
lineagemap trace --column revenue --model fct_orders

# Export full lineage graph as JSON
lineagemap trace --column revenue --json

# Custom manifest path or SQL dialect
lineagemap trace --column revenue --manifest path/to/manifest.json --dialect snowflake
```

### Web UI

Hover any node to see its full upstream (blue) and downstream (red) lineage instantly. Use the search bar to filter by column or model name.

## Self-hosting

LineageMap can be self-hosted so your whole team can access it in a browser, without anyone needing to install Python or run `dbt compile` locally.

### Prerequisites

- [Docker](https://docs.docker.com/get-docker/) installed
- A `manifest.json` from your dbt project (generated by `dbt compile`)

### Option 1: Docker (quickest)

```bash
docker run -p 3000:3000 \
  -v $(pwd)/manifest.json:/manifest.json:ro \
  ghcr.io/christianquebral/lineagemap:latest
```

Then open [http://localhost:3000](http://localhost:3000).

To use a different port:

```bash
docker run -p 8080:3000 \
  -v $(pwd)/manifest.json:/manifest.json:ro \
  ghcr.io/christianquebral/lineagemap:latest
```

### Option 2: Docker Compose (recommended for persistent deploys)

1. Copy `docker-compose.yml` from this repo into your project directory
2. Run `dbt compile` to generate `manifest.json`
3. Start the server:

```bash
docker compose up -d
```

To point at a manifest in a different location:

```bash
MANIFEST_PATH=/path/to/your/manifest.json docker compose up -d
```

To use a different port:

```bash
PORT=8080 docker compose up -d
```

Stop the server:

```bash
docker compose down
```

### Option 3: pip install

Install the package, then serve directly from your dbt project directory:

```bash
pip install "lineagemap[server]"
dbt compile
lineagemap serve --host 0.0.0.0 --manifest target/manifest.json
```

Share `http://<your-server-ip>:3000` with your team.

By default (`--host 127.0.0.1`) the server is localhost-only. Pass `--host 0.0.0.0` to make it network-accessible.

To keep it running after you close the terminal:

```bash
# Using nohup
nohup lineagemap serve --host 0.0.0.0 --manifest target/manifest.json &

# Or with screen
screen -S lineagemap
lineagemap serve --host 0.0.0.0 --manifest target/manifest.json
# Detach with Ctrl+A, D
```

For a more robust setup (auto-restart on reboot), use Docker Compose (Option 2) or configure a systemd service.

### Keeping lineage up to date automatically

LineageMap exposes a `POST /api/reload` endpoint that re-parses the manifest and updates every connected browser — no restart needed.

**1. Set a reload token on your server**

This protects the endpoint so only your CI pipeline can trigger a reload.

In your `.env` or shell:

```bash
export LINEAGEMAP_RELOAD_TOKEN=your-secret-token
```

With Docker Compose:

```bash
LINEAGEMAP_RELOAD_TOKEN=your-secret-token docker compose up -d
```

**2. Trigger a reload from CI**

After `dbt compile` runs in your dbt project's CI pipeline, add a step to notify LineageMap:

```bash
curl -X POST https://lineagemap.your-company.com/api/reload \
  -H "Authorization: Bearer your-secret-token"
```

Or use the ready-made GitHub Action — copy `.github/workflows/lineage-sync.yml` from this repo into your dbt project and add two repository secrets:

| Secret | Value |
|---|---|
| `LINEAGEMAP_URL` | `http://lineagemap.your-company.com:3000` |
| `LINEAGEMAP_RELOAD_TOKEN` | your secret token |

The workflow fires automatically on pushes to `main` that touch `models/` or `dbt_project.yml`.

### Protecting access with basic auth

To require a username and password before anyone can view the dashboard, use the included nginx overlay.

**1. Generate a password file**

```bash
# macOS
brew install httpd && htpasswd -c .htpasswd <username>

# Linux
apt install apache2-utils && htpasswd -c .htpasswd <username>
```

**2. Start with auth enabled**

```bash
docker compose -f docker-compose.yml -f docker-compose.auth.yml up -d
```

Your team accesses `http://your-server:3000` and is prompted for credentials. The `/api/reload` endpoint bypasses basic auth so CI can still trigger reloads using only the Bearer token.

## How it works

1. Reads `manifest.json` from `dbt compile` (no live warehouse connection needed)
2. Parses each model's compiled SQL with [sqlglot](https://github.com/tobymao/sqlglot) to extract column-level dependencies
3. Builds a column lineage graph you can explore in the terminal or web UI

## Why LineageMap

**Honest landscape note:** dbt Core v2.0 (Fusion) now ships column-level lineage in the OSS core, and [dbt-column-lineage](https://github.com/nickvdyck/dbt-column-lineage) covers similar ground with the same sqlglot approach. The CLI problem is largely solved.

What remains unoccupied: a **polished web UI** and a **hosted/shareable tier** (upload a manifest, send a stakeholder a link, no local setup required). That is what LineageMap is building toward.

|                              | LineageMap    | dbt Fusion    | dbt-column-lineage | dbt Cloud | DataHub      |
|------------------------------|:-------------:|:-------------:|:------------------:|:---------:|:------------:|
| Column-level lineage         | ✅             | ✅             | ✅                  | ✅         | ✅            |
| One-command setup            | ✅             | ✅ (built-in)  | ✅                  | N/A       | ❌            |
| Interactive web UI           | ✅             | ❌             | ❌                  | ✅         | ✅            |
| Shareable URL                | Coming (v0.3) | ❌             | ❌                  | ✅         | ❌            |
| No warehouse connection      | ✅             | ✅             | ✅                  | ✅         | ❌            |
| Self-hostable                | ✅             | ✅             | ✅                  | ❌         | ✅ (complex)  |
| Cost                         | Free          | Free          | Free               | $$$       | Free (DIY)   |

## Roadmap

- [x] Phase 1: CLI + SQL parser (`lineagemap trace`)
- [x] Phase 2: Local web UI with interactive DAG visualization
- [ ] Phase 3: Hosted tier: upload `manifest.json` → get a shareable URL
- [ ] Phase 4: GitHub Action: post updated lineage URL on every dbt PR

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). Good first issues are tagged in the tracker.

## License

[Business Source License 1.1](LICENSE). Converts to Apache 2.0 on 2030-06-26. Free for personal, internal, and non-competing production use.
