Skip to content

# Column-Level Lineage

Provisa tracks column-level data lineage statically — computed from SQL definitions and command contracts, no execution required. Two views are available: a per-statement DAG and a federation-wide provenance graph spanning all registered views and materialized views (MVs).

## The lineage explorer

Navigate to Lineage in the UI (`/lineage`). Paste a SQL statement and click Build statement graph to see its column-level DAG. Click Federation graph to load the provenance graph over every MV in the registry. [tool-verified: LineagePage.tsx:28-119]

## Statement-level DAG (REQ-1160)

Each named output column in your SQL becomes a node. The builder traces it back through every CTE, subquery, join, and inline command call to its source columns, building a directed graph from source inputs to final outputs.

### Worked example

    SELECT o.id, e.embedding, upper(e.geo) AS geo_u
    FROM   orders o
    JOIN   enrich_grpc_set('main.public.orders') e ON o.id = e.id

This statement produces three output columns. The graph for `geo_u` looks like:

    orders.geo  ──[enrich_grpc_set(...)]──►  e.geo  ──[UPPER]──►  geo_u
    orders.id   ─╮                                              (taint closure)
    orders.region ─╯

- `orders.id`, `orders.region`, and `orders.geo` are source nodes (the narrow input contract of `enrich_grpc_set` declares `id` and `region`; the full taint-closure connects all declared inputs to all outputs). [tool-verified: `_splice_commands` in graph.py:223-242]
- `e.embedding` and `e.geo` are command nodes — the `enrich_grpc_set` boundary.
- `geo_u` is a derived node produced by the `UPPER` SQL function.

The command boundary is not opaque. Because `enrich_grpc_set` declares its input columns (`id`, `region`) and output columns (`id`, `embedding`, `geo`), the lineage engine splices the taint closure continuously from the source relation's declared columns to each output. [tool-verified: `_splice_commands` and `_input_relation` in graph.py:245-271]

### Node kinds and visual cues

[tool-verified: LineageDag.tsx:25-29, KIND_COLOR constants; LineagePage.tsx:21-26 LEGEND]

Node kind | Color | Meaning
--- | --- | ---
`source` | Green | A base table column
`derived` | Blue | Produced by a SQL expression (function, operator, CTE)
`command` | Purple | An output column from a registered command

Additional rings on a node:

- Orange ring — a final output column of the statement.
- Double border — the column's relation is a materialized view (MV/CTAS snapshot).
- Red ring — member of a cycle classified as an error.
- Yellow ring — member of a cycle classified as a feedback loop.

[tool-verified: LineageDag.tsx:88-103 Cytoscape style selectors]

### Named transforms on edges

Every edge carries the raw SQL expression that produces the target column, plus a list of named operations: SQL functions (`sql_function`), arithmetic/logical operators (`operator`), registered commands (`command`), bare column references (`identity`), and literals (`constant`). [tool-verified: TransformOp and name_transform in graph.py:36-145]

An edge from a command call is rendered as a dashed purple line in the UI. [tool-verified: LineageDag.tsx:122-124]

## Federation-wide graph (REQ-1161)

The federation graph merges every registered MV's per-statement lineage into one provenance graph. Node identity is `relation.column` — a view's output column and another view's input reference to the same column collapse to one node. The result is a single DAG from base source columns to every derived dataset in the platform. [tool-verified: `build_federation_graph` in merge.py:205-229 and `qualify_outputs` in graph.py:275-299]

Use `focus`, `direction`, and `depth` to scope the view at federation scale without recomputing the graph. [tool-verified: `slice_graph` in merge.py:160-189]

## Cycles (REQ-1161)

Cycles are described, not rejected. The lineage engine detects every directed cycle and classifies it. [tool-verified: `Cycle.classification` property in merge.py:43-46]

Classification | Border color | Meaning
--- | --- | ---
`feedback` | Yellow | The cycle crosses a materialized node — a legal, time-lagged feedback loop. The MV snapshot is the version boundary that makes it well-defined.
`error` | Red | No materialization boundary on the loop — a circular definition with no stable evaluation order. Likely a design error.

[tool-verified: LineagePage.tsx:83-98 cycle alert rendering; merge.py:38-48]

A `feedback` cycle is not a failure. An enrichment MV that feeds back a derived column into its own source relation is a valid pattern as long as one node on the loop is materialized — the snapshot isolates the two halves temporally. An `error` cycle needs operator judgment: it usually means two views reference each other with no snapshot in between.

## API

Both endpoints are static — they read definitions and contracts, not data.

### POST /admin/lineage/graph

Returns the column-level DAG for a single SQL statement.

    POST /admin/lineage/graph
    Content-Type: application/json

    {
      "sql": "SELECT o.id, e.embedding FROM orders o JOIN enrich_grpc_set('main.public.orders') e ON o.id = e.id",
      "dialect": "postgres"
    }

[tool-verified: `lineage_graph` endpoint at lineage_router.py:45-54, LineageGraphRequest model at lineage_router.py:29-31]

Response shape [tool-verified: `LineageGraph.to_dict` in graph.py:82-105]:

    {
      "nodes": [
        {"id": "orders.id", "column": "id", "relation": "orders", "kind": "source", "materialized": false}
      ],
      "edges": [
        {
          "source": "orders.id",
          "target": "e.id",
          "transform": "enrich_grpc_set(...)",
          "ops": [{"name": "enrich_grpc_set", "kind": "command"}]
        }
      ],
      "outputs": ["id", "embedding"]
    }

Returns HTTP 422 when the SQL cannot be parsed. [tool-verified: lineage_router.py:51-54]

### GET /admin/lineage/federation

Returns the merged provenance graph over all MVs in the registry.

    GET /admin/lineage/federation
    GET /admin/lineage/federation?focus=orders.id&direction=downstream&depth=3

[tool-verified: `federation_graph` endpoint at lineage_router.py:73-98]

Query parameters [tool-verified: function signature at lineage_router.py:73-76]:

Parameter | Values | Default | Effect
--- | --- | --- | ---
`focus` | A node id | — | Scope the response to the sub-graph around this node
`direction` | `upstream` | `downstream` | `both` | `both` | Which direction to traverse from `focus`
`depth` | integer | unbounded | Maximum hop distance from `focus`

Response is the same shape as the statement graph, with a `cycles` field added [tool-verified: `MergedGraph.to_dict` in merge.py:60-64]:

    {
      "nodes": [...],
      "edges": [...],
      "outputs": [...],
      "cycles": [
        {
          "nodes": ["orders.region", "enriched_orders.region"],
          "has_materialization_boundary": true,
          "classification": "feedback"
        }
      ]
    }

## Using lineage to govern command contracts

Because the taint closure connects every declared input column to every declared output column, the breadth of that closure depends entirely on what you declare.

Consider a command that takes a full orders table (`id`, `region`, `amount`, `customer_id`, `discount`, `notes`, ...) and returns an `embedding`. If the input contract lists all those columns, every downstream column that uses the embedding will show lineage from all of them. That is accurate but not useful — it is hard to tell what actually mattered.

Declare only `id` and `text` (the columns the embedding model actually reads), and the lineage cone tightens to those two source columns. The derivation is both sound and precise.

See Commands for the mechanics of declaring a narrow input contract.