Persisted lineage graph

The persisted lineage graph (FS_LINEAGE_GRAPH) is a temporal table that records every dependency edge in your feature store — which views and tables feed into which processes and datasets. Unlike the on-the-fly dependency graph, which re-derives lineage on demand, the persisted graph is:

• Queryable in both directions: upstream sources (ancestors) and downstream consumers (descendants / impact).
• Fast: one table read, then in-memory traversal with networkx.
• Reusable: documentation, impact analysis, and visualization all share one persisted copy.
• Temporal: full VALIDTIME history; you can audit what the graph was at any point in the past.

---

Build the graph

Full rebuild (first time or recovery)

The first time you set up a feature store, or when you have a pre-existing store with registered processes and datasets, build the full graph in one call:

import tdfs4ds

n_edges = tdfs4ds.rebuild_lineage_graph()
print(f"Persisted {n_edges} edges")

This wipes the FS_LINEAGE_GRAPH table and re-derives every edge by walking the SHOW VIEW DDL of every registered process view and dataset (across all DATA_DOMAINs). Per-view failures are non-fatal — one unreadable DDL does not abort the rebuild.

To rebuild only the current data domain:

tdfs4ds.rebuild_lineage_graph(all_data_domains=False)

Automatic updates

Every time you call upload_features() with a new process, the write hook record_process_lineage() adds that process's edges to the graph. Since 0.3.1.220 the same happens for datasets: build_dataset(), rebuild_dataset() and build_grouped_dataset() record the dataset → process views → sources edges. So most feature stores keep their graph current automatically.

A dataset built by an earlier version is not in the graph until you run rebuild_lineage_graph() once (or, in the admin app, Lineage → Add them to the lineage, which records only the missing datasets).

To skip it for a single dataset (edge case):

tdfs4ds.build_dataset(..., record_lineage=False)

---

Query the graph

Once persisted, traverse the graph in-memory with deterministic functions:

Impact direction (what breaks if I change this?)

from tdfs4ds.lineage import get_descendants

# Every process/dataset that depends on this view
downstream = get_descendants(database='MY_DB', object_name='V_CUSTOMER_FEATURES')
for obj in downstream:
    print(obj)  # e.g. ("MY_DB", "DS_CHURN_RISK", "D")

The result is a list of (DATABASE, OBJECT_NAME, OBJECT_TYPE) tuples, ordered so a consumer always appears after its sources (safe for serial processing).

Upstream sources (where does this come from?)

from tdfs4ds.lineage import get_ancestors

# Every source table/view feeding into this dataset
upstream = get_ancestors(database='MY_DB', object_name='DS_CHURN_RISK')
for obj in upstream:
    print(obj)

Full graph reconstruction

To get the same {'nodes', 'edges', 'roots'} shape as build_teradata_dependency_graph() but from persisted data (no SHOW VIEW re-scan):

from tdfs4ds.lineage import get_persisted_graph

# Whole graph
graph = get_persisted_graph()

# Subgraph: this object + all ancestors + all descendants
graph = get_persisted_graph(seed='MY_DB.DS_CHURN_RISK')

---

Rebuild documentation from the graph

Once the lineage graph is persisted, you can document your entire feature store in a single pass without re-deriving lineage:

import tdfs4ds

# Document every process and dataset exactly once, topo-sorted (leaves first)
# so shared upstream views are used as context, not re-documented.
tdfs4ds.rebuild_documentation()

# Force regenerate everything (overwrites existing descriptions).
tdfs4ds.rebuild_documentation(force_update=True)

The rebuild uses the persisted graph as the source of truth — it topologically sorts the edges, then walks them with a shared visited-set so:

• Each process and dataset is documented exactly once.
• Intermediate views (non-process, non-dataset) are documented as side effects of the processes they feed, not as top-level targets.
• The LLM reuses earlier descriptions as context, building a coherent narrative.

This is orders of magnitude faster than re-deriving the graph with SHOW VIEW for every node. See Process documentation and Dataset documentation for what gets documented.

---

Scoped refresh (after editing a view)

When you edit a non-process view (an intermediate view that feeds processes but isn't itself registered), you don't need a full rebuild. Refresh just that view's edges and check the impact:

import tdfs4ds

# 1. Refresh just this view's edges (no global wipe).
# Also runs EXPLAIN on downstream processes to verify they still compile.
tdfs4ds.rebuild_lineage_graph(objects=['vw_aggregate_features'], check=True)

# 2. Re-document the impact set (the view + everything downstream).
tdfs4ds.rebuild_documentation(objects=['vw_aggregate_features'])

Name matching is safe

Object names are normalized and verified:

# These all refer to the same view — quotes, case, and schema are normalized.
tdfs4ds.rebuild_lineage_graph(objects=['vw_agg'])
tdfs4ds.rebuild_lineage_graph(objects=['VW_AGG'])
tdfs4ds.rebuild_lineage_graph(objects=['my_db.vw_agg'])
tdfs4ds.rebuild_lineage_graph(objects=['"my_db"."VW_AGG"'])

A typo raises ValueError — never a silent no-op.

Standalone impact check

To check if downstream processes still compile without updating the graph:

result = tdfs4ds.check_downstream_processes(['vw_aggregate_features'])
print(f"checked: {result['checked']} | ok: {len(result['ok'])} | broken: {len(result['broken'])}")

for broken in result['broken']:
    print(f"  BROKEN: {broken['view']} -> {broken['error']}")

This runs a cheap EXPLAIN SELECT * on every registered process downstream — enough to verify the SQL compiles (a changed view's dropped columns can silently break consumers).

---

Views that must stay in the graph (attached views)

rebuild_lineage_graph(extra_views=[...]) records a standalone view once; a later full rebuild without that argument closes its edges. For a view that must stay (for example a user view joining a dataset with a source table, read by no process), use tdfs4ds.attach_objects(['DB.VIEW']) (0.3.5.2): its edges are recorded at once and every full rebuild seeds attached views. An attached table that no view reads has no edge. See Attached views and tables in Manual documentation.

Visualize the graph

Interactive HTML (pyvis network diagram)

from tdfs4ds.lineage import plot_lineage_pyvis, get_persisted_graph

# Whole graph
plot_lineage_pyvis(get_persisted_graph(), output_path='lineage.html')

# Subgraph: one object + impact + upstream
plot_lineage_pyvis(
    get_persisted_graph(seed='MY_DB.DS_CHURN_RISK'),
    output_path='impact.html'
)

The HTML is self-contained (vis.js bundled) — open in a browser, pan/zoom, and click nodes to inspect them. A color legend is shown in the bottom-left corner:

Color	Type
Orange	Registered process view (feature-engineering view in FS_V_PROCESS_CATALOG)
Blue	Plain / intermediate view (not a registered process)
Green	Permanent source table
Purple	Dataset node

Hover over any node to see its full qualified name and type label.

Web UI

When running the Docker container or the python -m tdfs4ds.serve server, the lineage graph is also served as an interactive web panel:

http://localhost:8000/lineage

Optionally scope it to a single object with the seed query parameter:

http://localhost:8000/lineage?seed=MY_DB.DS_CHURN_RISK

See the Deployment guide (section: Dependency graph / lineage) for details.

Sankey diagram

For a flow-oriented view (from sources to consumers), use the Sankey:

from tdfs4ds.lineage import plot_lineage_sankey

plot_lineage_sankey(get_persisted_graph(), output_path='lineage.html')

See Sankey visualisation for more.

---

Rebuild documentation with cascade

When a process's documentation changes, you can automatically propagate the new meaning downstream to all dependent processes and datasets:

tdfs4ds.upload_features(
    ...,
    document=True,
    cascade_redocumentation=True,  # re-document all downstream
)

This is opt-in because it costs one LLM pass per affected descendant. It only cascades when the process's documentation actually changed; unchanged processes are skipped. The impact set is determined from the persisted lineage graph, so the graph must be current.

See Process documentation for details.

---

Understanding the edge structure

Each row in FS_LINEAGE_GRAPH represents one directed edge:

Column	Meaning
PARENT_DATABASE_NAME	Source object's database
PARENT_OBJECT_NAME	Source object's name
PARENT_OBJECT_TYPE	'T' (table), 'V' (view), or 'D' (dataset)
CHILD_DATABASE_NAME	Target object's database
CHILD_OBJECT_NAME	Target object's name
CHILD_OBJECT_TYPE	'T', 'V', or 'D'

Edge direction: parent → child means parent depends on child. So if you have an edge (MY_DB, V_AGG, V) → (MY_DB, T_RAW, T), it means the view V_AGG depends on the table T_RAW.

No PROCESS_ID on edges: an edge connects two objects, and most objects are not processes. To find the process that computes an object, use:

from tdfs4ds.lineage import resolve_process_id_for_object

process_id = resolve_process_id_for_object(db='MY_DB', obj='V_AGG')
# process_id is None if the view is not a registered process

---

Related topics

Topic	Purpose
Dependency graph	On-the-fly graph builder (no persistence); useful for one-off analysis.
Sankey visualisation	Flow diagram showing sources → consumers.
Migration manifest	Export the graph as JSON for migration planning.
Process documentation	Auto-document the processes that compute features.
Dataset documentation	Auto-document the datasets that combine features.
fs-lineage-graph skill	For Claude Code: rebuild lineage, document the store, check impact.
fs-lineage skill	For Claude Code: one-off dependency graph + Sankey (no persistence).