Skip to content

← Training overview

12. Lineage

A registered process is a SQL view that depends on other views and tables. A registered dataset is a view that depends on registered processes, which in turn depend on raw sources. tdfs4ds walks those dependencies for you, returns a graph object, and renders it as an interactive Sankey diagram.

Why look at lineage

  • Audit before refactor. Before you change a source table, see every feature, process, and dataset that depends on it.
  • Impact assessment. A request to deprecate RAW.LEGACY_CUSTOMERS? Walk forward and list everything you'd break.
  • Onboarding. "Where does risk_index come from?" → one diagram.
  • Migration planning. The graph feeds the migration manifest in Module 13.

Build the graph

from tdfs4ds.lineage import build_teradata_dependency_graph

graph = build_teradata_dependency_graph(
    sql_query="SELECT * FROM MY_DB.V_CUSTOMER_360",
    expand_datasets_via_process_catalog=True,
)

graph is a dict with three useful keys:

Key Content
nodes One entry per object: name, object_type ('table', 'view', 'process', 'dataset'), and table_info with PI / partition columns
edges Directed dependencies (source → consumer)
roots Nodes with no upstream dependencies — the raw sources

Two flags worth knowing:

  • stop_at_feature_store_datasets=True — recognise registered datasets as leaves rather than walking through them.
  • expand_datasets_via_process_catalog=True — when a dataset is hit, resolve its FEATURE_VERSION UUIDs against FS_V_PROCESS_CATALOG and link to the process views (not the raw storage tables). Defaults to True.

Visualise it

from tdfs4ds.lineage import plot_lineage_sankey

fig = plot_lineage_sankey(graph, title='Customer 360 lineage')
fig.show()                # interactive in a notebook
fig.write_html('lineage.html')   # share offline

The consumer agent's LINEAGE skill uses the same graph under the hood — chatting "Where does lifetime_amount come from?" returns the depth-ordered dependency list and a plain-language summary.

Diagnose partitioning at the same time

Every node carries primary_index_columns and partition_columns when known. Combine that with Module 8 › Partitioning to audit which source tables exploit partition elimination and which don't:

unpartitioned = [
    n for n, info in graph['table_info'].items()
    if info.get('object_type') == 'table' and not info.get('partition_columns')
]

Persistent lineage (the second approach)

The on-the-fly graph above re-derives lineage every call. For repeated queries and impact analysis ("what breaks if I change this?"), tdfs4ds also maintains a persisted, temporal lineage graph in FS_LINEAGE_GRAPH. It is:

  • Created automatically when missing.
  • Updated every time you call upload_features().
  • Queryable in both directions — ancestors (upstream) and descendants (impact).
  • Reusable for documentation, impact checking, and visualization.
from tdfs4ds.lineage import get_descendants, get_ancestors

# Impact: everything downstream of this view
get_descendants('MY_DB', 'V_CUSTOMER_FEATURES')

# Sources: everything upstream of this dataset
get_ancestors('MY_DB', 'DS_CHURN_RISK')

For the first time, or after adding many processes, rebuild the whole graph:

import tdfs4ds
tdfs4ds.rebuild_lineage_graph()        # re-derive all edges, store once

Then document your entire store from that persisted graph:

tdfs4ds.rebuild_documentation()        # document every process/dataset once, topo-sorted

See Persisted lineage graph for full details, and Rebuild documentation for how to auto-document your store in a single pass.

Learn more