Dependency graph¶

build_teradata_dependency_graph walks a SQL query (or a dataset DDL) and returns a structured graph of every table and view involved. For each node it captures the primary index and the partition columns — exactly what you need to decide whether a FilterManager rewrite is worthwhile.

Build the graph¶

from tdfs4ds.lineage import build_teradata_dependency_graph

graph = build_teradata_dependency_graph(sql_query=sql)

# Returns:
# {
#   'nodes': {
#       '<db>.<name>': {
#           'object_type': 'table' | 'view' | 'dataset',
#           'table_info': {
#               'primary_index_columns':  ['CUSTOMER_ID', ...],
#               'partition_columns':      ['TX_DATE'],
#               'partitioning_levels':    [{'level': 1, 'kind': 'RANGE_N', 'columns': ['TX_DATE']}],
#           },
#       },
#       ...
#   },
#   'edges': [('<dst>', '<src>'), ...],
#   'roots': ['<top-level>', ...],
# }

The walk uses SHOW VIEW / SHOW TABLE DDL — every table reachable through views is resolved transitively. sqlparse is used at parse time; no EXPLAIN is run.

Two source paths¶

# 1. From a registered process
from tdfs4ds.process_store.process_store_catalog_management import get_process_info
sql = get_process_info(process_id)['PROCESS_SQL']

# 2. From a dataset view DDL
sql = tdml.execute_sql("SHOW VIEW DS_CUSTOMER_SPEND").fetchall()[0][0].replace('\r', '\n')

Both are valid inputs to build_teradata_dependency_graph(sql_query=...).

Dataset expansion¶

When the graph crosses a dataset view, two behaviours are available:

`expand_datasets_via_process_catalog`	Behaviour
`True` (default)	Edges drawn to the process views that produce the dataset's columns. `FEATURE_VERSION` UUIDs in the dataset DDL are matched to `PROCESS_ID` entries in `FS_V_PROCESS_CATALOG`.
`False`	Edges drawn directly to the raw storage tables (`FS_T_…`).

The default surfaces the logical lineage — what computes the dataset — which is usually what users want. Set it to False for migration audits where physical dependencies matter.

Inspect node by node¶

for name, node in graph['nodes'].items():
    ti = node.get('table_info')
    if ti is None:
        continue
    print(f"{name}")
    print(f"  Primary Index : {ti['primary_index_columns']}")
    print(f"  Partition cols: {ti['partition_columns']}")
    for lvl in ti.get('partitioning_levels', []):
        print(f"    Level {lvl['level']}: {lvl['kind']} on {lvl['columns']}")

Decision aid for FilterManager¶

Field	Why it matters
`primary_index_columns`	Used for AMP routing. Filtering on the PI in WHERE does not help — it only helps in joins.
`partition_columns`	Used for partition elimination. Filtering on these in WHERE skips irrelevant partitions.

When a source table has partition_columns that the current process SQL does not reference, a FilterManager rewrite usually pays off. See fs-analyze for an end-to-end recommendation workflow.

When a node has no `table_info`¶

table_info is None when the DDL could not be fetched — typically:

The object is a temporary or volatile table.
The object is in a database the current user cannot read DDL from.
The object name in the SQL is mistyped (treat this as a warning).

The graph still records the node — the missing info is informative, not a failure.

Persisted lineage graph¶

build_teradata_dependency_graph re-derives lineage on the fly every call. For repeated queries — and for the reverse / impact direction ("what depends on this?") — tdfs4ds also maintains a persisted, temporal lineage graph in the FS_LINEAGE_GRAPH table. It is created when missing at setup() and connect(create_if_missing=True), and refreshed at every upload_features call by record_process_lineage.

One row per directed edge parent → child (parent depends on child).
Nodes reuse the business-dictionary object identity (DATABASE_NAME, OBJECT_NAME) plus OBJECT_TYPE ('T'/'V'/'D'). No PROCESS_ID is stored on an edge — an edge connects two objects, and most objects are not processes. Use resolve_process_id_for_object(db, obj) to map a node to its process (returns None for a plain view).
Edges are the output of build_teradata_dependency_graph, so the feature store's internal technical tables are skipped and datasets appear as nodes.

from tdfs4ds.lineage import (
    get_descendants, get_ancestors, get_persisted_graph, plot_lineage_pyvis,
)

# Impact direction: every object that depends on a source table/view/process.
get_descendants("MY_DB", "V_FEAT_ENG")

# Upstream sources a view depends on.
get_ancestors("MY_DB", "DS_CUSTOMER_SPEND")

# Reconstruct the {'nodes','edges','roots'} shape from persisted data
# (drop-in for plot_lineage_sankey / graph_to_migration_manifest).
graph = get_persisted_graph(seed="MY_DB.DS_CUSTOMER_SPEND")

# Interactive HTML network.
plot_lineage_pyvis(graph, output_path="lineage.html")

All traversal loads the table once into a networkx.DiGraph (load_lineage_digraph) and walks it in memory — no recursive SQL.

Full refresh / init¶

The write hook keeps lineage current as you upload features, but a store that already has registered processes/datasets — or one whose edges have drifted — can rebuild the whole graph in one pass:

import tdfs4ds

# Wipe FS_LINEAGE_GRAPH and re-derive every edge from every registered
# process view + dataset, across all data domains.
tdfs4ds.rebuild_lineage_graph()              # all_data_domains=True by default
tdfs4ds.rebuild_lineage_graph(all_data_domains=False)   # current DATA_DOMAIN only

It clears the current edges (CURRENT VALIDTIME DELETE, so VALIDTIME history is retained), then seeds record_process_lineage from each process VIEW_NAME and each dataset view. Per-seed failures are non-fatal — one unreadable view does not abort the rebuild. Returns the total edge count written.

Rebuild documentation from the persisted graph¶

Once the lineage graph is persisted, the whole feature store can be documented without re-deriving lineage — the graph is read once from the table instead of walking SHOW VIEW again:

import tdfs4ds

tdfs4ds.rebuild_lineage_graph()       # 1. persist the graph
tdfs4ds.rebuild_documentation()       # 2. document every process/dataset once
tdfs4ds.rebuild_documentation(force_update=True)   # regenerate everything

rebuild_documentation topologically sorts the persisted graph (leaves first), then walks it with shared visited-sets so each process and dataset is documented exactly once and shared upstream views are never re-documented. Seeds are the graph's top consumers (datasets + process views); base tables and intermediate views are documented only as a side effect of their owning process's walk. With the default force_update=False, already-documented objects are reused as context and only the gaps get an LLM call.

Refreshing just the objects you changed¶

When you edit a non-process view (an intermediate view that feeds processes but isn't itself registered), refresh only the affected slice instead of a full rebuild:

import tdfs4ds

# 1. Refresh just this view's lineage edges (no global wipe), and check impact.
tdfs4ds.rebuild_lineage_graph(objects=['vw_aggregate_features'], check=True)

# 2. Re-document the impact set: the view + everything downstream of it.
tdfs4ds.genai.rebuild_documentation(objects=['vw_aggregate_features'])

# Standalone: which dependent processes still compile after the change?
res = tdfs4ds.check_downstream_processes(['vw_aggregate_features'])
print(res['broken'])   # [{view, process_id, error}, ...] — empty if all OK

Name matching is safe. 'vw_aggregate_features', 'DEMO_USER.VW_AGGREGATE_FEATURES' and '"demo_user"."vw_aggregate_features"' all resolve to the same object — names are normalised (quotes stripped, upper-cased, db.name split) and verified against the persisted graph (and DBC.TablesV for a brand-new view). An unrecognised name raises a ValueError listing it, so a typo never silently does nothing.

Result-column impact. A changed view may rename or drop output columns, which can silently break the process views that read it. check_downstream_processes runs a cheap EXPLAIN SELECT * FROM <view> on every registered process downstream of the change — EXPLAIN only verifies the query still compiles, it does not run it — and reports any that fail so you know exactly which processes to fix. It runs automatically inside rebuild_lineage_graph(objects=..., check=True).

Cascade re-documentation¶

Because the graph knows what depends on a process, a changed process description can be propagated downstream. upload_features(..., cascade_redocumentation=True) re-documents every dependent process and dataset — but only when the process's own documentation actually changed — so a corrected parent meaning flows into the descendants' business descriptions and per-feature column docs. It is opt-in because it costs one LLM pass per affected descendant.