`fs-lineage`¶

Build a dependency graph and render an interactive Sankey diagram.

name: "Feature Store: Lineage" description: "For a developer who needs generated Python code and a Sankey diagram. Build the dependency graph for a registered process or dataset view, display primary index and partition columns for every source table, and render a Sankey diagram. Invoke when the user wants to understand data lineage or which tables a process depends on. SKIP for consumer-agent Q&A in natural language — use ca-lineage instead." argument-hint: "[process-id or dataset-view-name]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

Generate Python code to build and visualise the dependency graph for a registered tdfs4ds process or dataset view. The graph reveals source tables, their primary index columns, and their partition columns.

User context: $ARGUMENTS

Step	Skill	Why
Bigger picture	`fs-analyze`	Same dependency graph plus EXPLAIN score and FilterManager rewrite proposal.
LLM docs	`fs-document`	Document the process this graph belongs to.
Find process	`fs-inspect`	List the process catalog to pick a process ID.
Consumer	`fs-agent`	Lineage is also reachable through the conversational agent (`LINEAGE` intent).

Demos: demos/notebooks/A1 - Lineage and impact.ipynb.

Ask for any missing values before generating code: - Target — a process_id (UUID) or a dataset view name to analyse. - LLM config — only needed if document_process will also be called; skip if just doing lineage.

Emit the following pattern, substituting real values:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # teradataml context from TDFS4DS_TD_HOST / _USERNAME / _PASSWORD
tdfs4ds.connect(database='<DATABASE>', create_if_missing=True)
import teradataml as tdml
tdfs4ds.select_data_domain('<DATA_DOMAIN>')   # or set TDFS4DS_DATA_DOMAIN in .env

# ---------------------------------------------------------------------------
# 1. Retrieve the process SQL
# ---------------------------------------------------------------------------
from tdfs4ds.process_store.process_store_catalog_management import get_process_info

process_info = get_process_info('<PROCESS_ID>')
sql = process_info['PROCESS_SQL']

# Alternatively, retrieve the SQL for a dataset view directly:
# sql = tdml.execute_sql("SHOW VIEW <DATASET_VIEW_NAME>").fetchall()[0][0].replace('\r', '\n')

# ---------------------------------------------------------------------------
# 2. Build the dependency graph
#    Each node in graph['nodes'] contains:
#      - object_type: 'table' or 'view'
#      - table_info:
#          primary_index_columns: list of PI column names
#          partition_columns:     list of partition column names
#          partitioning_levels:   detailed per-level partition info
# ---------------------------------------------------------------------------
from tdfs4ds.lineage import build_teradata_dependency_graph

graph = build_teradata_dependency_graph(sql_query=sql)

# Inspect nodes and their table info
for name, node in graph['nodes'].items():
    print(f"\nTable/View: {name}")
    print(f"  Type            : {node['object_type']}")
    if node.get('table_info'):
        ti = node['table_info']
        print(f"  Primary Index   : {ti['primary_index_columns']}")
        print(f"  Partition cols  : {ti['partition_columns']}")
        if ti['partitioning_levels']:
            for lvl in ti['partitioning_levels']:
                print(f"    Level {lvl['level']}: {lvl['kind']} on {lvl['columns']}")

print("\nEdges:", graph['edges'])

# ---------------------------------------------------------------------------
# 3. Sankey visualisation
# ---------------------------------------------------------------------------
from tdfs4ds.lineage import plot_lineage_sankey, show_plotly_robust

fig = plot_lineage_sankey(graph, title='<TITLE>')

# Display inline (Jupyter):
fig.show()

# Or save to HTML:
# show_plotly_robust(fig, html_path='lineage.html')

What the graph output tells you¶

Field	Meaning
`primary_index_columns`	Columns Teradata uses to distribute rows across AMPs. Filtering on these avoids full-table scans.
`partition_columns`	Columns used for partition elimination. Filtering on these skips irrelevant partitions.
`partitioning_levels`	Multi-level partition details — `RANGE_N`, `CASE_N`, or `COLUMNAR` per level.
`edges`	Directed dependency edges `(target, source)`.
`roots`	Top-level targets with no dependents.

Rules¶

Use tdfs4ds.select_data_domain(...) as an alternative to tdfs4ds.DATA_DOMAIN = ...; both are equivalent.
get_process_info requires create_if_missing=True on connect() or the documentation tables must already exist.
If table_info is None for a node, the table DDL could not be retrieved (view, temp table, or permission issue).
For dataset lineage, use SHOW VIEW <view_name> to get the SQL, then pass it to build_teradata_dependency_graph.

fs-lineage¶

Related skills¶

What the graph output tells you¶

Rules¶

`fs-lineage`¶