tdfs4ds.lineage¶
module-attribute
¶
__all__ = ['analyze_sql_query', 'analyze_teradata_ddl', 'build_teradata_dependency_graph', 'graph_to_migration_manifest', 'plot_lineage_sankey', 'show_plotly_robust', 'lineage_graph_catalog_creation', 'record_process_lineage', 'rebuild_lineage_graph', 'load_lineage_digraph', 'get_descendants', 'get_ancestors', 'get_persisted_graph', 'resolve_process_id_for_object', 'resolve_objects', 'check_downstream_processes', 'plot_lineage_pyvis']
Extracts source and target tables/views from a SQL query.
Analyse Teradata CREATE TABLE DDL and return
{ 'primary_index_columns': [...], 'partition_columns': [...], 'partitioning_levels': [...], 'partitioning_by_column': {...} }
Critical behavior: - Ignores keyword-like text inside single-quoted literals - Searches PRIMARY INDEX / PARTITION BY only in the table-options tail (after the column-definition block).
build_teradata_dependency_graph(sql_query: str, *, default_database: Optional[str] = None, max_depth: int = 50, store_ddl: bool = False, stop_at_feature_store_datasets: bool = True, expand_datasets_via_process_catalog: bool = True) -> Dict[str, Any]
Builds a dependency graph by
1) analyzing the initial SQL via analyze_sql_query 2) for each discovered source object: - SHOW VIEW and re-run analyze_sql_query on its DDL to expand dependencies - or SHOW TABLE and run analyze_teradata_ddl to attach PI/partition metadata
Feature store dataset views (registered in the FS_V_DATASET_CATALOG) are treated as named "dataset" nodes when stop_at_feature_store_datasets=True (the default). Instead of drilling into their underlying SQL, the graph queries the dataset feature catalog (FS_V_{catalog}_FEATURES) to find which registered process views contribute to the dataset (one process = one view), and adds edges to those views directly. Set expand_datasets_via_process_catalog=False to treat datasets as pure leaves.
Graph edges are parent -> child (parent depends on child).
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
{ "nodes": { node_name: { ...NodeInfo... }, ... }, "edges": [ (parent, child), ... ], "roots": [ ...targets from initial SQL... ], |
Dict[str, Any]
|
} |
graph_to_migration_manifest(graph: Dict[str, Any], *, filter_database: Optional[str] = None) -> Dict[str, Any]
Converts a lineage graph into a migration manifest — a JSON-serialisable dict that lists every view and table discovered, with optional filtering to a single database (e.g. the feature store database).
Useful for planning a feature store migration: objects whose database matches
filter_database are the ones that need to be recreated or moved; edges
between them describe the recreation order.
Parameters¶
graph : dict
Output of build_teradata_dependency_graph.
filter_database : str, optional
When given, only nodes whose database field matches (case-insensitive)
are included. Edges whose source or target falls outside the filter are
also excluded. When None every node and edge is returned.
Returns¶
dict with keys: "views" : list of dicts {database, name, type} type is "view" or "dataset" "tables" : list of dicts {database, name, type, primary_index, partition_columns} type is "table" "edges" : list of dicts {from, to} (parent depends on child)
plot_lineage_sankey(graph: Dict[str, Any], *, title: str = 'Lineage Sankey', max_label_len: int = 45, show_full_name_in_hover: bool = True) -> go.Figure
Plot an interactive Sankey from a lineage graph like
{ "nodes": {name: {name, object_type, database, object, table_info, error, ...}, ...}, "edges": [(parent, child), ...], # parent depends on child "roots": [...] }
Hover displays relevant info (no DDL). Returns a plotly Figure.
Display Plotly figure robustly across environments.
- Tries normal fig.show() (Jupyter mime rendering)
- Falls back to browser renderer if nbformat is missing
- Optionally writes an HTML file
Create the FS_LINEAGE_GRAPH temporal edge table and its public
FS_V_LINEAGE_GRAPH view if they do not already exist.
Idempotent: mirrors documentation_tables_creation — a failed CREATE
whose error mentions "already exists" is swallowed.
Returns¶
str The name of the lineage graph table.
Persist the lineage subgraph rooted at view_name into FS_LINEAGE_GRAPH.
Edges are produced by :func:build_teradata_dependency_graph (the same
function used everywhere else, which skips FS-internal technical tables and
stops at datasets / process views). Idempotent refresh: every parent node
present in the freshly built graph has its existing edge rows deleted
(CURRENT VALIDTIME) before the new ones are inserted, so a re-run / rewrite
replaces exactly that subgraph and leaves no stale edges.
Parameters¶
view_name : str
The (possibly schema-qualified, possibly quoted) process view name,
e.g. my_db."V_FEAT" or a DataFrame _table_name.
art_sql : list of str, optional
UAF ART view mode. The INTO VOLATILE ART materialisation statements of
this process. When provided, lineage is traced through the ephemeral
volatile ART to the real series-input source: any edge in the view graph
that points at a volatile ART named by art_sql is rewritten into a
two-hop path feature_view -> <operator/layer node> -> series_input_source
(the sources parsed from the art_sql SERIES_SPEC/ART_SPEC
TABLE_NAME(...) clauses), so no dangling edge to a non-existent ART
remains. Non-fatal: on any parse problem the classic (view-only) graph is
persisted unchanged.
art_layers : list of str, optional
The declared LAYER name(s), used only to label the synthetic operator node
(e.g. vw_feat::ARTFITMETADATA). Cosmetic.
Returns¶
int Number of edges written (0 when the graph has no edges).
rebuild_lineage_graph(objects=None, all_data_domains: bool = True, check: bool = True, prune: bool = True, extra_views=None, require_exists: bool = True) -> int
Rebuild the persisted lineage graph — fully, or scoped to a list of objects.
Use this to initialise FS_LINEAGE_GRAPH on a store that already has
registered processes/datasets, to refresh a graph whose edges have drifted,
or to scoped-refresh just the objects you edited. The incremental write hook
(:func:record_process_lineage, called from upload_features) keeps lineage
current going forward; this function recomputes on demand.
The table is valid-time temporal: every write is diff-based, so re-running
on an unchanged store touches zero rows (no history churn). There is no blind
delete+insert — :func:record_process_lineage leaves unchanged edges alone,
inserts new ones, and closes (CURRENT VALIDTIME DELETE) ones that disappeared.
Full refresh (objects=None):
- Ensure the table/view exist (idempotent).
- Seed from every registered process view and every dataset (all data
domains by default), re-deriving each subgraph via
:func:
record_process_lineage(diff-based — unchanged edges are untouched). - When
prune=True(default), edges whose parent object is no longer a seed (e.g. a since-dropped process) are closed withCURRENT VALIDTIME DELETE. This is a targeted prune of stale parents, not a wipe — it does not re-open the edges that are still valid.
Scoped refresh (objects=[...]): no prune, no catalog-wide seed. The names
are validated by :func:resolve_objects (raises on any unmatched name — case /
quoting / db.name safe), then record_process_lineage refreshes each object's own
outgoing edges (diff-based). Use this when a (possibly non-process) view was edited.
When check=True, :func:check_downstream_processes runs afterwards.
Parameters¶
objects : str | Iterable[str] | None
When None (default), full refresh. Otherwise refresh only these objects.
all_data_domains : bool
Full-refresh only: all DATA_DOMAINs (default) or the active one.
check : bool
Scoped refresh only: run the downstream EXPLAIN compile check afterwards.
prune : bool
Full-refresh only: close edges of parents that are no longer registered
(default True). Set False to never delete (purely additive refresh).
extra_views : str | Iterable[str] | None
Full-refresh only: additional standalone view names to seed into the
graph alongside every registered process view + dataset. Use this to
capture a non-process / non-dataset view (e.g. a hand-built or
crystallized reporting view) that nothing registered references — it
would otherwise never enter FS_LINEAGE_GRAPH. Each name is validated
via :func:resolve_objects (case / quoting / db.name safe; raises on
any unmatched name). There is no automatic schema scan — only the
names you pass.
require_exists : bool
Scoped refresh only. True (default): an unknown name raises before
anything is written. False: unknown names are logged and skipped, and
the others are refreshed (what a migration wants).
Returns¶
int Number of edge changes written (0 when nothing changed).
Load the persisted lineage graph into a networkx.DiGraph with a single
read of FS_V_LINEAGE_GRAPH (CURRENT VALIDTIME).
Edge direction is parent -> child (parent depends on child). Each node carries
an object_type attribute ('T'/'V'/'D'). All ancestor / descendant traversal
is then done in-memory — no recursive SQL.
as_of (a date or timestamp) reads the graph as it was at that valid
time: the primary store's base table, VALIDTIME AS OF. Every lineage write
is diff-based, so its history is real. At a past date there is no on-the-fly
fallback (it would describe today's DDL) and no read federation.
tag_managers (default True) marks the TimeManager / FilterManager objects
with a node attribute role ('time manager', 'filter manager',
'manager table'), from the process catalog and the manager registry. The
stored OBJECT_TYPE is unchanged ('V'/'T'). Fail-open.
Fallback: when the persisted table is missing/empty or unreadable and
auto_build=True (default), the graph is re-derived on the fly via
:func:_build_digraph_on_the_fly so callers still work on a store whose graph
was never persisted (run :func:rebuild_lineage_graph to persist it and make
subsequent loads fast).
Returns¶
networkx.DiGraph
Objects that depend (directly or transitively) on database.object_name
— the impact / reverse direction (nx.ancestors over a depends-on
DiGraph, since edges point parent -> child = downstream -> upstream).
as_of: read the graph at that valid time (ignored when graph is given).
Returns canonical node ids ('DB.OBJECT').
Upstream sources that database.object_name depends on (directly or
transitively) — nx.descendants over the depends-on DiGraph.
as_of: read the graph at that valid time (ignored when graph is given).
Returns canonical node ids ('DB.OBJECT').
Return the persisted lineage in the same {'nodes', 'edges', 'roots'}
dict shape produced by :func:build_teradata_dependency_graph, so existing
consumers (plot_lineage_sankey, graph_to_migration_manifest,
skill_lineage) work unchanged on persisted data.
Each node also carries role ('time manager' / 'filter manager' /
'manager table' / None), set once at load.
Parameters¶
seed : str, optional
If given ('DB.OBJECT' or 'OBJECT'), restrict to the subgraph of
the seed plus all its ancestors and descendants.
as_of : str, optional
Read the graph as it was at that valid time.
Resolve an object node to its registered process PROCESS_ID, or None
when the object is not a registered process view (a plain view / table /
dataset).
Some views are processes, some are not — the lineage table stores objects
only, so the link is resolved here through the authoritative process catalog
(FS_V_PROCESS_CATALOG: VIEW_NAME <-> PROCESS_ID). Matching is done
on the de-quoted, uppercased (database, object) to be robust to the
quoting variance of stored VIEW_NAME values.
Normalise and verify a user-supplied list of object names so case / "
quoting / db.name variance never causes a silent mismatch.
Each input ("db"."NAME", db.name, NAME, any case) is parsed with
the same _parse_node_name + _norm helpers record_process_lineage uses to
write the table, so a resolved name compares equal to stored rows. An
object is considered to exist if it is a node in the persisted lineage graph,
or (fallback, for a freshly-created view not yet in the graph) present in
DBC.TablesV.
Parameters¶
objects : str | Iterable[str]
One name or a list of names.
require_exists : bool
When True (default), raise ValueError naming every unmatched input.
When False, unmatched inputs are logged and dropped.
graph : networkx.DiGraph, optional
Pre-loaded lineage graph (avoids a reload when the caller already has one).
Returns¶
(resolved, unmatched)
resolved is a list of canonical (DB_UPPER, OBJECT_UPPER) tuples
(default DB filled from tdfs4ds.SCHEMA when the input was unqualified);
unmatched is the list of original input strings that resolved to nothing.
Deterministic compile check for the process views downstream of changed objects.
When a non-process view changes its result columns, the process views that
read it can silently break. This runs EXPLAIN SELECT * FROM <process_view>
on every registered process downstream of the given object(s) — EXPLAIN only
verifies the query compiles (all columns resolve), it does not execute it.
Same compile-validity idiom as the pre-check in upload_features.
Parameters¶
objects : str | Iterable[str]
Changed object name(s); resolved via :func:resolve_objects.
Returns¶
dict
{"checked": int, "ok": [view, ...],
"broken": [{"view", "process_id", "error"}, ...]}.
Broken processes are also logged as a warning.
plot_lineage_pyvis(graph: Dict[str, Any], *, output_path: str = 'tdfs4ds_lineage.html', height: str = '750px', width: str = '100%', notebook: bool = False)
Render a lineage graph as an interactive pyvis network and write it to a self-contained HTML file.
Accepts the same {'nodes', 'edges', 'roots'} dict shape produced by
:func:get_persisted_graph / :func:build_teradata_dependency_graph.
Node colour encodes object type (view/table/dataset), matching
plot_lineage_sankey. Edge direction is parent -> child (depends-on).
pyvis is a tdfs4ds dependency; it is imported lazily so building/persisting lineage never pays the import cost unless a visualisation is requested.
Returns¶
pyvis.network.Network
The configured network (already written to output_path).