tdfs4ds.lineage.lineage_store¶
Persisted, temporal lineage graph for tdfs4ds.
Where network.py derives lineage on the fly (SHOW VIEW / SHOW TABLE +
regex), this module persists it. At each upload_features call the
process view's lineage subgraph is recorded into the FS_LINEAGE_GRAPH
temporal table (one row per directed parent -> child object edge). The
table is created when missing at setup() / connect(create_if_missing=True).
Design notes¶
- Nodes reuse the business-dictionary object identity
(DATABASE_NAME, OBJECT_NAME)+OBJECT_TYPE CHAR(1)('T'/'V'/'D'), so the lineage graph, the business dictionary and the documentation all speak the same vocabulary. - Edges carry no PROCESS_ID — an edge connects two objects, and most
objects (source tables, intermediate views, datasets) are not processes.
Node -> process resolution is done on demand through the process catalog
(
resolve_process_id_for_object). - Edges are produced by the existing authoritative builder
:func:
build_teradata_dependency_graph, which intentionally skips the feature store's internal technical tables and stops at the dataset / process-view level. We do not re-parse DDL here. - All graph traversal (ancestors / descendants) is done in-memory with networkx after a single full-table load — no recursive SQL, no per-hop round-trips.
module-attribute
¶
module-attribute
¶
_LIN_COLS = 'PARENT_DATABASE_NAME, PARENT_OBJECT_NAME, PARENT_OBJECT_TYPE, CHILD_DATABASE_NAME, CHILD_OBJECT_NAME, CHILD_OBJECT_TYPE'
module-attribute
¶
_PYVIS_COLOR = {'view': '#6baed6', 'process': '#e6550d', 'table': '#31a354', 'dataset': '#756bb1', 'unknown': '#bdbdbd'}
Return name as a Teradata identifier, quoting it only if required.
Idempotent: an already-quoted name is returned unchanged rather than being
wrapped a second time. Embedded quotes are escaped as "".
Parameters¶
name : str
The raw object/database name.
force : bool, default False
Quote even when the name does not need it. Use only at the few sites
that have always emitted unconditional quotes and whose exact output is
already baked into persisted DDL (see
feature_store/feature_query_retrieval.py) — elsewhere forcing would
change the SQL of existing stores for no reason.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
True when a failed CREATE means the object was already there.
The single authority for that question. It is deliberately NOT "this is an error we tolerate" — for an idempotent CREATE, an object that already exists is the nominal outcome, and reporting it as an error is what makes logs unusable for supervision.
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
build_teradata_dependency_graph(sql_query: str, *, default_database: Optional[str] = None, max_depth: int = 50, store_ddl: bool = False, stop_at_feature_store_datasets: bool = True, expand_datasets_via_process_catalog: bool = True) -> Dict[str, Any]
Builds a dependency graph by
1) analyzing the initial SQL via analyze_sql_query 2) for each discovered source object: - SHOW VIEW and re-run analyze_sql_query on its DDL to expand dependencies - or SHOW TABLE and run analyze_teradata_ddl to attach PI/partition metadata
Feature store dataset views (registered in the FS_V_DATASET_CATALOG) are treated as named "dataset" nodes when stop_at_feature_store_datasets=True (the default). Instead of drilling into their underlying SQL, the graph queries the dataset feature catalog (FS_V_{catalog}_FEATURES) to find which registered process views contribute to the dataset (one process = one view), and adds edges to those views directly. Set expand_datasets_via_process_catalog=False to treat datasets as pure leaves.
Graph edges are parent -> child (parent depends on child).
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
{ "nodes": { node_name: { ...NodeInfo... }, ... }, "edges": [ (parent, child), ... ], "roots": [ ...targets from initial SQL... ], |
Dict[str, Any]
|
} |
Accepts names like
- '"sales"."orders"'
- '
."orders"' - 'sales.orders'
- 'orders' (rare; treat as object only)
Returns (database, object).
Uppercase, de-quoted (database, object) for stable comparison/storage.
Rewrite the view-graph edges of a UAF ART view so lineage traces through the ephemeral volatile ART to the real series-input source.
The feature view's ART_SPEC(TABLE_NAME(<art>)) yields a dangling edge
feature_view -> <art> (the volatile ART is not in any catalog). Using
art_sql we know (a) the ART name(s) it declares and (b) the real series
source(s) it reads (SERIES_SPEC(TABLE_NAME(<src>))). We drop every edge whose
child is such an ART, insert a synthetic operator/layer node
<feature_obj>::<layers> (type 'V'), and link
feature_view -> op_node -> each series source.
Returns (new_rows, new_parents). Fail-open: on any problem returns the inputs
unchanged so the classic (view-only) graph is persisted.
Create the FS_LINEAGE_GRAPH temporal edge table and its public
FS_V_LINEAGE_GRAPH view if they do not already exist.
Idempotent: mirrors documentation_tables_creation — a failed CREATE
whose error mentions "already exists" is swallowed.
Returns¶
str The name of the lineage graph table.
Persist the lineage subgraph rooted at view_name into FS_LINEAGE_GRAPH.
Edges are produced by :func:build_teradata_dependency_graph (the same
function used everywhere else, which skips FS-internal technical tables and
stops at datasets / process views). Idempotent refresh: every parent node
present in the freshly built graph has its existing edge rows deleted
(CURRENT VALIDTIME) before the new ones are inserted, so a re-run / rewrite
replaces exactly that subgraph and leaves no stale edges.
Parameters¶
view_name : str
The (possibly schema-qualified, possibly quoted) process view name,
e.g. my_db."V_FEAT" or a DataFrame _table_name.
art_sql : list of str, optional
UAF ART view mode. The INTO VOLATILE ART materialisation statements of
this process. When provided, lineage is traced through the ephemeral
volatile ART to the real series-input source: any edge in the view graph
that points at a volatile ART named by art_sql is rewritten into a
two-hop path feature_view -> <operator/layer node> -> series_input_source
(the sources parsed from the art_sql SERIES_SPEC/ART_SPEC
TABLE_NAME(...) clauses), so no dangling edge to a non-existent ART
remains. Non-fatal: on any parse problem the classic (view-only) graph is
persisted unchanged.
art_layers : list of str, optional
The declared LAYER name(s), used only to label the synthetic operator node
(e.g. vw_feat::ARTFITMETADATA). Cosmetic.
Returns¶
int Number of edges written (0 when the graph has no edges).
rebuild_lineage_graph(objects=None, all_data_domains: bool = True, check: bool = True, prune: bool = True, extra_views=None, require_exists: bool = True) -> int
Rebuild the persisted lineage graph — fully, or scoped to a list of objects.
Use this to initialise FS_LINEAGE_GRAPH on a store that already has
registered processes/datasets, to refresh a graph whose edges have drifted,
or to scoped-refresh just the objects you edited. The incremental write hook
(:func:record_process_lineage, called from upload_features) keeps lineage
current going forward; this function recomputes on demand.
The table is valid-time temporal: every write is diff-based, so re-running
on an unchanged store touches zero rows (no history churn). There is no blind
delete+insert — :func:record_process_lineage leaves unchanged edges alone,
inserts new ones, and closes (CURRENT VALIDTIME DELETE) ones that disappeared.
Full refresh (objects=None):
- Ensure the table/view exist (idempotent).
- Seed from every registered process view and every dataset (all data
domains by default), re-deriving each subgraph via
:func:
record_process_lineage(diff-based — unchanged edges are untouched). - When
prune=True(default), edges whose parent object is no longer a seed (e.g. a since-dropped process) are closed withCURRENT VALIDTIME DELETE. This is a targeted prune of stale parents, not a wipe — it does not re-open the edges that are still valid.
Scoped refresh (objects=[...]): no prune, no catalog-wide seed. The names
are validated by :func:resolve_objects (raises on any unmatched name — case /
quoting / db.name safe), then record_process_lineage refreshes each object's own
outgoing edges (diff-based). Use this when a (possibly non-process) view was edited.
When check=True, :func:check_downstream_processes runs afterwards.
Parameters¶
objects : str | Iterable[str] | None
When None (default), full refresh. Otherwise refresh only these objects.
all_data_domains : bool
Full-refresh only: all DATA_DOMAINs (default) or the active one.
check : bool
Scoped refresh only: run the downstream EXPLAIN compile check afterwards.
prune : bool
Full-refresh only: close edges of parents that are no longer registered
(default True). Set False to never delete (purely additive refresh).
extra_views : str | Iterable[str] | None
Full-refresh only: additional standalone view names to seed into the
graph alongside every registered process view + dataset. Use this to
capture a non-process / non-dataset view (e.g. a hand-built or
crystallized reporting view) that nothing registered references — it
would otherwise never enter FS_LINEAGE_GRAPH. Each name is validated
via :func:resolve_objects (case / quoting / db.name safe; raises on
any unmatched name). There is no automatic schema scan — only the
names you pass.
require_exists : bool
Scoped refresh only. True (default): an unknown name raises before
anything is written. False: unknown names are logged and skipped, and
the others are refreshed (what a migration wants).
Returns¶
int Number of edge changes written (0 when nothing changed).
Build the lineage DiGraph without the persisted table — fallback for when
FS_LINEAGE_GRAPH is missing/empty (never built) or unreadable.
Re-derives the full graph in memory by running
build_teradata_dependency_graph over every registered process view +
dataset (the same seed set rebuild_lineage_graph uses) and merging the
edges. Slower than reading the table (it runs SHOW VIEW), but it means the
query layer (get_descendants / get_ancestors / impact check / pyvis)
still works on a store whose graph was never persisted.
The persisted lineage at a past valid time: the primary store's base table
read VALIDTIME AS OF (the public view carries its own CURRENT VALIDTIME).
Load the persisted lineage graph into a networkx.DiGraph with a single
read of FS_V_LINEAGE_GRAPH (CURRENT VALIDTIME).
Edge direction is parent -> child (parent depends on child). Each node carries
an object_type attribute ('T'/'V'/'D'). All ancestor / descendant traversal
is then done in-memory — no recursive SQL.
as_of (a date or timestamp) reads the graph as it was at that valid
time: the primary store's base table, VALIDTIME AS OF. Every lineage write
is diff-based, so its history is real. At a past date there is no on-the-fly
fallback (it would describe today's DDL) and no read federation.
tag_managers (default True) marks the TimeManager / FilterManager objects
with a node attribute role ('time manager', 'filter manager',
'manager table'), from the process catalog and the manager registry. The
stored OBJECT_TYPE is unchanged ('V'/'T'). Fail-open.
Fallback: when the persisted table is missing/empty or unreadable and
auto_build=True (default), the graph is re-derived on the fly via
:func:_build_digraph_on_the_fly so callers still work on a store whose graph
was never persisted (run :func:rebuild_lineage_graph to persist it and make
subsequent loads fast).
Returns¶
networkx.DiGraph
Canonical in-graph node id: 'DB.OBJECT' (uppercased, de-quoted).
Objects that depend (directly or transitively) on database.object_name
— the impact / reverse direction (nx.ancestors over a depends-on
DiGraph, since edges point parent -> child = downstream -> upstream).
as_of: read the graph at that valid time (ignored when graph is given).
Returns canonical node ids ('DB.OBJECT').
Upstream sources that database.object_name depends on (directly or
transitively) — nx.descendants over the depends-on DiGraph.
as_of: read the graph at that valid time (ignored when graph is given).
Returns canonical node ids ('DB.OBJECT').
Return the persisted lineage in the same {'nodes', 'edges', 'roots'}
dict shape produced by :func:build_teradata_dependency_graph, so existing
consumers (plot_lineage_sankey, graph_to_migration_manifest,
skill_lineage) work unchanged on persisted data.
Each node also carries role ('time manager' / 'filter manager' /
'manager table' / None), set once at load.
Parameters¶
seed : str, optional
If given ('DB.OBJECT' or 'OBJECT'), restrict to the subgraph of
the seed plus all its ancestors and descendants.
as_of : str, optional
Read the graph as it was at that valid time.
Close the lineage edges of an object that no longer exists. Never raises.
Its own edges (as parent) and the edges pointing at it (as child) are closed
with CURRENT VALIDTIME DELETE, so the graph at a past date (as_of)
still shows it. purge=True removes every period (NONSEQUENCED), matching
a purge of the catalog rows. Returns the number of statements that ran.
Resolve an object node to its registered process PROCESS_ID, or None
when the object is not a registered process view (a plain view / table /
dataset).
Some views are processes, some are not — the lineage table stores objects
only, so the link is resolved here through the authoritative process catalog
(FS_V_PROCESS_CATALOG: VIEW_NAME <-> PROCESS_ID). Matching is done
on the de-quoted, uppercased (database, object) to be robust to the
quoting variance of stored VIEW_NAME values.
True if (database, object_name) exists in DBC.TablesV (case-insensitive).
Normalise and verify a user-supplied list of object names so case / "
quoting / db.name variance never causes a silent mismatch.
Each input ("db"."NAME", db.name, NAME, any case) is parsed with
the same _parse_node_name + _norm helpers record_process_lineage uses to
write the table, so a resolved name compares equal to stored rows. An
object is considered to exist if it is a node in the persisted lineage graph,
or (fallback, for a freshly-created view not yet in the graph) present in
DBC.TablesV.
Parameters¶
objects : str | Iterable[str]
One name or a list of names.
require_exists : bool
When True (default), raise ValueError naming every unmatched input.
When False, unmatched inputs are logged and dropped.
graph : networkx.DiGraph, optional
Pre-loaded lineage graph (avoids a reload when the caller already has one).
Returns¶
(resolved, unmatched)
resolved is a list of canonical (DB_UPPER, OBJECT_UPPER) tuples
(default DB filled from tdfs4ds.SCHEMA when the input was unqualified);
unmatched is the list of original input strings that resolved to nothing.
Deterministic compile check for the process views downstream of changed objects.
When a non-process view changes its result columns, the process views that
read it can silently break. This runs EXPLAIN SELECT * FROM <process_view>
on every registered process downstream of the given object(s) — EXPLAIN only
verifies the query compiles (all columns resolve), it does not execute it.
Same compile-validity idiom as the pre-check in upload_features.
Parameters¶
objects : str | Iterable[str]
Changed object name(s); resolved via :func:resolve_objects.
Returns¶
dict
{"checked": int, "ok": [view, ...],
"broken": [{"view", "process_id", "error"}, ...]}.
Broken processes are also logged as a warning.
plot_lineage_pyvis(graph: Dict[str, Any], *, output_path: str = 'tdfs4ds_lineage.html', height: str = '750px', width: str = '100%', notebook: bool = False)
Render a lineage graph as an interactive pyvis network and write it to a self-contained HTML file.
Accepts the same {'nodes', 'edges', 'roots'} dict shape produced by
:func:get_persisted_graph / :func:build_teradata_dependency_graph.
Node colour encodes object type (view/table/dataset), matching
plot_lineage_sankey. Edge direction is parent -> child (depends-on).
pyvis is a tdfs4ds dependency; it is imported lazily so building/persisting lineage never pays the import cost unless a visualisation is requested.
Returns¶
pyvis.network.Network
The configured network (already written to output_path).