tdfs4ds.lineage.network¶
dataclass
¶
Graph node metadata.
__init__(name: str, object_type: str, database: Optional[str] = None, object: Optional[str] = None, ddl: Optional[str] = None, table_info: Optional[Dict[str, Any]] = None, error: Optional[str] = None) -> None
Extracts source and target tables/views from a SQL query.
Analyse Teradata CREATE TABLE DDL and return
{ 'primary_index_columns': [...], 'partition_columns': [...], 'partitioning_levels': [...], 'partitioning_by_column': {...} }
Critical behavior: - Ignores keyword-like text inside single-quoted literals - Searches PRIMARY INDEX / PARTITION BY only in the table-options tail (after the column-definition block).
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Execute a SQL query or a list of queries using the tdml module.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
str or list
|
A single SQL query string or a list of SQL query strings. |
required |
raise_error
|
bool
|
If True, re-raise exceptions after printing them. Default is False. |
False
|
fetch_results
|
bool
|
If True, fetch and return the results of the SQL execution. Default is False. |
False
|
quiet_errors
|
tuple
|
Lowercase substrings identifying failures the caller
EXPECTS and will handle itself. A matching failure is logged at DEBUG
instead of ERROR; whether it is re-raised is still governed by
|
()
|
log_errors
|
bool
|
False when the caller re-raises ( |
True
|
Returns: The result of the SQL execution if a single query is passed. None if a list of queries is passed or an exception occurs.
Return name as a Teradata identifier, quoting it only if required.
Idempotent: an already-quoted name is returned unchanged rather than being
wrapped a second time. Embedded quotes are escaped as "".
Parameters¶
name : str
The raw object/database name.
force : bool, default False
Quote even when the name does not need it. Use only at the few sites
that have always emitted unconditional quotes and whose exact output is
already baked into persisted DDL (see
feature_store/feature_query_retrieval.py) — elsewhere forcing would
change the SQL of existing stores for no reason.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Accepts names like
- '"sales"."orders"'
- '
."orders"' - 'sales.orders'
- 'orders' (rare; treat as object only)
Returns (database, object).
Thin alias for the package-wide quoter (kept for existing call sites).
Note this is now CONDITIONAL: an ordinary name comes back bare, so the SHOW statements below are byte-identical to what they were for a normal database and correctly quoted for one that needs it.
Build a safe qualified name for SHOW statements.
- If node_name includes db, use it.
- If it is
Executes SQL and returns a list of strings from the first column. Works for SHOW VIEW / SHOW TABLE outputs (usually one column with text lines).
Pre-fetches all registered feature store datasets as a set of (DATASET_DATABASE.upper(), DATASET_NAME.upper()) tuples. Returns an empty set if the catalog is unreachable.
Parses a dataset view DDL and returns the unique FEATURE_VERSION values found in it.
FEATURE_VERSION is the PROCESS_ID of the process that produced the feature, so this gives us all process UUIDs whose features are consumed by the dataset. Order-preserving deduplication is applied.
_fetch_view_names_for_process_ids(schema: str, process_catalog_view: str, process_ids: List[str]) -> List[str]
Given a list of PROCESS_IDs, returns the VIEW_NAME for each from the process catalog. VIEW_NAME is the feature-engineering view registered for that process (e.g. '"demo_user"."FEAT_ENG_CUST"'), not the internal feature-store storage tables.
Returns an empty list if the catalog is unreachable or no rows are found.
Try SHOW VIEW first; if fails, try SHOW TABLE. Returns (object_type, ddl_text). object_type is "view" or "table".
build_teradata_dependency_graph(sql_query: str, *, default_database: Optional[str] = None, max_depth: int = 50, store_ddl: bool = False, stop_at_feature_store_datasets: bool = True, expand_datasets_via_process_catalog: bool = True) -> Dict[str, Any]
Builds a dependency graph by
1) analyzing the initial SQL via analyze_sql_query 2) for each discovered source object: - SHOW VIEW and re-run analyze_sql_query on its DDL to expand dependencies - or SHOW TABLE and run analyze_teradata_ddl to attach PI/partition metadata
Feature store dataset views (registered in the FS_V_DATASET_CATALOG) are treated as named "dataset" nodes when stop_at_feature_store_datasets=True (the default). Instead of drilling into their underlying SQL, the graph queries the dataset feature catalog (FS_V_{catalog}_FEATURES) to find which registered process views contribute to the dataset (one process = one view), and adds edges to those views directly. Set expand_datasets_via_process_catalog=False to treat datasets as pure leaves.
Graph edges are parent -> child (parent depends on child).
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
{ "nodes": { node_name: { ...NodeInfo... }, ... }, "edges": [ (parent, child), ... ], "roots": [ ...targets from initial SQL... ], |
Dict[str, Any]
|
} |
graph_to_migration_manifest(graph: Dict[str, Any], *, filter_database: Optional[str] = None) -> Dict[str, Any]
Converts a lineage graph into a migration manifest — a JSON-serialisable dict that lists every view and table discovered, with optional filtering to a single database (e.g. the feature store database).
Useful for planning a feature store migration: objects whose database matches
filter_database are the ones that need to be recreated or moved; edges
between them describe the recreation order.
Parameters¶
graph : dict
Output of build_teradata_dependency_graph.
filter_database : str, optional
When given, only nodes whose database field matches (case-insensitive)
are included. Edges whose source or target falls outside the filter are
also excluded. When None every node and edge is returned.
Returns¶
dict with keys: "views" : list of dicts {database, name, type} type is "view" or "dataset" "tables" : list of dicts {database, name, type, primary_index, partition_columns} type is "table" "edges" : list of dicts {from, to} (parent depends on child)
plot_lineage_sankey(graph: Dict[str, Any], *, title: str = 'Lineage Sankey', max_label_len: int = 45, show_full_name_in_hover: bool = True) -> go.Figure
Plot an interactive Sankey from a lineage graph like
{ "nodes": {name: {name, object_type, database, object, table_info, error, ...}, ...}, "edges": [(parent, child), ...], # parent depends on child "roots": [...] }
Hover displays relevant info (no DDL). Returns a plotly Figure.
Display Plotly figure robustly across environments.
- Tries normal fig.show() (Jupyter mime rendering)
- Falls back to browser renderer if nbformat is missing
- Optionally writes an HTML file