tdfs4ds.lineage.network_health¶
The lineage network as a whole: does every view compile, is the network in one piece, which branches lead nowhere (0.3.4.0).
Works on the persisted lineage graph (load_lineage_digraph): nodes are
'DB.OBJECT' with an object_type ('T' table, 'V' view, 'D' dataset), and an
edge goes from a consumer to what it reads (downstream -> upstream). So the
inputs of a node are its descendants, its consumers its ancestors.
Three layers, like the rest of the workflow assessment:
- :func:
compile_viewsrunsexplain(view)on each view (the only I/O, and it is injected); - :func:
network_modelis pure: root broken views and what they break, the connected networks, the dead branches; - :func:
network_findingsturns the model into workflow findings.
Names the graph holds that are not database objects (SQL tokens such as
APPLY or 1, a known defect of the lineage builder) are left out of
every count: a token shared by two SQL texts would otherwise join two
unrelated networks.
Every view to EXPLAIN: the views and datasets of the lineage graph that exist in the database, plus every registered process and dataset view (also those the graph does not hold). With existing None (dictionary unreadable) only the registered views: an intermediate name could be a SQL token. Sorted.
True when error is the database's own answer (the view really does not
compile), False when the request never reached it (no session, a dropped
connection): only the first is a finding. Same rule as the connection-error
classifier: a message formatted from a database response carries
[Teradata Database].
True when error is a refusal for lack of a right ("The user does not have SELECT access to ..."): the view may be fine, this user cannot verify it.
compile_views(views: Iterable[str], explain: Callable[[str], Optional[str]], *, max_views: int = 1000, progress=None) -> dict
{"results": {view: None | error text}, "not_checked": [...],
"unanswered": {view: error}, "no_access": {view: error}}. explain(view)
returns None when EXPLAIN SELECT * FROM view compiles. An error the
database did not give (see :func:database_answered) goes to unanswered,
never to results: a lost session must not read as broken views. A refusal
for lack of a right goes to no_access (0.3.4.5): nothing says the view is
broken, only that this user cannot check it. Never raises.
Views of the lineage graph from which no edge leaves. Pure.
A view always reads something, so a view the graph gives no input means the lineage stopped there: its definition could not be read when the lineage was recorded (no right on it, typically), or it was never expanded. Dataset views ('D') are excluded (their inputs are processes, recorded separately).
Names absent from existing are NOT dropped: DBC.TablesV lists only the
objects this user has a right on, so an object it may not read is absent
there too. They come last (SQL tokens the builder stored as names are among
them); :func:probe_dead_ends tells them apart (SHOW VIEW: 3807 vs a refusal).
probe_dead_ends(views: Iterable[str], show: Callable[[str], tuple], *, max_views: int = 200, progress=None) -> Dict[str, dict]
Why the lineage stops at each view. show(view) returns (ddl, error)
(the I/O, injected: SHOW VIEW). {view: {"reason", "detail", "fix"}} with reason
"no_access": the definition cannot be read by this user (the right is named);"not_recorded": it can be read and names inputs, the graph just lacks them;"unreadable": another database refusal (whole message).
A view whose readable definition names no input (SELECT 1, feature store
tables only) is fine and left out. Never raises.
network_model(G, *, process_nodes: dict, dataset_nodes: dict, existing=None, compiled: Optional[dict] = None, consumed_pids=None, dead_ends: Optional[dict] = None, attached_nodes=None) -> dict
The network seen as a whole. Pure.
process_nodes: {node: {"pid", "label"}} for the registered processes
(node = their view); dataset_nodes: {node: {"label"}}. existing: the
'DB.OBJECT' set read from the dictionary (None = unknown). compiled: the
output of :func:compile_views. consumed_pids: processes something reads
according to the catalogs (dataset views), added to what the graph says.
dead_ends: the output of :func:probe_dead_ends (where the lineage stops).
attached_nodes: the 'DB.OBJECT' nodes of the attached views (0.3.5.2): they consume
what they read, so a process they read is not unused and their branch is not detached.
Findings (area='lineage') from :func:network_model. reported: the
objects the lineage-gap check already reported (no second finding for them).
removal_call(objects) -> str: the removal script call to name in a fix.