fs-lineage-graph

Maintain the persisted lineage graph (FS_LINEAGE_GRAPH): full/scoped rebuild, downstream EXPLAIN impact check, re-document the impact slice, and render an interactive pyvis network.

---

name: "Feature Store: Persisted Lineage Graph" description: "For a developer maintaining the persisted lineage graph (FS_LINEAGE_GRAPH). Build or fully refresh the graph, scoped-refresh just the objects you edited, check which downstream processes still compile after a change, re-document from the persisted graph, and render an interactive pyvis network. Invoke when the user edited a view/process and wants to refresh lineage/documentation without re-deriving everything, or asks \"what depends on this?\" / \"did my change break anything downstream?\". SKIP for a one-off on-the-fly Sankey from a single process — use fs-lineage; SKIP for natural-language business-user Q&A — use ca-lineage." argument-hint: "[object name(s) you changed, or empty for a full rebuild]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

---

Generate Python to manage the persisted, temporal lineage graph FS_LINEAGE_GRAPH. Unlike fs-lineage (which re-derives lineage on the fly for one seed), this graph is stored once and queried in-memory with networkx — so you get the reverse / impact direction ("what depends on X?") cheaply, and can re-document only the affected slice after an edit.

User context: $ARGUMENTS

Related skills

Step	Skill	Why
One-off lineage	fs-lineage	On-the-fly dependency graph + Sankey for a single process/dataset (no persistence).
Documentation	fs-document	Document a single process; this skill re-documents the whole store or an impact slice.
Find objects	fs-inspect	List process/dataset catalogs to pick object names.
Business Q&A	ca-lineage	Natural-language "where does this come from?" through the consumer agent.

Demos: demos/notebooks/A1 - Lineage and impact.ipynb.

Ask for any missing values before generating code: - Objects — the object name(s) you edited (for a scoped refresh), or none for a full rebuild. Names are case / "-quote / db.name insensitive and verified (a typo raises). - LLM config — only needed if rebuild_documentation will run; lineage + impact check are deterministic and need no model.

Emit the relevant pattern, substituting real values:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # teradataml context from TDFS4DS_TD_HOST / _USERNAME / _PASSWORD
tdfs4ds.connect(database='<DATABASE>', create_if_missing=True)
import teradataml as tdml
tdfs4ds.select_data_domain('<DATA_DOMAIN>')   # or set TDFS4DS_DATA_DOMAIN in .env

# ---------------------------------------------------------------------------
# 1. Full rebuild / init  — re-derive the whole graph from every registered
#    process view + dataset (all data domains). Diff-based, not a wipe:
#    unchanged edges are left alone, new ones inserted, vanished ones closed,
#    and edges of objects that no longer exist pruned.
# ---------------------------------------------------------------------------
n_edges = tdfs4ds.rebuild_lineage_graph()          # all_data_domains=True
print(f"persisted {n_edges} edges")

# ---------------------------------------------------------------------------
# 2. Scoped refresh  — you edited a (possibly NON-process) view. Refresh only
#    its edges (no global wipe) and check whether downstream processes still
#    compile.  Names are verified; a typo raises ValueError.
# ---------------------------------------------------------------------------
tdfs4ds.rebuild_lineage_graph(objects=['<view_you_changed>'], check=True)

# Standalone impact check (no graph write): run EXPLAIN SELECT * on every
# registered process downstream of the change — verifies the query still
# COMPILES (a changed view's result columns can silently break consumers).
res = tdfs4ds.check_downstream_processes(['<view_you_changed>'])
print("checked:", res['checked'], "| ok:", len(res['ok']), "| broken:", len(res['broken']))
for b in res['broken']:
    print("  BROKEN:", b['view'], "->", b['error'])   # fix these processes

# ---------------------------------------------------------------------------
# 3. Query the persisted graph in-memory (networkx — one table read, no SHOW VIEW)
# ---------------------------------------------------------------------------
from tdfs4ds.lineage import get_descendants, get_ancestors, get_persisted_graph

get_descendants('<DB>', '<OBJECT>')   # impact: everything that depends on it
get_ancestors('<DB>', '<OBJECT>')     # upstream sources it depends on

# ---------------------------------------------------------------------------
# 4. Re-document from the persisted graph (needs an instruct model configured)
# ---------------------------------------------------------------------------
# Whole store, each process/dataset documented exactly once:
tdfs4ds.genai.rebuild_documentation()                       # force_update=False
# Only the impact slice of an edited object (object + all downstream), force-regenerated:
tdfs4ds.genai.rebuild_documentation(objects=['<view_you_changed>'])

# ---------------------------------------------------------------------------
# 5. Interactive pyvis visualisation (writes a self-contained HTML file)
# ---------------------------------------------------------------------------
from tdfs4ds.lineage import plot_lineage_pyvis

plot_lineage_pyvis(get_persisted_graph(), output_path='lineage.html')                 # whole graph
plot_lineage_pyvis(get_persisted_graph(seed='<DB>.<OBJECT>'), output_path='impact.html')  # neighbourhood

What each function does

Call	Effect
rebuild_lineage_graph()	Re-derive the whole graph from every process view + dataset (diff-based: no wipe, stale parents pruned).
rebuild_lineage_graph(objects=[...], check=True)	Refresh only those objects' edges (no wipe) + run the downstream compile check.
check_downstream_processes([...])	EXPLAIN SELECT * on every registered process downstream → {checked, ok, broken:[{view,process_id,error}], not_verifiable:[...]}. A refusal for lack of a right ("does not have SELECT access to ...") goes to not_verifiable, never to broken (0.3.4.5): the view may be fine, this user cannot check it.
get_descendants(db, obj) / get_ancestors(db, obj)	In-memory impact / upstream sets (one table read).
rebuild_documentation() / rebuild_documentation(objects=[...])	Document the whole store, or only the impact slice of edited objects (force-regenerated).
plot_lineage_pyvis(graph, output_path=...)	Interactive HTML network (datasets chocolate, views steelblue, tables seagreen).

Rules

• Edges are parent → child = parent depends on child. get_descendants is the impact direction (what breaks if this changes); get_ancestors is the sources direction.
• Object names are normalised (quotes stripped, upper-cased, db.name split) and verified against the persisted graph then DBC.TablesV. An unrecognised name raises ValueError — never a silent no-op.
• A view that must stay in the graph (for instance a user view joining a dataset with a source table, read by no process) is attached: tdfs4ds.attach_objects(['DB.VIEW']) (0.3.5.2). Its edges are recorded at once and every full rebuild_lineage_graph() seeds it. Prefer it to extra_views=, which a later full rebuild without that argument does not keep. An attached table read by no view has no edge. Removing a dataset or process an attached view reads is refused; the removal plan detaches the view and never drops it.
• The graph is created when missing by setup() / connect(create_if_missing=True); run rebuild_lineage_graph() once to populate it on a pre-existing store.
• rebuild_documentation needs tdfs4ds.INSTRUCT_MODEL_PROVIDER + ..._MODEL set; lineage refresh and check_downstream_processes are deterministic and need no model.
• tdstone2 TDS_* objects (scoring views, model/code repositories) are legitimate upstream nodes and appear in the graph — they are not feature-store internals.