tdfs4ds.dependencies¶
What depends on what in a feature store, and how to remove objects in order.
A dataset reads feature values of one process (FEATURE_VERSION = <PROCESS_ID>
in its view), and a process view can read a dataset or another process's view.
Removing a process, a feature or a dataset that other objects still read leaves
them broken or frozen without a word: the dataset keeps returning the values the
removed process last wrote, and a lineage rebuild then drops the link.
Three layers, as in the workflow assessment:
- :func:
collect_dependency_inputs- the only I/O (catalogs, persisted lineage, dataset view DDL, existence in DBC), fail-open: a failed read is recorded inunverified, never read as "nothing depends on it"; - :func:
build_dependency_graph, :func:dependents, :func:plan_removal_fromand :func:lineage_gaps- pure; - :func:
generate_removal_script/ :func:write_removal_script- a runnable.pythat is a dry run unless started with--apply.
:func:check_removal is the guardrail remove_process, remove_feature and
remove_dataset call before deleting anything. No language model is involved.
module-attribute
¶
module-attribute
¶
_PAIR_RE = compile("FEATURE_ID\\s*=\\s*'?\\s*(\\d+)\\s*'?\\s*AND\\s+FEATURE_VERSION\\s*=\\s*'([^']+)'", IGNORECASE)
module-attribute
¶
_TEMPLATE = '#!/usr/bin/env python\n"""Removal script generated by tdfs4ds {version} on {generated_at}.\n\nRemoves, in dependency order (the objects that read others first):\n{summary}\n\nRun it as is for a DRY RUN: each step says what it would remove, nothing is\nchanged. Run it with --apply to remove. It stops at the first failed step and\nlists the steps left; when every step ran, it checks that nothing is left behind\n(catalog rows, documentation, provenance, SQL versions, vector-index entries)\nand exits with code 2 otherwise. Stored values are {values}.\n\nConnection: TDFS4DS_TD_HOST / USERNAME / PASSWORD (or LOGDATA) in the\nenvironment or a .env file, as tdfs4ds.reconnect_teradata() reads them.\n"""\nimport json\nimport sys\n\nPLAN = json.loads({plan_literal})\n\n\ndef main():\n apply = "--apply" in sys.argv[1:]\n import tdfs4ds\n from tdfs4ds.utils.reconnect import reconnect_teradata\n if not reconnect_teradata():\n sys.exit("reconnect_teradata() failed: set TDFS4DS_TD_HOST / USERNAME / PASSWORD.")\n if PLAN.get("schema"):\n tdfs4ds.SCHEMA = PLAN["schema"]\n if PLAN.get("schema_view"):\n tdfs4ds.SCHEMA_VIEW = PLAN["schema_view"]\n from tdfs4ds.dependencies import execute_step, preview_step\n steps = PLAN["steps"]\n print(("APPLY" if apply else "DRY RUN") + f": {{len(steps)}} step(s)")\n for i, step in enumerate(steps, 1):\n print(f"[{{i}}/{{len(steps)}}] {{step[\'action\']}} {{step[\'object\']}} <- {{step[\'reason\']}}")\n if not apply:\n print(" would remove: " + preview_step(step))\n continue\n try:\n execute_step(step)\n print(" done")\n except Exception as exc:\n print(f" FAILED: {{exc}}")\n left = steps[i:]\n if left:\n print("Steps not run:")\n for s in left:\n print(f" - {{s[\'action\']}} {{s[\'object\']}}")\n sys.exit(1)\n if not apply:\n print("Nothing was changed. Run again with --apply to remove.")\n return\n from tdfs4ds.dependencies import verify_removal\n print("Checking what is left (catalogs, documentation, vector index)...")\n check = verify_removal(PLAN)\n for s in check["steps"]:\n print((" ok " if not s["left"] else " LEFT ") + s["action"] + " " + s["object"]\n + ("" if not s["left"] else ": " + "; ".join(s["left"])))\n if not check["clean"]:\n print("Some objects left documentation or index entries behind (listed above).")\n sys.exit(2)\n print("Clean: nothing left behind.")\n\n\nif __name__ == "__main__":\n try:\n main()\n finally:\n try: # end the session: an open one keeps a slot on the PE\n import teradataml as tdml\n tdml.remove_context()\n except Exception:\n pass\n'
Bases: ValueError
A removal refused because other objects depend on the object, or because what depends on it could not be read.
dependents: [{"kind", "label", "reason"}] in removal order;
unverified: the reads that failed; plan: a :func:plan_removal
result that removes the object and every dependent, in order.
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
An attached view (genai/attached_objects.py, 0.3.5.2), keyed by its 'DB.OBJECT' node id.
[(FEATURE_ID, FEATURE_VERSION, column, storage database)] a dataset
view reads. Pure.
The DDL is parsed with :func:tdfs4ds.dataset.ddl_spec.parse_dataset_ddl;
when it is not a view tdfs4ds can read back, the FEATURE_ID = n AND
FEATURE_VERSION = '<pid>' predicates are read directly (no column name, no
database). The database tells which store the ids belong to: FEATURE_IDs and
PROCESS_IDs are those of the store owning the storage table.
(reads, current): the 6-tuples of :func:dataset_table_reads and whether the
view reads the current valid time (None when the view cannot be parsed). Pure.
dataset_table_reads(ddl: str) -> List[Tuple[int, str, Optional[str], Optional[str], Optional[str], Optional[str]]]
[(FEATURE_ID, FEATURE_VERSION, column, storage database, storage table,
join type)] a dataset view reads, from the parsed spec. Pure.
:func:dataset_reads keeps its 4-tuples for the callers that unpack them. With
the regex fallback (a view tdfs4ds cannot read back) the table and the join
type are None; join type is 'INNER' or 'FULL OUTER'.
{'DB.OBJECT'} of the tables and views of databases, or None when DBC
could not be read. One read.
collect_dependency_inputs(*, snapshot=None, graph=None, dataset_ids=None, max_ddl: int = 300, existence: bool = False, read_graph: bool = True, emit=None) -> dict
Everything the pure layer needs.
snapshot / graph: already-read catalogs (store_consistency.
collect_catalog_snapshot) and lineage graph, to avoid reading them twice.
dataset_ids: read the view DDL of these datasets only (None = every
registered dataset, at most max_ddl). existence: also read which views
and tables exist (DBC), for the lineage-gap checks.
Objects a process view reads: the registered processes and datasets it
reaches through plain views (stopping there), and every object on the way.
Returns (processes, datasets, unregistered_datasets, visited).
The dependency DiGraph (upstream -> dependent), pure.
Nodes P:<pid>, D:<dataset id>, F:<domain>|<entity>|<name> (or
F#<id> for a FEATURE_ID absent from the catalog) carry kind, label
and missing (referenced but not registered: a gap). The graph also keeps,
in graph.graph, the inputs of each process view (inputs), the
datasets whose view was not read (reads_unknown) and the existence set.
What removing targets breaks, pure.
targets: node keys, or (node key, process_id) for a feature removed as
produced by one process (nothing breaks when another process computes it).
Returns {"order": [node keys, dependents first, targets last],
"dependents": [{"key", "kind", "label", "reason", "missing"}],
"lineage_needed": bool} (a dataset or process was reached while the
lineage graph is unknown: its readers could not be listed).
(targets, unknown). An object is a PROCESS_ID or a process view name, a
DATASET_ID or a dataset view name (db.name or bare), or a feature as a
tuple (name, entity, data_domain[, process_id]) or a dict with keys
feature, entity, data_domain, process_id.
plan_removal_from(H, targets, *, purge_values: bool = False, drop_views: bool = False, unknown=(), unverified=()) -> dict
The ordered removal steps for targets and everything that depends on them, from a dependency graph. Pure; nothing is executed.
Datasets that may read the given processes' values or features: their feature list (no view read needed) names one of those FEATURE_IDs.
plan_removal(objects, *, purge_values: bool = False, drop_views: bool = False, inputs: dict = None, max_rounds: int = 5) -> dict
The removal steps for objects and every object depending on them, in order.
objects: process ids or process view names, dataset ids or names, and
features as (name, entity, data_domain[, process_id]). Nothing is
executed: pass the result to :func:write_removal_script, or read
plan["steps"]. purge_values=True adds the deletion of the removed
versions' stored values (irreversible); drop_views=True drops the views.
Only the dataset views that may be concerned are read (their feature list names a feature of a removed process), round after round as the plan grows.
Refuse a removal that would break other objects. Called by
remove_process / remove_feature / remove_dataset unless
force=True.
kind: 'process' (ref = PROCESS_ID), 'dataset' (ref = DATASET_ID or
name) or 'feature' (ref = (name, entity, data_domain)). Returns the
plan when nothing depends on the object; raises
:class:DependentObjectsError when something does, or when what depends on
it could not be read.
Run one plan step. Called by the generated script with --apply only;
the removal log records it with origin.
Current business-dictionary rows of an object (description, sections, columns), or of one of its columns.
The vector-index entries of an object (or of some of its columns).
Chroma: {"backend": "chroma", "entries": n}, counted by id. Teradata Vector
Store: entries cannot be counted (embeddings are keyed by row position) and
follow the documentation at each refresh, so the state of the collections is
reported: {"backend": "teradata", "failed": [collections not ready]}.
What one step removes, counted now (read only): catalog rows, documentation
(business dictionary, process documentation, provenance), stored SQL
versions, vector-index entries, stored values. Before a removal it is what
will go (the dry run); after it, everything must be 0 (:func:verify_removal).
A count that could not be read is None.
What a footprint still holds: names with a count above 0, an index not clean, or a count that could not be read.
What one step would remove, read only (counts and existence).
After a removal, check that every step left nothing behind: catalog rows,
documentation (business dictionary, process documentation, provenance),
stored SQL versions, vector-index entries and, with purge_values, stored
values. Read only.
Returns {"clean": bool, "steps": [{"action", "object", "left": [...]}]}.
A step whose feature catalog entry stays for another process is checked for
this process's part only. Values kept on purpose (no purge_values) are
not counted.
A runnable .py for a :func:plan_removal result: a dry run unless
started with --apply. Pure ASCII. Refuses a plan with no step.
Write :func:generate_removal_script to out_path (default
{tdfs4ds.SCHEDULING_SCRIPT_DIR}/tdfs4ds_removal_<ts>.py); return the path.
Holes in the lineage: datasets reading a removed process or feature,
features no process computes, process views reading an object that is gone
or an unregistered dataset, catalog rows whose view is gone, and lineage
edges to views that are no registered process. Each gap carries the dead
branch (what :func:dependents reaches from it) and the processes it
affects. explain: callable(view) -> error text or None, run on the
processes whose inputs include an object absent from DBC. Pure but for it.
The feature reads of every dataset of this store whose view was parsed (with
their storage table), for the storage-level check: {dataset_id, dataset,
fid, version, column, database, table, join, current, feature, domain, pids}.
pids: the producing process and the processes reading the dataset. Pure.
Add, to each gap left by a removed process, when and by whom it was
removed and what its view was, from FS_REMOVAL_LOG. Fail-open.
None when EXPLAIN SELECT * FROM <view> compiles, else the error text.