tdfs4ds.lineage.time_drivers¶
Which TimeManager a process really reads (0.3.1.240).
Two notions are kept apart:
- declared - the pairing recorded in
FS_TIME_MANAGER(byupload_features(..., time_manager=),run(..., time_manager=),roll_outor a migration). It is whatrun(pid)uses when no manager is passed. - real - the process view depends on a manager view, directly or through
intermediate views. Read from the persisted lineage graph (
FS_LINEAGE_GRAPH).
A TimeManager object is a view over its <name>_HIDDEN table. Known managers come
from the process catalog (TIME_* / FILTER_* columns), the optional manager
registry, and the manager being checked. A view in the lineage that reads the hidden
table of a known TimeManager is a manager too (a soft clone: same dates, own cursor).
The core (:func:manager_index, :func:managers_read, :func:verdict_for) is pure
and takes a networkx.DiGraph whose edges go parent -> child (= reader -> source),
like :func:tdfs4ds.lineage.lineage_store.load_lineage_digraph. Lineage outcomes
mirror :mod:tdfs4ds.scheduling.upstream_check: an empty graph and a process absent
from it are both UNKNOWN - never reported as "reads no manager".
Minimal object carrying what record_process_time_manager reads.
(view_node, table_node) of a TimeManager / FilterManager object.
manager_index(proc_df=None, registry_df=None, extra: Iterable = ()) -> Dict[str, Dict[str, Optional[str]]]
Known manager objects: {'time': {view_node: table_node}, 'filter': {...}}.
proc_df: the process catalog (TIME_* / FILTER_* columns).
registry_df: FS_V_MANAGER_REGISTRY rows (KIND, DATABASE_NAME,
VIEW_NAME, TABLE_NAME). extra: (kind, view_node, table_node).
Time-manager views: the known ones plus any view in G that reads only the
hidden table of a known one (a soft clone: clone_timer(clone_mode='soft')).
A view reading that table and something else is a consumer, not a manager.
Filter managers are excluded.
Mark manager objects in G with a node attribute role (in place).
TimeManager views (including soft clones found in the graph) get
'time manager', FilterManager views 'filter manager', their hidden
tables 'manager table'. Returns {node: role}. Pure.
:func:manager_index from the process catalog's manager columns and the
registry: one small raw read. Fail-open (an empty index).
(lineage_status, time_manager_views) read by process_node, deep.
UNKNOWN when the graph is empty or does not contain the process view.
A manager is read when its view is among the process view's ancestors (sources,
any depth). A process reading a manager's hidden table directly (no view) is
reported as reading that manager.
Verdict on a manager passed by the caller.
shared_manager_views(G, manager_node: str, process_nodes=None, *, process_node: Optional[str] = None, process_id: Optional[str] = None, readers_graph=None)
Views that name a TimeManager and that processes read (0.3.6.8). Pure.
A process reading a manager through such a view cannot get its own soft clone:
rewriting the view for one process changes it for every reader. Returns
[{view, manager, readers}], one per view of G reading manager_node that is
neither a process view (process_nodes: {node: pid}), a dataset (its
time_manager= build is its own setting) nor a manager.
process_node: keep only the views that process reads (any depth), and count process_id among the readers (before its lineage is recorded). readers_graph: the graph the readers are read from (default G): the processes above each view. Views read by no process are left out. Empty when the graph does not know the manager.
Lineage DiGraph of one view, built from its DDL (no persisted graph needed).
Used before a process is registered, when FS_LINEAGE_GRAPH cannot know it yet.
time_manager_dependency(process_id: str, time_manager=None, *, graph=None, proc_df=None, registry_df=None, view_name: Optional[str] = None) -> Dict[str, Any]
What a process reads and what it declares.
Returns {"process_id", "view", "status" (LINKED/NOT_LINKED/UNKNOWN),
"reads": [db.view, ...], "declared": db.view or None, "passed": db.view or None,
"verdict" (OK/UNRELATED/MISMATCH/UNKNOWN, only with *time_manager*)}; with
time_manager also same_calendar (soft clones of it the process reads) and
other_managers. Never raises. graph / proc_df / registry_df: an already loaded
lineage graph / process catalog / manager registry (read when omitted); view_name:
the process view when the process is not registered yet.
check_passed_manager(process_id, time_manager, where: str = 'run', *, graph=None, proc_df=None, view_name: Optional[str] = None) -> Dict[str, Any]
Check a manager handed to where against what the process view reads.
Returns the :func:time_manager_dependency dict plus "record" (whether the
pairing may be recorded). Under TIME_MANAGER_CHECK='error' an UNRELATED or
MISMATCH manager raises ValueError (nothing recorded, nothing computed);
under 'warn' (default) it is a warning and the pairing is not recorded;
'off' skips the check.
A real-time run of a process whose view reads a TimeManager: warn.
The values would be stamped with the run time while the view computes at the manager's business date. Returns the dependency when a warning was emitted.
Declare (or correct) the TimeManager of processes from what their views read.
For each process (all registered ones by default) the lineage graph says which
TimeManager the view reads, deep. Returns a pandas DataFrame with one row per
process: PROCESS_ID, VIEW_NAME, DECLARED, READS, ACTION
(RECORD / CORRECT / OK / AMBIGUOUS / NONE / UNKNOWN) and
APPLIED. dry_run=True (default) writes nothing; dry_run=False records
RECORD and CORRECT rows in FS_TIME_MANAGER. A process reading two
managers is left alone (AMBIGUOUS). process_ids: the processes to check (default
all); graph / proc_df: an already loaded lineage graph / process catalog.