tdfs4ds.serve.store_consistency¶
Catalog snapshot at a valid time, and the store's consistency findings.
Every count in the admin app's Report, EFS map and documentation coverage is read from the catalogs, which are valid-time tables: a count only means something next to the valid time it was read at. This module is the one place that reads the catalogs for those panels and says which valid time it used:
collect_catalog_snapshot(as_of=None)- the feature, process and dataset catalogs at the current valid time (theFS_V_*views, unchanged reads), orVALIDTIME AS OFa timestamp (base tables, primary store only), plus the feature history needed to tell a retired feature from one that never existed.build_consistency_model(...)(pure) - what does not line up: features without a producer, processes whose view is gone or that never ran, storage tables nothing references, datasets reading features the catalog no longer has, and - when a storage scan exists - FEATURE_IDs stored without a catalog row. Every finding names the command that fixes it; nothing here writes.scan_storage_orphans(...)- the one expensive read (one grouped pass per storage table), run only on demand.
A feature is the triplet (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME): a name registered under two entities, or in two domains, is two features.
module-attribute
¶
KIND_LABELS = {'no_producer': 'Registered feature with no current producer', 'retired': 'Feature retired from the catalog', 'table_missing': 'Storage table missing', 'stored_unregistered_known': 'Stored values of a de-registered feature', 'stored_unregistered_unknown': 'Stored values with no catalog row ever', 'no_value': 'Registered feature with no value at this valid time', 'view_missing': 'Process view missing', 'not_registered': 'Process whose features are not registered', 'never_run': 'Process never run', 'unused': 'Process whose features no dataset reads', 'several_producers': 'Feature listed by several processes', 'table_unreferenced': 'Storage table no feature references', 'table_retired_only': 'Storage table referenced only by retired features', 'dataset_stale': 'Dataset reading features absent from the catalog', 'dataset_view_missing': 'Dataset view missing', 'dataset_not_in_lineage': 'Dataset absent from the lineage graph'}
module-attribute
¶
_CSS = '\n<style>\n.tdc-wrap{font-family:inherit}\n.tdc-recon{background:#EAF0F6;border-radius:8px;padding:10px 14px;margin:6px 0 12px;color:#00233C}\n.tdc-vt{font-size:.85rem;color:#5A6B7B;margin:0 0 8px}\n.tdc-wrap details{border:1px solid #E3E8EE;border-radius:8px;margin:6px 0;background:#fff}\n.tdc-wrap summary{cursor:pointer;padding:8px 12px;font-weight:600;color:#00233C}\n.tdc-wrap table{border-collapse:collapse;width:100%;font-size:.85rem}\n.tdc-wrap th{background:#00233C;color:#fff;text-align:left;padding:5px 8px}\n.tdc-wrap td{border-top:1px solid #E3E8EE;padding:5px 8px;vertical-align:top}\n.tdc-wrap code{font-size:.8rem;word-break:break-all}\n.tdc-sev{display:inline-block;border-radius:10px;padding:0 8px;font-size:.75rem;font-weight:700}\n.tdc-error{background:#FDECEA;color:#B42318}.tdc-warning{background:#FFF4E5;color:#B54708}\n.tdc-info{background:#EAF0F6;color:#00233C}\n.tdc-badge{background:#FF5F00;color:#fff;border-radius:10px;padding:0 8px;font-size:.75rem;margin-left:6px}\n.tdc-ok{color:#027A48}\n.tdc-note{font-size:.8rem;color:#5A6B7B;margin:4px 12px}\n.tdc-wrap details>p{margin:6px 12px}\n</style>'
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Order-insensitive entity key: catalog and process lists differ in order.
pandas frame with the primary index moved back into the columns.
Raw SQL to pandas. Not DataFrame.from_query: it wraps the statement, and a temporal qualifier must open the request.
None / '' / 'current' -> None (current valid time); else the body of a
TIMESTAMP '...' literal. A value without an offset is read as UTC - the
admin app says so next to the input - never in the session time zone.
The dataset catalog's table names, from DatasetCatalog's naming authority (without its constructor, which creates missing objects).
The four VALIDTIME AS OF reads (exposed for tests).
The feature catalog at the current valid time, whatever the session's temporal default (the public view has no qualifier of its own).
Every feature ever registered in a store, with the span of its catalog rows.
table_schema: the store's table database (None = the primary, the
historical SQL unchanged). A store's history is read from its base table:
its public view (LOCK ROW FOR ACCESS SELECT *) carries no valid time, so
a NONSEQUENCED read through it is Error 9330 (seen live).
Feature history of the primary and, store by store, of each attached
read-only store (0.3.3.15), rows tagged FEATURE_STORE when there are
several. Reading another store's base table needs SELECT on it, which a
reader may not have: such a store is skipped (logged), never a failure of the
primary's read.
Feature / process / dataset catalogs at a valid time, plus feature history.
as_of=None reads the same accessors as before (FS_V_* views,
read federation included). A timestamp reads the primary store's base
tables VALIDTIME AS OF it. Raises ValueError on an unreadable date.
The persisted lineage graph at the snapshot's valid time (managers tagged), or None when it cannot be read. No on-the-fly build: an unpersisted graph is reported as such, not rebuilt from today's DDL.
What exists in the database now (DBC.TablesV, batched): the storage tables
of the store's databases, and the existence of every catalog-referenced
table and view. Fail-open: ok=False when DBC could not be read.
{entity_key: {"tables": n, "pi": {COL: k}, "part": {COL: k}}}: for each
entity, how many of its storage tables have each entity column in their
primary index (k) and in their partitioning. One batched dictionary read
(DBC IndicesV + PartitioningConstraintsV). {} when it cannot be read.
{PROCESS_ID: number of follow-up rows}; None when unreadable.
One grouped pass per storage table: rows per FEATURE_ID, all versions and
at the valid time. {(DB, TABLE): {"all": {fid: rows}, "at": {fid: rows}}}
("error" instead when the table could not be read).
by_version=True (the dataset-row check) groups by (FEATURE_ID,
FEATURE_VERSION) instead and adds "by_version": {"all": {(fid, version):
rows}, "at": {...}} to each table; "all" / "at" are then the sums per
FEATURE_ID, equal to what the default pass returns. The default output and SQL
are unchanged.
Number of processes: distinct PROCESS_ID (a joined catalog row can repeat one).
The one definition of the feature, process and dataset counts.
Used by the Report, the EFS map and the consumer agent, so every surface gives the same numbers for the same scope. Pure.
- a feature is a (data domain, entity, name) triplet; a process is a distinct PROCESS_ID; a dataset is a distinct DATASET_ID;
domain/entityrestrict the scope (entity compared order-insensitively);run_pids: process ids having at least one follow-up row, or None when the follow-up was not read (never_runis then None, not 0).
Returns the counts plus the notes the Report shows: no_producer (registered
feature no current process lists), several_producers, not_registered (process
none of whose features is registered) and never_run; producers maps each
feature triplet to the sorted process ids listing it (the feature's versions).
build_consistency_model(snapshot, dictionary=None, run_counts=None, scan=None, lineage_nodes=None) -> dict
Findings + the reconciliation of the feature count. Pure.
lineage_nodes: node ids ('DB.OBJECT') of the persisted lineage graph, or
None when it was not read (an empty graph is also not checked: that is a
store without lineage, reported elsewhere, not N missing datasets).
One sentence explaining every feature number the panels show.
'Consistency & orphans' body: reconciliation, then one group per area.