tdfs4ds.catalog_export¶
The feature catalog joined with its producing processes and its documentation.
One row per feature x producing process, exported as HTML or XLSX, at a valid
time, for all data domains or some. Three layers, like serve/efs_map.py:
- :func:
collect_feature_export- the only I/O, batched reads (never one per feature), every temporal table read at the same valid time; - :func:
build_feature_export- pure: the join and the tables; - :func:
render_feature_export_html/ :func:write_feature_export_xlsx.
Public: :func:export_feature_catalog (a file) and
:func:feature_catalog_documented (a DataFrame).
Rules:
- a feature is the triplet (data domain, entity, name) and is linked to its
processes through
catalog_counts' producers - never by name within a domain, which can hold one name for two entities; - a value that is not available is written
unknown(a known empty value has its own word: a feature in no dataset isnone, a process that never ran isnever run); - a view's definition is not versioned: the SQL is always today's text, and an export at a past date says so.
module-attribute
¶
__all__ = ['export_feature_catalog', 'feature_catalog_documented', 'collect_feature_export', 'build_feature_export', 'render_feature_export_html', 'write_feature_export_xlsx', 'FEATURE_COLUMNS', 'PROCESS_COLUMNS', 'UNKNOWN']
module-attribute
¶
SQL_TODAY_NOTE = 'SQL as it is today: no process version was stored for that valid time (process SQL versions are stored from 0.3.3.12 on).'
module-attribute
¶
SQL_STORED_NOTE = 'SQL of the process version in force at that valid time, as stored when it was captured; a process without a stored version shows its SQL as it is today.'
module-attribute
¶
SQL_SCOPE_NOTE = "SQL: each process's view first (the result), then every view it reads, down to the source tables; the chain stops at dataset views (feature store) and Time/FilterManagers, listed as inputs."
module-attribute
¶
FEATURE_COLUMNS = ['DATA_DOMAIN', 'ENTITY', 'FEATURE_NAME', 'FEATURE_ID', 'FEATURE_TYPE', 'STORAGE_TABLE', 'FEATURE_DESCRIPTION', 'DESCRIPTION_SOURCE', 'DESCRIPTION_EDITOR', 'DESCRIPTION_SINCE', 'PROCESS_ID', 'PROCESS_VIEW', 'PROCESS_TYPE', 'FILTER_MANAGER', 'TIME_MANAGER', 'BUSINESS_LOGIC', 'ENTITY_DESCRIPTION', 'DATASETS', 'LAST_RUN', 'LAST_STATUS', 'USER_SCORE', 'OPTIMIZATION_SCORE']
module-attribute
¶
PROCESS_COLUMNS = ['PROCESS_ID', 'DATA_DOMAIN', 'ENTITY', 'PROCESS_VIEW', 'PROCESS_TYPE', 'PROCESS_VERSION', 'FEATURES', 'FILTER_MANAGER', 'TIME_MANAGER', 'BUSINESS_LOGIC', 'ENTITY_DESCRIPTION', 'LAST_RUN', 'LAST_STATUS', 'USER_SCORE', 'OPTIMIZATION_SCORE', 'SQL_VERSIONS', 'LAST_SQL_CHANGE', 'PROCESS_VERSION_AT', 'RUNS_30D', 'RUN_FREQUENCY', 'MEDIAN_RUN_SEC', 'MEDIAN_RUN_CPU_SEC', 'SQL']
module-attribute
¶
INPUT_KINDS = (KIND_TABLE, KIND_DATASET, KIND_TIME_MANAGER, KIND_FILTER_MANAGER, KIND_MANAGER, KIND_FS, KIND_UNKNOWN)
module-attribute
¶
module-attribute
¶
KIND_COLOURS = {KIND_TABLE: '475569', KIND_DATASET: '0D9488', KIND_TIME_MANAGER: '6D28D9', KIND_FILTER_MANAGER: '6D28D9', KIND_MANAGER: '6D28D9', KIND_FS: '64748B', KIND_UNKNOWN: '94A3B8'}
module-attribute
¶
_CSS = '\n.td-fc{font-family:Inter,Arial,sans-serif;color:#14273A;background:#fff;margin:0;padding:24px 32px;line-height:1.45}\n.td-fc h1{color:#00233C;font-size:26px;margin:0 0 4px}.td-fc h2{color:#00233C;font-size:20px;margin:28px 0 10px}\n.td-fc .sub{color:#5A6B7B;margin:0 0 16px}.td-fc .bar{height:4px;background:#FF5F00;width:80px;margin:6px 0 18px}\n.td-fc table{border-collapse:collapse;width:100%;font-size:13px}\n.td-fc th{background:#00233C;color:#fff;text-align:left;padding:6px 8px;position:sticky;top:0}\n.td-fc td{border-bottom:1px solid #E2E8F0;padding:6px 8px;vertical-align:top}\n.td-fc tr:nth-child(even) td{background:#F5F7FA}.td-fc .u{color:#9AA9B8;font-style:italic}\n.td-fc .kv td:first-child{font-weight:600;width:220px;color:#00233C}\n.td-fc .proc{border:1px solid #E2E8F0;border-radius:10px;padding:12px 16px;margin:12px 0}\n.td-fc .proc h3{margin:0 0 8px;color:#00233C;font-size:16px}\n.td-fc pre{background:#00233C;color:#E8ECFB;padding:12px;border-radius:8px;overflow:auto;font-size:12px;\n white-space:pre-wrap}.td-fc summary{cursor:pointer;color:#B84400;font-weight:600}\n.td-fc .note{background:#FFF1E8;border-left:4px solid #FF5F00;padding:8px 12px;margin:8px 0}\n.td-fc a{color:#B84400}\n'
module-attribute
¶
XLSX_PROCESS_COLUMNS = ['PROCESS_VIEW', 'PROCESS_ID', 'DATA_DOMAIN', 'ENTITY', 'PROCESS_TYPE', 'PROCESS_VERSION', 'FEATURES', 'FILTER_MANAGER', 'TIME_MANAGER', 'BUSINESS_LOGIC', 'ENTITY_DESCRIPTION', 'LAST_RUN', 'LAST_STATUS', 'USER_SCORE', 'OPTIMIZATION_SCORE', 'SQL_VERSIONS', 'LAST_SQL_CHANGE', 'PROCESS_VERSION_AT', 'RUNS_30D', 'RUN_FREQUENCY', 'MEDIAN_RUN_SEC', 'MEDIAN_RUN_CPU_SEC', 'SQL']
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
A FEATURE_ID as an int, whatever form it is read in (int, ' 6', 6.0, Decimal); None if not one.
(qualifier, source) of a documentation / run read.
At the current valid time with read-only stores offering capability:
every such store's public view (which applies CURRENT VALIDTIME itself, so
no qualifier; a qualifier over a union of views is Error 9330). Otherwise
the primary's base table opened with _vt(t), as before 0.3.3.15: past
valid times are read from the primary store only.
One read; a failure is recorded under key (the whole message) and gives an empty frame.
sql_head WHERE <column> IN (...) in batches of 500, concatenated.
None -> None (all); a name or a list -> the store's spelling of each.
Case-insensitive. An unknown name raises with the existing ones: a typo must not give an empty export that looks complete.
The art_sql statements of a UAF ART process (METADATA JSON), else [].
Where feature_name is computed in sql: :func:tdfs4ds.agent.sql_context.computation_context
(expression per level, grain, window, filters, and each level's character span). None if unreadable.
[(start, end)] of every mention of the feature name, quoted or not, as a whole identifier.
A reference in the app: data-step makes it clickable (app.js), the class gives its colour.
highlight_sql_html(sql: str, feature_name: str, computation: Optional[dict] = None, refs: Optional[list] = None, ref_markup=None) -> str
The SQL as <pre class="td-sqlhl"> with the feature's computation and its mentions marked.
<span class="def">: the feature's item at each level of the computation (the outer SELECT item, then the derived tables / CTEs it reads);<mark class="name">: every other mention of the feature name;- refs (:func:
reference_spans): each reference wrapped byref_markup(target, inner_html)(:func:ref_markup_appby default).
Everything is escaped; the marks never nest across each other (the text is cut at every span boundary and each piece wrapped on its own).
{"text", "error"}: a process view's SQL (SHOW VIEW, today's text). Never raises.
entry with a UAF ART process's art_sql statements appended (they run before the view is read).
The input kind of a node the chain does not walk through, or None for a view to walk.
The views to show for a process view, and the inputs where the chain stops (pure).
graph: a networkx.DiGraph with edges parent -> child (a view -> the
object it reads) and node attributes object_type ('V'/'T'/'D') and
role, as :func:tdfs4ds.lineage.lineage_store.load_lineage_digraph
returns it.
Returns {"views": [node, ...], "read_by": {node: [step numbers]},
"inputs": [{"name", "kind", "read_by"}], "in_graph": bool, "cycle": bool}.
views[0] is the process view; each view comes before the views it reads.
{node: TableKind} for the nodes that exist (one DBC read), or None if DBC is unreadable.
The view's lineage built from its SQL, datasets as leaves (lineage never recorded for it).
read_sql_chain(view_name, graph=None, process_row=None, cache: Optional[dict] = None, dbc_kinds=None) -> dict
The SQL of a process view and of every view it reads, down to the sources. Never raises.
graph: the lineage graph (loaded when None); cache: {node: {"text",
"error"}} shared between calls so a view is read once; dbc_kinds: the
DBC lookup (injected in tests). Returns {"steps": [{"name", "text",
"error", "read_by"}], "inputs": [...], "note": str, "left_out": [...]};
steps[0] is the process view (with_art_sql applied when
process_row is given).
The chain as one text: each step under a -- [k/n] marker, the inputs last.
[(start, end, target)] of every reference in sql to a step or an input of chain.
target is ("step", k) or ("input", kind). A qualified name is
matched as a whole name (DB.X never inside DB.X_HIDDEN); a bare
name only right after FROM / JOIN. Spans never overlap.
Everything the export needs, read at one valid time.
as_of: None = current valid time, else a date / timestamp (no
offset = UTC). data_domains: None = all, a name or a list.
with_sql: read each process view's SQL (one SHOW VIEW per view).
A failed read is recorded in errors and its values become unknown;
an unreadable catalog raises.
{PROCESS_ID upper: {"count", "last_change"}} from version periods
(PROCESS_ID, PROCESS_VERSION, ValidStart). last_change is the start of
the latest version when the SQL changed at least once, else none. Pure.
The Features and Processes tables and the summary. Pure.
The model restricted to some producing processes (pure).
processes: None = all (the model is returned unchanged); a
PROCESS_ID or process view name, or a list of them (case-insensitive; a
view may be qualified or not). An unknown one raises ValueError with
the existing views. The summary counts what is kept (a feature with
several producing processes is still counted as such when only one of them
is kept); process_filter names the processes kept.
A process's SQL chain for the HTML export: a sub-heading per step, references coloured and linked to the step they name, the inputs last (inline styles, no script).
One self-contained page (no script, no external request).
A text cut into pieces an Excel cell can hold (the caller puts them on consecutive rows).
[(piece, continued)]: one entry per SQL line; a line longer than a cell
continues on the next rows (continued=True). Joining lines with a newline,
and continuation pieces with nothing, gives the text back exactly.
Inverse of :func:_sql_rows (used by tests and readers).
A SQL line as Excel rich text: references bold in their target's colour; plain text if none.
Write a process's SQL chain under its header (row r); returns the last row written.
Each step: a header row filled with the step's colour, then one row per SQL line with the references in their target's colour (rich text); a line naming a step links to that step's header (the first one, when it names several). The inputs come last, each in its kind's colour.
Sheets About, Features, Processes and SQL, linked together; returns path.
export_feature_catalog(path: Optional[str] = None, *, fmt: Optional[str] = None, as_of=None, data_domains=None, with_sql: bool = True, processes=None) -> str
Write the feature catalog, joined with its processes and documentation, to a file.
Parameters¶
path : str, optional
Where to write. Default: tdfs4ds.DASHBOARD_REPORT_DIR /
tdfs4ds_feature_catalog_<store>[_<date>]_<timestamp>.<fmt>.
fmt : {'xlsx', 'html'}, optional
Taken from the extension of path when omitted; 'xlsx' by default.
as_of : str or datetime, optional
Valid time: None = current, else a date or timestamp (no offset =
UTC). Catalogs, business dictionary, process documentation and
provenance are read at that time; the SQL is today's (view definitions
are not versioned, and the export says so).
data_domains : str or list of str, optional
None = all; one name or a list (case-insensitive). An unknown name
raises ValueError with the existing ones.
with_sql : bool
Include each process's SQL (one SHOW VIEW per process view).
processes : str or list of str, optional
None = all; a PROCESS_ID or process view name, or a list of them:
only the features those processes produce (see :func:filter_processes).
Returns¶
str The path written.
feature_catalog_documented(as_of=None, data_domains=None, with_sql: bool = False, processes=None) -> pd.DataFrame
The Features table of :func:export_feature_catalog as a DataFrame.
One row per feature x producing process. With with_sql=True a SQL
column carries the producing process's SQL. processes keeps only some
producing processes (see :func:filter_processes).