tdfs4ds.feature_store.statistics

tdfs4ds.feature_store.statistics

Review, complete and refresh the optimizer statistics of the feature storage tables.

Every FS_T_* table should carry the statistics expected_statistics describes — the same rule ingestion uses when it first collects them. Ingestion afterwards only REFRESHES what a table already has (``COLLECT STATS ON

``), so a statistic added to that rule later never reaches an existing table on its own. ``check_storage_statistics`` closes that gap: * it lists, per storage table, the statistics defined (``DBC.StatsV``) against the expected set, with the last collection date; * ``add_missing=True`` defines what is missing, with the same options ingestion would have used — so later merges keep it current; * ``refresh=True`` re-collects everything the table carries, now. It never drops a statistic: one that is defined but not expected (a DBA's own) is reported under ``EXTRA`` and left alone. The dictionary is read in batched ``IN`` lists — three round trips for the whole store, not three per table.

OK module-attribute

OK = 'OK'

MISSING module-attribute

MISSING = 'MISSING'

NO_STATS module-attribute

NO_STATS = 'NO_STATS'

TABLE_MISSING module-attribute

TABLE_MISSING = 'TABLE_MISSING'

_CHUNK module-attribute

_CHUNK = 200

_COLUMNS module-attribute

_COLUMNS = ['DATABASE_NAME', 'TABLE_NAME', 'DATA_DOMAIN', 'ENTITY', 'ROWS', 'N_DEFINED', 'N_EXPECTED', 'MISSING', 'EXTRA', 'OLDEST_COLLECTION', 'NEWEST_COLLECTION', 'STATUS', 'ACTION']

logger_safe

logger_safe(level, message, *args, **kwargs)

Wrapper around the global logger that only emits logs when tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.

qualified

qualified(database, object_name, force=False)

Build db.obj with each part quoted only as needed.

qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names); qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None database yields the bare object name, so unqualified callers keep working.

force=True quotes both parts unconditionally — only for the sites that have always emitted quotes and whose exact text is already persisted (dataset view DDL, returned volatile-table names).

sql_literal

sql_literal(value)

Escape value for use inside a single-quoted SQL string literal.

For the DBC probes and catalog predicates that compare a name as data (WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.

_norm

_norm(name)

_colset

_colset(cols)

_label

_label(cols)

process_storage_tables

process_storage_tables(process_ids=None, feature_df=None, process_df=None)

{PROCESS_ID: {"DB.TABLE", ...}} — the storage tables each process writes.

Process catalog (FEATURE_NAMES + DATA_DOMAIN) joined to the feature catalog (FEATURE_NAME + DATA_DOMAIN → FEATURE_DATABASE.FEATURE_TABLE). feature_df / process_df: catalog frames the caller already materialised (all data domains), to avoid reading them twice. Fail-open: {} on any error.

_storage_tables

_storage_tables(domains=None, processes=None, tables=None)

{(DB, TABLE): {'database', 'table', 'domains', 'entity'}} in scope.

_dbc_rows

_dbc_rows(view, columns, keys, where='')

Rows of a DBC view for the given (DB, TABLE) keys, batched. Fail-open.

tdml_rows

tdml_rows(sql)

_err

_err(exc)

The whole database message, without the driver's Go stack frames.

_read_dictionary

_read_dictionary(keys)

Existing tables, primary index, partitioning and statistics, per (DB, TABLE).

_expected

_expected(entity, primary_index, partitioning)

_collect_statement

_collect_statement(cols, extended, table_sql)

check_storage_statistics

check_storage_statistics(domains=None, processes=None, tables=None, add_missing=False, refresh=False, display=True)

Review the statistics of the feature storage tables; optionally complete and refresh them.

Parameters

domains : list of str, optional Restrict to the storage tables of these data domains. processes : list of str, optional Restrict to the tables these processes (PROCESS_IDs) write. tables : list of str, optional Restrict to these tables ('database.FS_T_...'). add_missing : bool Define the missing statistics (collecting them immediately), with the options ingestion uses. Later merges then keep them current. refresh : bool Re-collect every statistic the table carries (COLLECT STATS ON), the call ingestion makes after each merge. Multi-column statistics keep their THRESHOLD 15 PERCENT: Teradata skips them until the table has changed by 15%, which is why their collection date can be older. display : bool Print the report.

With both flags off (the default) nothing is executed: the call only reads the data dictionary.

Returns

pandas.DataFrame One row per table: STATUS is OK, MISSING (some expected statistics are not defined), NO_STATS, or TABLE_MISSING (the feature catalog references a table that no longer exists — never acted on); MISSING / EXTRA list column sets; OLDEST_COLLECTION / NEWEST_COLLECTION bound the collection dates; ACTION says what was done.