Skip to content

fs-inspect

Audit the feature store — catalogs, follow-up runs, re-execution, removal.

name: "Feature Store: Inspect" description: "Generate code to audit the tdfs4ds feature store — list registered processes, feature and dataset catalogs, execution follow-up history, the feature catalog joined with processes and documentation (export to XLSX/HTML), and how to re-run or remove processes. Invoke when the user wants to explore, monitor, or troubleshoot the feature store. SKIP when the user needs performance analysis, EXPLAIN scoring, or a FilterManager rewrite proposal for a specific process — use fs-analyze instead." argument-hint: "[data-domain][scope: processes|features|datasets|followup|all]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

Generate Python code to explore and monitor the tdfs4ds feature store.

User context: $ARGUMENTS

Step Skill Why
Before fs-setup Connect first if no Teradata context is open.
Action fs-document / fs-lineage Once a process is found, run documentation or lineage.
Followup fs-rollout Re-execute a registered process across time.
Stuck fs-analyze If a run is slow, perform the full scalability analysis.

Demos: demos/notebooks/04 - Re-run and monitor.ipynb, A2 - Operating the store.ipynb, 10 - Process versions.ipynb, 10b - The SQL beneath a process.ipynb.

Ask for any missing values before generating code: - Data domain — project namespace (must match what was used during registration). - Scope — what to inspect: processes, features, datasets, followup, or all.

Emit the relevant sections from the following pattern:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # teradataml context from TDFS4DS_TD_HOST / _USERNAME / _PASSWORD
tdfs4ds.connect(database='<DATABASE>')
import teradataml as tdml
tdfs4ds.select_data_domain('<DATA_DOMAIN>')   # or set TDFS4DS_DATA_DOMAIN in .env

# ---------------------------------------------------------------------------
# Catalogs
# ---------------------------------------------------------------------------
process_cat = tdfs4ds.process_catalog()   # PROCESS_ID, VIEW_NAME, ENTITY_NAME, ...
feature_cat = tdfs4ds.feature_catalog()   # FEATURE_NAME, ENTITY_NAME, FEATURE_TABLE, ...
dataset_cat = tdfs4ds.dataset_catalog()   # VIEW_NAME, ENTITY_NAME, COMMENT, ...

tdfs4ds.get_dataset_entity()              # entity per dataset view
tdfs4ds.get_dataset_features()            # feature versions per dataset view

# ---------------------------------------------------------------------------
# Each feature with its producing process, documentation and SQL (all domains by default)
# ---------------------------------------------------------------------------
df = tdfs4ds.feature_catalog_documented(data_domains='<DATA_DOMAIN>')   # DataFrame
# tdfs4ds.export_feature_catalog('catalog.xlsx')                         # or .html; as_of='2026-03-31' for a past date

# ---------------------------------------------------------------------------
# Discover entities and features (scoped to DATA_DOMAIN)
# ---------------------------------------------------------------------------
from tdfs4ds.feature_store.feature_query_retrieval import (
    get_list_entity,
    get_available_features,
)

get_list_entity()

entity = '<ENTITY_NAME>'
get_available_features(entity_id=[entity])

# ---------------------------------------------------------------------------
# Execution follow-up — most recent runs first
# ---------------------------------------------------------------------------
follow_up = (
    tdfs4ds.process_store.process_followup.follow_up_report()
    .sort('START_DATETIME', ascending=False)
)
follow_up

# ---------------------------------------------------------------------------
# Look up a process ID by view name
# ---------------------------------------------------------------------------
from tdfs4ds.process_store.process_query_administration import get_process_id

list_views = process_cat[['VIEW_NAME']].to_pandas().VIEW_NAME.values
view_name  = [v for v in list_views if '<SUBSTRING>' in v][0]
process_id = get_process_id(view_name=view_name)
print(f'{view_name}  →  {process_id}')

# ---------------------------------------------------------------------------
# Re-execute a process (refresh features at current time)
# ---------------------------------------------------------------------------
tdfs4ds.run(process_id=process_id)

# ---------------------------------------------------------------------------
# Remove a process from the catalog (metadata only — feature data is preserved)
# ---------------------------------------------------------------------------
# tdfs4ds.process_store.process_query_administration.remove_process(process_id)

print('tdfs4ds version:', tdfs4ds.__version__)

Follow-up status values

Status Meaning
COMPLETED Process ran successfully
RUNNING Currently executing, or crashed before writing FAILED
FAILED,... Exception raised; suffix contains the error message

A stale RUNNING record after a crash means followup_close did not execute — the feature data for that run is incomplete.

Resource usage of each execution

Every execution runs under a query band, so its DBQL rows join its follow-up row. The figures are kept beyond DBQL's retention in FS_EXECUTION_RESOURCES / FS_EXECUTION_STATEMENTS (copied at each run()):

frame, info = tdfs4ds.read_execution_resources(process_ids=None, days=30, as_of=None)  # CPU, I/O, spool, PROCESS_VERSION, DBQL_STATUS
stats = tdfs4ds.process_run_statistics(frame, by_version=True)   # runs, frequency, median run / CPU per process version
statements, info = tdfs4ds.execution_statements(execution_id)     # per statement, SQL while DBQL holds it
tdfs4ds.harvest_execution_resources()                            # copy what DBQL has finished writing now

info["state"] says when DBQL is not readable (GRANT SELECT ON DBC.QryLogV) or logs nothing (BEGIN QUERY LOGGING WITH SQL ON <user>); DBQL writes every 10 minutes, so the latest executions read PENDING. A run groups a call's segments / roll-out dates: count runs, not follow-up rows. The admin app shows this as Operations › Execution log, and in the feature catalog card.

When was each entity last confirmed? (track_entity_refresh)

An unchanged value is not rewritten, so a feature row's ValidStart is the last change; FS_FOLLOW_UP is per run and segment. A process registered with upload_features(..., track_entity_refresh=True) (or tdfs4ds.set_process_options(pid, track_entity_refresh=True)) also records, per batch, the entities it confirmed in an admin table FS_R_* (not a feature, in no catalog, listed by the EFS map as Entity refresh (admin)):

tdfs4ds.entity_refresh('<PROCESS_ID>', entities=None, as_of=None, history=False)   # REFRESH_TIME / LOAD_TIME / RUN_ID per entity
s = tdfs4ds.entity_refresh_summary('<PROCESS_ID>')   # by_refresh_time, n_not_confirmed + not_confirmed sample, state / message

A failed write is a warning (the run stays COMPLETED); a run by an older client writes nothing, and the summary then says not tracked (older client or failed write) instead of treating the silence as dropped entities. With a FilterManager a run confirms only its segment.

Housekeeping: statistics, temporary tables, SQL versions

# Statistics of the storage tables: read-only diff of DBC.StatsV against what ingestion expects
tdfs4ds.check_storage_statistics(domains=['<DATA_DOMAIN>'])          # OK / MISSING / NO_STATS / TABLE_MISSING
# tdfs4ds.check_storage_statistics(domains=[...], add_missing=True)  # define the missing ones (ingestion's options)

# tdfs_tmp_* tables left by registrations before 0.3.3.0: list, then drop
tdfs4ds.cleanup_temp_tables(dry_run=True)                           # exact tdfs_tmp_<12 hex>, older than 1 h
# tdfs4ds.cleanup_temp_tables(dry_run=False)

# The SQL beneath a process, version by version (the view and every view it reads)
tdfs4ds.process_sql_versions('<PROCESS_ID>')                        # versions with their validity
tdfs4ds.process_sql('<PROCESS_ID>', version=None, as_of=None)       # the stored chain (None = current)
print(tdfs4ds.process_sql_diff('<PROCESS_ID>', 1, 2))               # what changed between two versions
# tdfs4ds.capture_process_sql(all_processes=True, dry_run=True)     # record processes registered before versioning

A new process version is recorded when the SQL under the view changes (a TimeManager / FilterManager moving its date is not a change). The PROCESS_ID and the stored values' FEATURE_VERSION never change; a value's process version is the one in force at its ValidStart.

Querying the catalogs in SQL

The objects behind the accessors (read the FS_V_* views; the tables are written by tdfs4ds):

What Table View
features FS_FEATURE_CATALOG FS_V_FEATURE_CATALOG
processes FS_PROCESS_CATALOG FS_V_PROCESS_CATALOG
datasets FS_FS_DATASET_CATALOG FS_V_FS_DATASET_CATALOG
dataset features / entities FS_FS_DATASET_FEATURES / _ENTITY FS_V_FS_DATASET_FEATURES / _ENTITY
lineage FS_LINEAGE_GRAPH FS_V_LINEAGE_GRAPH
  • The doubled FS_FS_ is real: DATASET_CATALOG_NAME is 'FS_DATASET' and the class adds its own FS_.
  • The feature catalog names the entity column ENTITY_NAME, the process catalog ENTITY_ID.
  • FEATURE_VIEW (feature catalog) is the store's serving view of the values; the view that computes the feature is VIEW_NAME in the process catalog.
  • FEATURE_NAMES (process catalog) is a comma-separated string: join with commas on both sides, or TOTAL_SPEND also matches TOTAL_SPEND_12M: ',' || p.FEATURE_NAMES || ',' LIKE '%,' || TRIM(f.FEATURE_NAME) || ',%'. Prefer process_catalog() / feature_catalog() joins in Python when you can.

Teradata traps when reading them:

  • LIKE has no [...] classes: LIKE 'FS[_]%' matches nothing. Escape with LIKE 'FS!_%' ESCAPE '!'.
  • DBC.TablesV.TableName is padded: TRIM() it. Read DBC with tdml.execute_sql, not DataFrame.from_query (a view over DBC needs WITH GRANT OPTION, Error 3523).
  • BEGIN(VALIDTIME) / END(VALIDTIME) are refused (Error 3706): read the ValidStart / ValidEnd columns under NONSEQUENCED VALIDTIME.
  • No ORDER BY inside DataFrame.from_query (Error 3706): use .sort(), or rank with QUALIFY ROW_NUMBER() OVER (...) for a top-N.
  • A tdml.DataFrame holds a query, not rows: printing one kept across a run() re-runs it and shows the new values. Take .get_values() / .to_pandas() when you need a "before" snapshot.

Rules

  • All catalog queries are scoped to DATA_DOMAIN — always set it before querying.
  • get_process_id(view_name=...) looks up by the crystallized view name (e.g. 'FEAT_ENG_CUST'), not the feature or entity name.
  • remove_process removes only catalog metadata; underlying temporal feature tables and data are preserved. It never touches the feature catalog.
  • Since 0.3.3.27, remove_process, remove_feature and remove_dataset(..., dry_run=False) refuse when other objects depend on the object (a dataset reading the process's values, a process view reading its view or the dataset), or when that cannot be read; they raise tdfs4ds.DependentObjectsError listing the dependents in the order they must be removed. Do not reach for force=True (it leaves the dependents broken): propose path = tdfs4ds.write_removal_script(tdfs4ds.plan_removal([pid])) (add purge_values=True only when the user wants the stored history deleted), then python <path> (dry run) and python <path> --apply; after --apply the script checks that no catalog row, documentation, provenance, SQL version or vector-index entry is left (exit code 2 otherwise; tdfs4ds.dependencies.verify_removal(plan) does the same check). Every removal and refusal is recorded in FS_REMOVAL_LOG (tdfs4ds.get_removal_log()). The admin app's Operations › Remove objects plans, dry-runs and downloads the script; it never removes anything itself.
  • A view attached with attach_objects (0.3.5.2) that reads the dataset or process counts as a dependent: the removal is refused and the plan has a detach_object step for it (the view itself is never dropped; rewrite or drop it by hand).
  • To remove a process with its features: check the impact first (tdfs4ds.lineage.lineage_store.get_descendants(db, view)), then call tdfs4ds.remove_feature(name, entity_id=[...], data_domain=..., process_id=pid) for each feature of the process, then remove_process(pid). With process_id, a feature also computed by another process keeps its catalog entry (only pid's documentation of it goes); a feature computed by pid alone is removed entirely. Never call remove_feature without process_id on a shared feature: it removes it for every producer.
  • The clock of a process's data domain (0.3.6.0): tdfs4ds.get_process_info(pid)['DOMAIN_TIME_MANAGER'] (None = the domain has no clock), tdfs4ds.get_domain_time_manager('<DOMAIN>'), and which processes are on it: tdfs4ds.align_domain_time_managers('<DOMAIN>', dry_run=True) (ACTION OK / CLONE / NONE / OTHER). The admin app's process card (Explore › Feature catalog) has a Domain clock row; the Report's Time drivers lists each clock with the processes on and off it.
  • For a deliverable listing the features with their processes and documentation (a data catalog for a business team, an audit), use tdfs4ds.export_feature_catalog(path, data_domains=..., as_of=...): one row per feature × producing process, XLSX (About / Features / Processes / SQL, linked: a process name opens its SQL, one line per row, with links back) or HTML. Missing values read unknown; the SQL is today's text even at a past as_of. The admin app shows it as Explore › Feature catalog.