Skip to content

Export the feature catalog

The feature catalog alone says which features exist. The export joins each feature with the process that produces it and its documentation: one row per feature and producing process, as an Excel workbook or a self-contained HTML page. The admin app shows the same table in its Feature catalog panel.

import tdfs4ds

tdfs4ds.export_feature_catalog('catalog.xlsx')                       # every domain, current valid time
tdfs4ds.export_feature_catalog('catalog.html')                       # same content, one HTML file
tdfs4ds.export_feature_catalog('claims.xlsx', data_domains='CLAIMS')  # one domain
tdfs4ds.export_feature_catalog('q1.xlsx', data_domains=['CLAIMS', 'RETAIL'], as_of='2026-03-31')
tdfs4ds.export_feature_catalog('prov.xlsx', processes=['V_PROVIDER', 'V_PHARMACY'])  # some processes

df = tdfs4ds.feature_catalog_documented(data_domains='CLAIMS')       # the same table as a DataFrame

Without a path, the file goes to tdfs4ds.DASHBOARD_REPORT_DIR as tdfs4ds_feature_catalog_<store>[_<date>]_<timestamp>.<xlsx|html>.

What it contains

Group Columns
Feature DATA_DOMAIN, ENTITY, FEATURE_NAME, FEATURE_ID, FEATURE_TYPE, STORAGE_TABLE
Documentation FEATURE_DESCRIPTION, DESCRIPTION_SOURCE (manual, imported, propagated, generated…), DESCRIPTION_EDITOR, DESCRIPTION_SINCE
Process PROCESS_ID, PROCESS_VIEW, PROCESS_TYPE, FILTER_MANAGER, TIME_MANAGER, BUSINESS_LOGIC, ENTITY_DESCRIPTION
Use and health DATASETS, LAST_RUN, LAST_STATUS, USER_SCORE, OPTIMIZATION_SCORE (the EXPLAIN scores)
  • The XLSX has four sheets, linked together:
  • About: store, valid time, domains, counts, and a link to each sheet.
  • Features: the table above. Rows are grouped by domain, entity and process, and the domain, entity and feature-name columns stay visible when you scroll right. The process name is a link to its SQL.
  • Processes: one row per process. Its name and its SQL column open the SQL, and its feature list opens its first feature.
  • SQL: each process's SQL, one line per row, under a header with links back to the process's first feature and to its row in Processes.

Features and Processes are Excel tables (filter dropdowns, banded rows). A link can jump to a row but not apply a filter; since a process's features are grouped, the others follow the first one. - The HTML has the features table and, for each process, its details and its SQL folded under a SQL heading. A feature's process name links to it.

A feature is the triplet (data domain, entity, name). The same name under two entities is two features, each linked to its own process. A feature listed by several processes (versions) has one row per process. A feature that no current process lists has one row reading no producing process.

The feature description is the one the consumer agent and the app use: the business-dictionary column of the process view (where hand edits are written). When there is none, the export uses the description stored with the process documentation.

Values that are not available read unknown. For example: - the EXPLAIN scores when the process was never documented by an LLM; - an undocumented feature; - any source that could not be read (the About sheet and the top of the HTML name it, with the whole error).

Two values have their own words: a feature in no dataset reads none, and a process that has no run reads never run.

Domains, processes and valid time

  • data_domains: None (all), one name, or a list. Names are matched case-insensitively. An unknown name raises with the list of existing ones: a typo never produces an empty file that looks complete.
  • processes: None (all), or one or more producing processes, each given by its PROCESS_ID or its view name (with or without the database, any case). Only the features they produce are kept; the About sheet names them and counts what is kept. An unknown process raises with the list of existing views.
  • as_of: the store is temporal, so the catalog and the documentation change over time.
  • as_of=None is the current valid time. A date or timestamp (no offset = UTC) reads the catalogs, the business dictionary, the process documentation and the provenance as they were at that time.
  • The last run is the last one that started at or before that time.
  • The SQL is always today's text: view definitions are not versioned. An export at a past date says so.
  • A catalog row's valid time is its registration time, except for features first registered during a TimeManager roll-out: those carry the business date of the roll-out. At a date between that business date and the registration of their process, they appear with no producing process.

The SQL: from the process view down to the sources

A process view often reads other views, so its own SQL is not enough to see how a feature is computed. The SQL of a process is therefore its chain:

  1. the process view first: it gives the result;
  2. then every view it reads, each before the views it reads in turn, down to the source tables.

Each step says which step(s) read it. The chain stops at: - source tables; - dataset views, because they belong to the feature store: their own processes are documented on their own; - TimeManager and FilterManager views, including a soft clone (a view reading another manager's _HIDDEN table); - the feature store's own objects (FS_T_*, FS_V_*).

These are listed as the chain's inputs.

How the chain is found: - From the lineage graph, which is read once per export. - From the SQL itself for a process missing from the graph; the export then suggests running tdfs4ds.rebuild_lineage_graph(). - Objects the graph holds by mistake (SQL tokens such as APPLY) that do not exist in the database are left out and named. - Each view is read once, even when several processes share it.

Colours. Each step has its own colour, and each kind of input has its own colour: - in the app, a reference to a step is tinted in that step's colour and opens the step when clicked; - in the HTML export, it links to the step; - in Excel, it is bold in that colour (rich text), and the line links to the step. A cell holds one link, so a line naming several steps links to the first.

The About sheet has the legend.

The SQL in Excel

The SQL sheet holds one line of SQL per row, so a process's SQL has no overall size limit. Each step of the chain starts with a header row in its colour. An Excel cell holds at most 32,767 characters, so a single line longer than that continues on the next rows, marked continued in the next column. Join those pieces without a line break to get the line back. Nothing is cut.

In the admin app

Explore › Feature catalog shows the same rows, 20 per page. - Filters: data domains (none selected = all), processes (none selected = all; the list follows the selected domains), a date (blank = current) and a search over feature, entity, description and process. Show only this process, in the side panel, filters the table on the process shown. - The table keeps its width and scrolls horizontally inside its box, whose bottom stays on screen, so the scroll bar is visible when the side panel is open; the header stays visible while the rows scroll. - Process detail: selecting a row opens a side panel with the feature, its process (with its data domain and entity) and the SQL chain, read when you open it. - The process view is open and the views it reads are folded; click a reference to open the step it names. - Drag the panel's left edge to resize it; Escape closes it. - The feature in its SQL: a Computed as block gives the feature's expression at each level of the view and its grain. In the SQL, the computation is underlined and every mention of the feature is marked; go to the computation scrolls to it. - Several producing processes: such a feature has one row per process, marked 1 of N producing processes. The side panel lists the other processes, each opening its own SQL. - Exports: Export HTML and Export XLSX download the file for the current domains, processes and date, SQL included. - Refresh: the panel reads the catalog once per store and date; Refresh reads it again.