Each returns a teradataml DataFrame — chain .to_pandas() for local
manipulation.
To get each feature together with its producing process, its documentation and the process SQL (as XLSX,
HTML or a DataFrame, for some domains, at a past date), see
Export the feature catalog.
fromtdfs4ds.process_store.process_query_administrationimportget_process_id# Fully qualified nameprocess_id=get_process_id('"my_db"."FEAT_ENG_CUSTOMER"')# Or just the view name (defaults to tdfs4ds.SCHEMA)process_id=get_process_id(view_name='FEAT_ENG_CUSTOMER')
FS_FOLLOW_UP is an event log, not a temporal table. Each row has the time the
run started (START_DATETIME) and the business date it computed
(VALIDTIME_DATE; NULL for a real-time run). followup_business_dates() reads
it per process and segment:
fromtdfs4ds.process_store.process_followupimportfollowup_business_datesfu=tdfs4ds.process_store.process_followup.follow_up_report().to_pandas().reset_index()followup_business_dates(fu)# {pid: {last_business_date, last_run_at, last_failure_at, ...}}followup_business_dates(fu,as_of='2026-10-01')# only the runs started by then
Currently executing — or crashed before writing FAILED
FAILED,…
Exception raised; the suffix carries the error message
A stale RUNNING row means the process was interrupted before its status
could be written (a killed kernel, a lost session). The feature data for that
execution is incomplete; re-run the process before relying on those values.
A failure that Python sees is recorded as FAILED,… (since 0.3.1.210).
These pairs are added to whatever band your session already has, and that
band is restored after the execution. In DBQL (DBC.QryLogV.QueryBand) they
link each query to its follow-up row:
usage=tdfs4ds.followup_resource_usage(process_id=None,run_id=None,days=30)# one row per execution: every follow-up column, then the resource usage of its queriesprint(tdfs4ds.followup_resource_usage_sql())# the SQL, to reuse in your own toolstdfs4ds.create_followup_resource_view()# FS_V_FOLLOW_UP_RESOURCES, if the view DB may read DBQL
Column
Meaning (summed over the execution's queries)
N_QUERIES, N_ERRORS
queries run, and how many failed
AMP_CPU_SEC
total AMP CPU
MAX_AMP_CPU_SEC
CPU of the busiest AMP, heaviest query
IMPACT_CPU_SEC
busiest AMP × active AMPs: what the system effectively spent
CPU_SKEW_PCT
1 − AMP_CPU / IMPACT_CPU; 0 = evenly spread, high = one AMP did the work
PARSER_CPU_SEC
parse and optimise CPU
TOTAL_IO, PHYSICAL_IO
logical and physical I/O
IO_SKEW_PCT
the same skew rule on I/O (MaxAmpIO)
MAX_SPOOL_BYTES
peak spool of any one query
FIRST_QUERY_START, LAST_RESPONSE
DBQL times (system clock)
DBQL_STATUS
LOGGED, PARTIAL, PENDING or NOT LOGGED — see below
DBQL is not immediate. Logged queries sit in a cache and reach the DBQL
tables at the flush interval (DBQLFlushRate, 10 minutes by default). So every
follow-up row is returned, with DBQL_STATUS saying whether its measures can
be trusted yet:
LOGGED: complete.
PARTIAL: some rows found, but the execution is still running or ended less
than flush_minutes ago, so the totals may still grow.
PENDING: nothing written yet. Ask again after the flush; a DBA can force it
with FLUSH QUERY LOGGING WITH ALL;.
NOT LOGGED: old enough and still absent. Logging was off for the user, or
the logging rule excluded these queries.
Pass flush_minutes= to match your system's flush rate, and only_logged=True
to keep only the executions that have measures.
Three things are needed:
- SELECT on DBC.QryLogV;
- DBQL logging for the user that runs the processes, e.g.
BEGIN QUERY LOGGING WITH SQL ON <user>;;
- some patience: DBQL writes from a cache, so the latest executions appear
after the flush interval (10 minutes by default).
check_configuration() checks all three. The view additionally needs its
database to hold SELECT … WITH GRANT OPTION on DBC.QryLogV.
How rows are paired. A follow-up row has no execution id, and one cannot
be added: older clients sharing the store would then fail to insert follow-up
rows (Error 3812). A row is therefore identified by its RUN_ID,
PROCESS_ID and its rank by START_DATETIME, and TDFS4DS_EXEC_SEQ carries
that rank.
Two edge cases can pair a row with the wrong execution, both within the same
run:
- an execution whose follow-up insert failed shifts the later ones;
- two executions of the same process starting in the same second may swap.
The statistics refresh after the last filter of a run is not tagged.
Set tdfs4ds.QUERY_BAND = False (or TDFS4DS_QUERY_BAND=false) to turn the
tagging off.
Keeping the figures beyond DBQL's retention (since 0.3.3.19)¶
DBQL keeps a few days at most on most systems. The figures of each execution are
copied into two tables of the feature store, created by setup() / connect()
when missing (optional: without them only what DBQL still holds is shown):
FS_EXECUTION_RESOURCES (view FS_V_EXECUTION_RESOURCES): one row per
execution, the columns above plus PROCESS_VERSION, the process version in
force at the execution's valid time (its business date, else its start);
FS_EXECUTION_STATEMENTS (view FS_V_EXECUTION_STATEMENTS): one row per
statement of an execution (type, times, CPU, I/O, spool, error). The SQL text
is not copied: it is read from DBC.QryLogSQLV while DBQL holds it.
tdfs4ds.harvest_execution_resources()# copy what DBQL has finished writingframe,info=tdfs4ds.read_execution_resources(process_ids=None,days=30,as_of=None)stats=tdfs4ds.process_run_statistics(frame,by_version=True)statements,info=tdfs4ds.execution_statements(execution_id)
When the copy runs: at the start of each run() (at most every 10 minutes
in a Python process, never fatal), when the app's Execution log opens, and from
Maintenance → Execution history. It is one INSERT … SELECT per table, run in
the database, and it copies only the executions DBQL has finished writing
(LOGGED) and not copied yet.
Reading:read_execution_resources returns the stored executions plus the
live DBQL join for the ones not stored yet, so a run that just ended is listed
(PENDING). Its info says when DBQL cannot be read (the follow-up rows are
then listed without figures) or logs nothing.
Runs, not rows: a FilterManager run writes one follow-up row per segment and
roll_out one per business date. process_run_statistics groups them into runs
(same RUN_ID, each row starting within 10 minutes of the previous one's end)
before computing how often a process runs and what a run costs.
TDFS4DS_EXECUTION_RESOURCES=false turns the history off. Primary store only.
In the app (since 0.3.3.20), Parameters → Execution log: query band and history switches both on or off for the app
process, creates the history tables when needed and says whether this user's runs can be measured (query
band settable, DBQL readable, queries logged). Put the variables in the app's .env to keep the setting.
Operations → Execution log: each execution with its process version, segment
or valid time, status (the whole error for a failure), duration, CPU, skew, I/O
and peak spool. Selecting one lists its statements. The second tab gives each
process's runs, frequency and consumption per process version, and a chart of
each execution over time with the version changes marked. Export to XLSX.
Feature catalog: the card of a feature's process gains Runs and
consumption for the 30 days up to the catalog time viewed. Its versions table
marks the version in force at that time and gives each version's median run and
CPU, so the effect of a SQL change is visible. The export carries
PROCESS_VERSION_AT, RUNS_30D, RUN_FREQUENCY, MEDIAN_RUN_SEC and
MEDIAN_RUN_CPU_SEC.
# Re-run at the current FEATURE_STORE_TIMEtdfs4ds.run(process_id=process_id)# Remove a process from the catalog. Feature DATA is preserved.tdfs4ds.process_store.process_query_administration.remove_process(process_id)
remove_process is metadata-only: the underlying temporal feature tables
keep every row they ever had.
What depends on it: removals are refused, then planned (since 0.3.3.27)¶
A dataset reads the values of one process (FEATURE_VERSION = <PROCESS_ID>
in its view), and a process view can read a dataset or another process's view.
Removing a process those objects read used to succeed silently. The dataset kept
returning the values the removed process last wrote and was never refreshed,
and the next lineage rebuild dropped the link.
So remove_process, remove_feature and remove_dataset(..., dry_run=False)
first look at what depends on the object. When something does, they refuse
and raise tdfs4ds.DependentObjectsError. Its message lists the dependents in
the order they must be removed: the objects that read others come first.
Cannot remove process 3b3e...: 2 object(s) depend on it. In the order they must be removed:
1. process DB_LAB.FEAT_WA_CUST_TICKET (9f1c...) - its view reads DB_LAB.DS_WA_CUST_MONTHLY
2. dataset DB_LAB.DS_WA_CUST_MONTHLY - reads sum_amount, count_amount computed by DB_LAB.FEAT_WA_CUST_MONTHLY (3b3e...)
The removal is refused too when what depends on the object cannot be read: a
dataset view that cannot be read, or an empty lineage graph. force=True removes
the object alone, as before 0.3.3.27, and leaves the dependents broken.
How dependents are found:
Removed object
What breaks
Read from
process
datasets reading its values (not another process's version of the same feature), process views reading its view, and everything downstream of those
dataset view DDL, persisted lineage
feature
datasets reading it, and downstream; nothing when process_id= scopes the removal and another process still computes it
dataset feature list, lineage
dataset
process views reading it, and downstream
persisted lineage
Removing an object with its dependents.plan_removal lists the steps in
order. write_removal_script turns them into a Python script that only removes
when run with --apply:
plan=tdfs4ds.plan_removal(['<process id>'],# also view names, datasets,purge_values=False,# or features as (name, [entity], domain[, process_id])drop_views=False)forsinplan['steps']:print(s['action'],s['object'],'<-',s['reason'])path=tdfs4ds.write_removal_script(plan)# under tdfs4ds.SCHEDULING_SCRIPT_DIR
pythontdfs4ds_removal_20261022_153000.py# dry run: what each step would removepythontdfs4ds_removal_20261022_153000.py--apply# removes, in order; stops at the first failure
Checked afterwards. When every step has run, the script checks that nothing
is left behind and exits with code 2 if something is. It looks for:
- catalog rows;
- business dictionary rows (description, sections, columns), process
documentation and provenance;
- stored SQL versions;
- vector-index entries. With Chroma they are counted by id. With the Teradata
Vector Store, entries cannot be counted (embeddings are keyed by row position),
so the check is that every collection is ready after its refresh.
The dry run shows the same counts before anything is removed.
tdfs4ds.dependencies.verify_removal(plan) runs the check on its own; in the app,
use Check what is left.
The steps of the plan:
- Order: processes that read others go first, then their datasets, then the
requested object.
- Features: each removed process is followed by remove_feature(...,
process_id=pid) for its features. That call removes a feature's catalog entry
only when no remaining process computes it.
- Stored values:purge_values=True adds the deletion of the removed
versions' rows from FS_T_*. It is irreversible; by default the history stays.
The admin app has the same flow under Operations → Remove objects. You
choose processes, datasets and features; the page shows the ordered plan, runs a
dry run (counts only) and downloads the script. After you have run the script,
Check what is left counts what each step left behind. The page itself never
removes anything.
The removal log. Every removal, and every refusal, is one row of the
optional table FS_REMOVAL_LOG (view FS_V_REMOVAL_LOG, created at setup() /
connect()). A row holds:
- what was removed: the object's id and its name (a process's view, so it can
still be named once its catalog rows are gone);
- who removed it (database USER) and when;
- whether it was forced, and the dependents;
- the origin: api, script or app;
- the outcome: REFUSED, COMPLETED or FAILED,<error>.
remove_process never touches the feature catalog. To remove the features a
process computes as well, pass the process to remove_feature (0.3.3.23):
fromtdfs4ds.lineage.lineage_storeimportget_descendantsfromtdfs4ds.utils.identifiersimportsplit_qualifiedpid='<process id>'info=tdfs4ds.get_process_info(pid)# 1. Impact first: everything that reads the process view, directly or notprint(get_descendants(*split_qualified(info['VIEW_NAME'])))# datasets, processes, views# 2. Its features: remove_feature decides, per feature, whether the catalog entry goesentity=[e.strip().split(' ')[0]foreininfo['ENTITY_COLUMNS']]fornameininfo['FEATURE_COLUMNS']:tdfs4ds.remove_feature(name.strip(),entity_id=entity,data_domain=info['DATA_DOMAIN'],process_id=pid)# 3. The process itself (drop_view=True only when nothing reads the view)tdfs4ds.process_store.process_query_administration.remove_process(pid)
When datasets or other processes read them, these calls are refused (see
above). tdfs4ds.plan_removal([pid]) writes the whole sequence, dependents
included, in order.
A feature is the triplet (data domain, entity, name) and has one catalog
entry for all the processes computing it. With process_id=:
The feature is computed by
remove_feature(..., process_id=pid)
pid only
removes the feature entirely (catalog entry, documentation, index entries)
pid and other processes
keeps the catalog entry; removes only pid's documentation of the feature (on pid's view)
The call returns {"mode": "full" | "process", "others": [...], "listed": bool}.
If the producing processes cannot be read, nothing is removed and None is
returned. Without process_id, remove_feature removes the feature whatever
process computes it, as before. Stored values are never deleted: they stay in
their FS_T_* table, and the consistency report's storage scan lists the
values of a removed feature as orphans.
fromtdfs4ds.feature_store.feature_query_retrievalimport(get_list_entity,get_list_features,get_available_features,get_feature_versions,)get_list_entity()# entities in the active DATA_DOMAINget_list_features(entity_name='CUSTOMER_ID')get_available_features(entity_id=['CUSTOMER_ID'])
Entity refresh: when was each entity last confirmed?¶
The ingestion MERGE writes nothing when a value is unchanged, so the ValidStart of a feature
row is the time of the last change, not of the last confirmation; and FS_FOLLOW_UP is per run
and segment, so an entity the source stopped returning looks as fresh as one the run recomputed.
Register the process with track_entity_refresh=True (or set_process_options(pid,
track_entity_refresh=True)) and each batch also records, in an admin table, that it confirmed each
of its entities:
tdfs4ds.entity_refresh('<PROCESS_ID>')# last confirmation of each entitytdfs4ds.entity_refresh('<PROCESS_ID>',entities=[1,2],history=True)# every period of some entitiestdfs4ds.entity_refresh('<PROCESS_ID>',as_of='2024-01-15')# the table as of a valid time (UTC)s=tdfs4ds.entity_refresh_summary('<PROCESS_ID>')s['by_refresh_time']# entities per last refresh times['not_confirmed']# a sample of the entities the latest run did not confirm (s['n_not_confirmed'] = how many)s['state'],s['message']# 'tracked' / 'not_tracked' / 'never_run' / 'no_table'
Where: one table FS_R_* per (data domain, entity key, primary index, placement tag), named by the
rule of the FS_T_* tables plus the placement tag of the process (see
Relocating features): processes in tables of their own never share a refresh
table. Columns: the entity, PROCESS_ID, REFRESH_TIME (the run's valid time), LOAD_TIME (database
clock), RUN_ID, and a valid-time period: each run opens a new period per entity it produced. It is not
a feature: no catalog or FS_V_* view exposes it, no dataset reads it. The EFS map lists it as Entity
refresh (admin). Created by the first tracked write.
Valid time: as the values: VALIDTIME PERIOD '(t, UNTIL_CHANGED)' with the run's valid time,
CURRENT VALIDTIME without one. A re-run at an earlier date rewrites the later periods, like the
values MERGE: replay a history from its first date.
Fail-open: a failed write is a warning with the full database message; the run stays COMPLETED
and the history has a gap. A batch that staged 0 rows records nothing. An entity ended by
close_absent_entities keeps its last refresh row. With a FilterManager, a run confirms only its
segment: the other segments show as not confirmed until their turn.
Older clients record nothing. The summary compares with FS_FOLLOW_UP: COMPLETED runs without a
refresh row read not tracked (older client or failed write), never "every entity was dropped".
Removal and migration: remove_process deletes the refresh rows of the process (and its
FS_PROCESS_SETTINGS row); dependencies.footprint counts them. Migration copies the setting, not the
FS_R_* tables (a recompute regenerates them). The parallel planner serialises two tracked processes
that share a refresh table.
Each feature storage table (FS_T_*) needs optimizer statistics. Without them
Teradata can misestimate dataset queries by orders of magnitude. Ingestion
defines them the first time it writes to a table. After each merge it only
refreshes the statistics the table already has (`COLLECT STATS ON
`). So a statistic added to the expected set later, like
`(entity columns, FEATURE_ID)` in 0.3.1.216, does not reach older tables on its
own.
report=tdfs4ds.check_storage_statistics()# reads DBC onlyreport[['TABLE_NAME','STATUS','MISSING','OLDEST_COLLECTION']]tdfs4ds.check_storage_statistics(add_missing=True)# define what is missingtdfs4ds.check_storage_statistics(refresh=True)# re-collect nowtdfs4ds.check_storage_statistics(domains=['MY_DOMAIN'],add_missing=True,refresh=True)
| `STATUS` | Meaning |
|----------|---------|
| `OK` | every expected statistic is defined |
| `MISSING` | some are not; `MISSING` lists them |
| `NO_STATS` | none defined (an empty or never-ingested table) |
| `TABLE_MISSING` | the feature catalog lists the table but it no longer exists. Nothing is done to it |
- **Missing statistics are added with ingestion's own options**, so later
merges keep them current: single-column ones plainly, multi-column ones with
`USING SAMPLE 25 PERCENT AND THRESHOLD 15 PERCENT`.
- **The threshold is why some dates look old.** A multi-column statistic is
refreshed only once 15% of the table has changed, so its
`OLDEST_COLLECTION` can lag the others.
- **Your own statistics are safe.** One defined outside tdfs4ds appears under
`EXTRA` and is never dropped.
- **Scope:** `domains=`, `processes=` (process ids) or `tables=`
(`'db.FS_T_...'`).
In the admin app, the same check runs from **Maintenance → Storage
statistics**, with *Check*, *Add missing*, *Refresh* and *Add missing +
refresh* buttons scoped by data domain and process. The **Report** shows a
read-only *Storage statistics* section, which is included in the exported HTML.
## Workflow assessment (since 0.3.3.25)
The consistency checks tell whether the catalogs and the storage line up. The
workflow assessment tells whether the pipeline is **well built**: are the entity
keys the right ones, did the processes run in the order their dependencies and
business dates require, and do they follow the same clock? Every check is
deterministic (counts, the data dictionary, the lineage graph, the follow-up,
stored SQL hashes); no language model is used, and nothing is changed.
report=tdfs4ds.assess_workflow(data_domains=None,# or ['MY_DOMAIN']processes=None,# or process ids / view namesentity_scan=True,# False skips the only read of the storage tablesdays=90,# follow-up window of the run checks)report['counts']# {'error': n, 'warning': n, 'info': n}report['findings']# area, kind, object, detail, severity, fixreport['unverifiable']# checks that could not run, with the reason
**Lineage integrity (since 0.3.3.27).** Gaps left by a removal or a dropped
object:
| Kind | Meaning |
|---|---|
| `dataset_reads_removed_process` | A dataset still reads the values of a process that is no longer registered: it returns what that process last wrote and is never refreshed. With the removal log, the finding names the process's view, when it was removed and by whom. |
| `dataset_reads_removed_feature` | A dataset reads a feature that left the catalog. |
| `feature_without_process` | A registered feature no process computes any more. |
| `process_view_missing`, `dataset_view_missing` | A catalog row whose view is gone. |
| `process_reads_missing_object`, `process_view_broken` | A process view reads an object that no longer exists, or no longer compiles (checked by EXPLAIN, only on processes reading an object absent from the data dictionary). |
| `process_reads_unregistered_dataset` | A process view reads a dataset view that is no longer in the dataset catalog. |
| `lineage_edge_to_unknown` | The lineage graph links a dataset to a view that is no registered process. |
| `dataset_reads_stale_table` | A dataset reads a feature from a storage table the catalog no longer uses for it (`tdfs4ds.rewrite_dataset_tables(dataset)` points it at the catalog's table). A type change moves a feature to another table on the next run, with no history copy, and the dataset keeps reading the old table: frozen values, then none. The finding shows `old -> new`. |
| `dataset_reads_missing_table` | A dataset reads a storage table that no longer exists (querying it fails). |
| `dataset_feature_no_rows` | Storage level, on demand (check `dataset_rows`): the table holds no row for the feature's `(FEATURE_ID, FEATURE_VERSION)` at the valid time the dataset reads. An INNER dataset then returns no row at all (error); a FULL OUTER one reads NULL for the column (warning). The follow-up tells a process that **never completed** from rows that **are gone**. |
| `relocation_reverted` | Storage level (check `dataset_rows`): a feature was relocated to its own storage table (see [Relocating features](relocating-features.md)) but a client older than 0.3.3.30 ran its process and wrote to the old table, or registered it there again. Its history is split over two tables and the datasets read either. Fix: upgrade the clients, then run `relocate_features` on the same plan again. |
The view compiles in all three cases: an `EXPLAIN` passes. The first two are
read from the data dictionary and run with the cheap checks; the third counts
the rows (one grouped pass per storage table the datasets read) with the entity
scan. A feature missing from the catalog altogether is
`dataset_reads_removed_feature`. Rewriting only the table references of a
dataset is `tdfs4ds.rewrite_dataset_tables(dataset)` (dry run by default; see
[Relocating features](relocating-features.md)); rebuilding it with `build_dataset`
also works.
Each error names the **dead branch**: the datasets and processes that depend on
the gap. Its fix is the call that removes that branch in order,
`tdfs4ds.write_removal_script(tdfs4ds.plan_removal([...]))`. In the admin app,
its *Remove dead branch…* link opens **Remove objects** with the branch already
chosen.
**Lineage network (since 0.3.4.0, check `network`).** The lineage seen as a
whole, from the persisted lineage graph and the data dictionary:
- **Every view is compiled.** `EXPLAIN SELECT * FROM ` runs on every
process view, dataset view and view the lineage graph holds (not only on the
processes reading a missing object). Only the views whose own inputs compile
are reported: when a view fails, every view reading it fails too, and they are
listed under it with the processes and datasets it stops. An error the
database did not give (a lost session) is listed under *Not verified*, never
as a broken view.
- **Is the network in one piece?** The connected networks of the lineage, with
their processes, datasets, views and tables. Independent pipelines are
expected to be separate; a process meant to read another one should be in its
network. A process or dataset with no lineage link at all is a warning
(`rebuild_lineage_graph(objects=[...], check=True)` records it). Names of the
graph that are not database objects (SQL tokens) are left out, so they cannot
join two networks.
- **Dead branches.** A process nobody reads (no dataset, no other process), the
views and tables of the lineage that no registered process or dataset reaches
(usually left by a removed process), and the branches the gaps above leave.
| Kind | Meaning |
|---|---|
| `view_does_not_compile` | EXPLAIN fails on the view; the other views, processes and datasets it stops are named. |
| `lineage_split` | The lineage is several separate networks (info). |
| `lineage_isolated` | A process or dataset with no lineage link. |
| `process_not_read` | A process whose view no dataset or process reads: a dead end. |
| `detached_branch` | Views and tables of the lineage that no registered process or dataset reads. |
The Lineage section of the report always shows this block, also when nothing is
found, so a clean result says what it covered.
**Entity keys.** People often put every id column they know into `entity_id`.
For each entity of two columns or more, the scan counts the distinct entity
tuples of its largest storage table with each key column left out, over the
whole history and now:
| Finding | Meaning |
|---------|---------|
| `redundant_key_column` | The other key columns are unique without this one over the whole history: it adds nothing to uniqueness, and since it is in the primary index a lookup by the real key cannot use the index. |
| `key_column_changes_over_time` | Determined by the other columns now, but it changed: each change started a new entity, which split that entity's history. Store it as a feature instead. |
| `entity_key_superset` | Two entity keys of one domain, one containing the other: features of one entity may be split across two sets of tables. |
| `entity_column_type_mismatch` | An entity column typed in different families across tables (integer vs character, LATIN vs UNICODE): joins convert one side. |
| `storage_skew` | A storage table concentrated on a few AMPs. |
Both entity fixes name the same key (`entity_id=[...]`), checked unique now. A
dependency observed in the data is **not a rule**: the finding states how many
entity tuples it covers; confirm with the business before changing an entity.
**Order and synchronisation**, for each direct dependency of the process graph
(read from the persisted lineage), business date by business date:
| Finding | Meaning | Fix |
|---------|---------|-----|
| `ran_before_upstream` | The process started before its upstream had finished. | Run in plan order (`analyze_parallel_plan`, `generate_execution_script`), re-run with `force_compute=True`. |
| `upstream_recomputed_after` | The upstream was recomputed after the process ran: the process holds values from the older upstream. | `roll_out([pid], tm, force_compute=True)` over those dates. |
| `completed_on_failed_upstream` | The upstream had failed or written no rows, and the process still completed. | Re-run the upstream, then the process with `force_compute=True`; use `check_upstream=True`. |
| `upstream_missing_at_date`, `ahead_of_upstream` | On one timeline, the process computed dates its upstream has not. | Roll the upstream out first. |
| `mixed_time_modes`, `different_time_drivers` | The two ends follow different clocks. | One TimeManager, or `unify_time_managers`. |
| `point_in_time_leak` | A TimeManager process reads a dataset built at the current valid time: at a past date it computes from today's values. | `build_dataset(..., time_manager=tm)`. |
| `fixed_as_of_input`, `input_other_time_manager` | The dataset it reads is frozen, or follows another manager. | Rebuild it on the process's TimeManager. |
Real-time pairs are only checked for overlapping runs: two real-time processes
on different cadences are normal.
**Primary index and partitioning against the sources.** Ingestion stages the
view's result in a volatile table hashed on the storage table's primary index,
then merges it. When the largest source table is hashed on the same columns, the
rows never leave their AMP. The assessment follows each entity column through the
view SQL to the source column it comes from (unchanged, no `CAST`), reads the
primary indexes and partitioning from the data dictionary, and EXPLAINs the
staging step with the current index and with the aligned one (EXPLAIN only,
nothing is created):
| Finding | Meaning |
|---------|---------|
| `pi_not_aligned_with_source` | The largest source is hashed on columns the view passes as entity columns, the storage tables otherwise. A warning when EXPLAIN shows rows redistributed now and local with the aligned index; nothing when EXPLAIN shows no difference. |
| `entity_pi_inconsistent` | One entity's storage tables use different primary indexes. |
| `partition_not_aligned_with_source` | The largest source is partitioned on a column passed as an entity column; the storage tables are not. |
| `segments_rescan_source` | A FilterManager segments on columns that do not follow the largest source's partitioning, so each segment reads all its partitions. |
| `segment_filter_defeats_pruning` | The segment filter names a partitioning column, yet with the segment value known the plan still reads every row: the view SQL prevents pruning. |
| `time_filter_by_join`, `time_filter_not_on_partitioning`, `time_filter_defeats_pruning` | The same questions for the TimeManager's business date. |
| `fs_partition_not_on_filter` | Segments on entity columns that do not partition the storage tables. |
**How pruning is judged.** First the data dictionary: does the FilterManager or
TimeManager filter name a partitioning column of the source? A filter on **one**
partitioning level is enough. Measured on a source partitioned by day and by
category, a category filter alone read 365 of 2,555 partitions, a date alone 7,
and both a single one. Then the plan, with the manager's value known:
`DYNAMIC EXPLAIN` with incremental planning forced (`DynamicPlan=SYSTEMX`, the
session's query band is restored afterwards). A static EXPLAIN leaves the
subquery's value as a parameter and shows an all-rows scan even for well-written
SQL. When the filter matches the partitioning but this plan still reads every
row, the process SQL is what prevents pruning: a function or `CAST` on the
column, an `OR` with another column, a type mismatch. On a PTI table the pruned
read is worded "from a time range of".
**Favour scalar filtering** for both managers, with the column left bare and any
arithmetic on the manager's side:
The one exception is a view scored on the Analytical Compute Cluster (tdstone2
with Apply), which needs a JOIN to the manager: a scalar subquery there is
Error 6881.
Two rules shape these recommendations:
- **Per entity, never per process.** The primary index is part of the storage
table name, and a dataset joins the tables of an entity on the AMP only when
they share one index. Changing it means removing the process and its features
and registering again with `primary_index=` (registering the same view again
keeps the recorded index), then replaying the history.
- **Segmenting on the source's partitioning is the most efficient segmentation,
when the business logic allows it**: a segment must hold every row its
computation needs (no aggregation or window across the partitioning column).
**Runs and process SQL:** FilterManager segments missing or failed at the
latest date, runs that wrote no rows, rows stuck at RUNNING, live SQL different
from the stored version (`sql_drift`) and histories computed under several SQL
versions.
A check that cannot run (lineage graph empty, data dictionary unreadable, SQL
versioning off) is listed under `unverifiable`, never reported as passing.
**Executive summary.** The report opens with the verdict (how many processes
have an error, a warning, information only, nothing to report), the most
frequent issues, and a health grid. The grid has one row per process, grouped by
execution wave (the order the execution plan runs them in), and one column per
dimension: lineage, entity keys, order and time, inputs in time, runs, process
SQL, storage and pruning. Each cell shows the worst severity found. A cell reads *not
checked* when its check did not run or could not run for that process, for
example the entity scan, or a process absent from the lineage graph.
tdfs4ds.write_workflow_report(report,fmt='html')# or 'md', 'csv'; returns the path
In the admin app: **Operations → Workflow assessment**, with a data-domain list
(↻ re-reads it). The entity scan and the dataset-feature row count run on demand with *Scan entity keys*. The
report exports as a standalone HTML page, Markdown or CSV. The **Report** has a
*Workflow assessment* section. Notebook `A3 - Assessing the workflow` walks
through the main findings.
## EFS map (admin app)
The **EFS map** panel shows every object of the feature store, for architects
and DBAs, in two planes.
- **Metadata plane**: the catalogs that describe, govern and drive the data.
They are grouped as: feature catalog; process catalog; run history and run
drivers (the FilterManager / TimeManager objects the processes use); dataset
catalog; business dictionary and documentation; lineage graph. Each hidden
table (T) is shown with the public view (V) it is read through, with its row
count and perm size.
- **Data plane**: the storage tables `FS_T_*`, grouped by **entity** (the key
column set). There is one table per data domain, entity and value type. Each
entity group shows three lanes:
- the process views that write its tables;
- the tables themselves;
- the datasets that read them.
Each table card shows:
- the number of features and the number of rows (with a bar relative to the largest table);
- the value type, perm size and AMP skew (`1 − avg/max` of per-AMP perm);
- the primary index and the partitioning expression;
- its companion `FS_V_*` view;
- which processes write it and which datasets read it.
A table listed in the feature catalog but absent from the database is flagged
**TABLE MISSING**.
Row counts come from the table-level statistics by default. This is cheap, but
the count is as of the last collection and current and history rows are
counted together. **Count rows exactly** runs `COUNT(*)` in `CURRENT` and
`NONSEQUENCED VALIDTIME`, one query per 25 tables, and shows current and
history separately.
**Rows per feature** on a table card plots its rows per feature (current and
history, one bar per feature and process version). A `FEATURE_ID` that is not
in the catalog for that table is shown in red. **Export HTML** downloads the
map as one standalone file.
## Feature counts, valid time and orphans (admin app)
**What is counted as a feature.** A feature is the triplet (data domain,
entity, feature name). The same name registered under two entities, or in two
data domains, is two features. The Report, the EFS map and the consumer agent
all count features this way.
The *Documentation coverage* card is the exception: it is labelled
**Features (process outputs)**. It counts one row per name a current process
lists, because documentation is stored on the process view's columns. A line
under that card reconciles it with the catalog, in the form ` in the
catalog · without a current producer · listed by several processes`.
**Catalog valid time.** The catalogs are valid-time tables, so a count is only
meaningful at a given instant.
- The **Report** and the **EFS map** each have a *Catalog valid time (UTC)*
input. Blank means the current valid time. A date-time reads the catalogs as
they were at that instant (primary store only).
- The time used is printed next to the counts.
- Whether a table or view exists, and the physical row counts, are always read
from the database as it is now.
**Entity keys.** In the Report's inventory tree, each entity column is tagged
**PI** when it is in the primary index of the entity's storage tables and **P**
when the tables are partitioned on it; an unmarked column is neither. `PI*` /
`P*` means only some of the entity's tables agree (the tooltip says how many).
**Consistency & orphans.** This Report section lists, with the command that
fixes each finding (nothing on the page writes):
- **Features**
- registered with no current producer;
- listed by several processes;
- retired from the catalog;
- whose storage table is missing.
- **Processes**
- whose view is gone;
- whose features are not registered (declared, never run);
- that never ran;
- whose features no dataset reads.
- **Storage tables**: `FS_T_*` tables that no catalog row points to, or that
only retired features point to.
- **Datasets**: those that read features the catalog no longer has, or whose
view is gone.
A line at the top reconciles every feature number shown by the panels.
**Scan storage for orphans.** This button reads every storage table once
(`FEATURE_ID, COUNT(*)`, all versions and at the valid time). It adds two kinds
of finding:
- `FEATURE_ID`s stored without a catalog row, either known in the catalog
history or never registered;
- registered features that have no value at that valid time.
Run it when convenient: on a large store it reads every storage table. Its
result also adds orphan badges to the EFS map's table cards. The EFS map lists
**Unreferenced storage tables** in its data plane.
**Process counts.** A process is a distinct `PROCESS_ID`; every panel and the
consumer agent count them that way (`store_consistency.catalog_counts`), with
the same notes (never run, not registered).
**Time drivers.** This Report section lists, per process, the TimeManager it
declares and the one its view reads (from the lineage graph), with a status
(`OK`, `UNDECLARED`, `MISMATCH`, `DECLARED_UNUSED`, `NONE`, `UNKNOWN`), the
manager's current date and the last business date the process computed, and
the synchronisation groups: one manager view, soft clones, hard clones,
independent. See [TimeManager](time-manager.md#synchronisation-in-the-report).
The **EFS map** shows the same section, between its metadata and data planes; at
a past catalog date it reads the lineage at that date and the managers' current
cursors, and says so.
**Datasets missing from the lineage graph.** A registered dataset whose view is
not in `FS_LINEAGE_GRAPH` is reported (`dataset_not_in_lineage`) with the
`rebuild_lineage_graph(objects=[...])` call that adds it. Every path that builds
a dataset records its lineage, `restore_dataset_catalog` records it for the
datasets it registers or repairs, and `drop_dataset` closes the dropped
dataset's edges (their history stays, for the lineage at a past date).
**Defective datasets (since 0.3.3.19).** In the Dataset builder's *Existing
datasets* list, a dataset whose sample fails is marked **Defective**, with a short
cause and the database's whole message:
- *the dataset view no longer exists (only its catalog entry remains)*: the view
was dropped and the catalog still lists it;
- *an object it reads no longer exists*;
- *access refused on an object it reads*;
- *the view no longer compiles*;
- *the query runs out of spool*.
**Check all** marks them for the whole list without reading data: one
`DBC.TablesV` read finds the dropped views, then `EXPLAIN SELECT * FROM` each
remaining dataset.
**Remove from catalog** (admin view, primary store only) first shows what goes:
the catalog, entity, feature, build and window rows (closed in valid time,
history kept), the lineage edges, the business documentation rows with their
provenance, and the vector-index entries. The view itself is kept unless *Also
drop the view* is ticked; when the view is already gone, only the catalog entry
is removed. The same in Python:
tdfs4ds.remove_dataset("DB_LAB.DS_CUSTOMER")# dry run: what would gotdfs4ds.remove_dataset("DB_LAB.DS_CUSTOMER",dry_run=False)# view kepttdfs4ds.remove_dataset("DS_CUSTOMER",dry_run=False,drop_view=True)
`drop_dataset(dataset_id)` still removes the catalog rows only.
**Lineage panel.** The default **Flow** layout reads left to right (sources,
intermediate views, processes, datasets), zooms and pans, highlights an object's
whole upstream and downstream path on click and shows its documentation and DDL
below, searches objects by name, and can hide source tables and managers or show
processes and datasets only. TimeManager and FilterManager objects are drawn as
drivers (dashed links). The Sankey diagram, unreadable on large workflows, is no
longer offered there (`plot_lineage_sankey` stays available in Python).
## Documentation coverage (admin app)
The **Documentation** section of the rail has two panels: what is documented
and indexed, and what is missing.
**Objects counted:**
| Kind | Complete when |
|---|---|
| Process | its view has a description and every feature column has one |
| Feature | its column of the process view has a description |
| Dataset | it has a description and the five sections (`OVERVIEW`, `ENTITY`, `FEATURE_THEMES`, `BUSINESS_QUESTIONS`, `INTENDED_AUDIENCE`) |
| Intermediate view | it has a description (views between sources and process views, from the lineage graph) |
| Source table | it has a description. tdfs4ds never generates these: write them in **Objects › Edit documentation**, with `tdfs4ds.set_object_documentation`, or load a data dictionary with `upload_business_dictionary_objects` (see [Writing documentation by hand](../ai/manual-documentation.md)) |
An object with some documentation but not all of it is *partial*; a blank
description counts as missing. The feature store's own tables (`FS_T_*`,
`*_HIDDEN`) are not counted. Lineage nodes that are not objects of the database
(tokens the lineage builder stored as names, such as `APPLY`) are left out and
listed. Documentation of objects the store no longer has is listed as well.
**Vector index status**, per documented row:
- **Chroma: exact.** The expected documents are built as `build_vector_index`
builds them, then compared with what the collection stores: *indexed*,
*stale* (text, metadata or embedding model changed) or *not indexed*. Ids
stored but no longer documented are counted as orphans.
- **Teradata Vector Store: inferred.** The service exposes no reliable per-row
membership (its embeddings are keyed by a row position recomputed on
refresh). A row is *indexed (inferred)* when its collection is `ready` and the
row was documented before the collection's last update, and *pending* when
it was documented after. A missing, failed or building collection is shown
as such.
An object takes the worst status of its rows.
**Coverage** shows, per kind, documented / partial / missing with the index
issues, one line per collection (status, last update), and the lists of gaps.
A gap opens the object in **Objects**. *Export gaps* downloads them as CSV.
**Objects** lists every object with filters (kind, documentation status,
index status, data domain, text). Clicking a row shows its documentation as in
the Lineage panel, with its status.
**Actions**, streamed with a live log:
- *Document missing* (Coverage) or *Document selected* (Objects): documents
each object (a process with `document_process_incremental`, a dataset with
`document_dataset_incremental`, an intermediate view as
`rebuild_documentation` does; source tables are skipped), then updates the
vector index. A process is documented once even when several of its
features are selected.
- *Update vector index*: `build_vector_index(force_rebuild=False)`. On Chroma
it embeds only missing or changed rows.
The rail is grouped by theme: Assistant, Explore, Documentation, Operations,
Performance, Settings. Hovering an entry shows what it does. A section
collapses by clicking its heading. The browser remembers it, and the section
of the panel in the URL always opens.
## Version + diagnostics
tdfs4ds.__version__# package versiontdfs4ds.list_skills()# bundled agent skillstdfs4ds.skill_catalog()# full skill metadata