tdfs4ds.feature_store.entity_refresh¶
When did a process last confirm each entity? track_entity_refresh (0.3.3.30).
The ingestion MERGE writes nothing when a value is unchanged (verified live), so the
ValidStart of a feature row is the time of the last change, not of the last
confirmation; and FS_FOLLOW_UP is per run and segment, so an entity the source stopped
returning looks exactly as fresh as one the run recomputed. A process registered with
upload_features(..., track_entity_refresh=True) (or set_process_options) therefore also
records, at each run, that it produced each entity of the batch.
What is written¶
One admin table per (data domain, entity key, primary index, placement tag), FS_R_*,
named by :func:refresh_table_name - the one name authority for the writer and the readers.
It is built with the naming rule of the FS_T_* tables (get_feature_store_table_name)
plus the placement tag of the process's features, so processes relocated to a tag of their
own never share (and never lock) a refresh table. Columns: the entity columns,
PROCESS_ID, REFRESH_TIME (the run's valid time), LOAD_TIME (database clock),
RUN_ID and the valid-time period; primary index = the process's. A run writes one new
period per entity it produced (REFRESH_TIME differs each run, so the MERGE opens a period).
It is not a feature: nothing in the feature catalog, the process catalog, the dataset
catalog or any FS_V_* view exposes it, and no dataset can read it. The readers are
:func:entity_refresh and :func:entity_refresh_summary. The FS_R_ prefix keeps it out of
every FS_T_ detector.
Rules¶
- Fail-open. A failed write is a WARNING with the full database message; the ingestion
stays
COMPLETED. It runs only when the batch staged rows (a run that staged 0 or an unknown number of rows confirms nothing). - Lazy. The table is created by the first tracked write (
create_object_if_missing). - Valid time. The write follows the values' MERGE:
VALIDTIME PERIOD '(t, END_PERIOD)'with the run's valid timet(_ingestion_time(valid_start)),CURRENT VALIDTIMEwithout one. A re-run at an earlier date rewrites the later refresh periods, exactly as the values MERGE does (replay a history from its first date). - close_absent_entities keeps nothing here: an entity it closed keeps its last refresh row.
- Old clients. A run by a client without this feature writes no refresh row; the summary
compares with
FS_FOLLOW_UPand says "not tracked (older client or failed write)" instead of reading the silence as a dropped entity.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
The FS_R_* table of an (entity, data domain, primary index, placement tag).
The rule of the FS_T_ names (get_feature_store_table_name, which already encodes the
domain, the entity key and the primary index; partitioning is not part of it: the refresh
table is not partitioned) with the process's placement tag, under a fixed pseudo-type so
that it can never collide with a storage table. table_tag=None is the untagged name.
The placement tag the refresh table of a process follows: the tag shared by its
features, or the first tag (alphabetical) when they differ; None without any placement.
Reads FS_FEATURE_PLACEMENT only when it exists (one cached check per store).
CREATE of an FS_R_ table: entity columns typed like the storage tables'.
refresh_merge_sql(table: str, entity_columns, volatile_ref: str, process_id: str, run_id, store_time=None) -> str
The MERGE that records the entities of the staged batch. Pure.
One row per (entity, process): matched = a new period with the run's REFRESH_TIME,
not matched = the first one. The valid time is the values MERGE's: VALIDTIME PERIOD
'(t, END_PERIOD)' with a run valid time, CURRENT VALIDTIME without. Every column of
the INSERT is named.
track_entity_refresh(entity_id, volatile_table_name, process_id, primary_index=None, feature_names=None, valid_start=None, run_id=None, executor=None, at=None) -> dict
Record that process_id produced the entities of the staged batch. Never raises.
Called by the ingestion right after the values are stored, in the same session (the
staged volatile table must still exist). Returns {"table", "valid_time"} on success
and {"error": <full database message>} after a WARNING otherwise: the ingestion that
called it stays COMPLETED. executor replaces teradataml.execute_sql (tests). at
replaces the run's valid time (a valid_time_column load: its latest valid time).
The FS_R_ table of a registered process (None when the process is unknown).
{PROCESS_ID: (SCHEMA upper, table upper)} of the processes that track their refresh
(a process with the option off is absent). For the parallel planner: the refresh MERGE
takes a table lock too. Fail-open to {}.
Every FS_R_* table of the primary store (one DBC read). Empty on failure.
Refresh rows (all periods) of a process over every FS_R_ table. None when unreadable.
Delete the refresh rows of a process from every FS_R_ table (all periods).
AND (...) for a list of entity values: scalars (one key column), tuples / lists or
dicts (one value per key column, in column order).
When process_id last confirmed each entity (admin reader of its FS_R_ table).
Columns: the entity columns, REFRESH_TIME (the valid time of the run that confirmed the
entity), LOAD_TIME (database clock when it was written), RUN_ID. By default the
period in force now; as_of reads the table at another valid time (a date without an
offset is UTC), history=True returns every period with ValidStart / ValidEnd.
entities restricts to some values: scalars for a one-column entity, tuples (in the
order of the entity columns) or dicts otherwise. An empty frame when the process does not
track its refresh (or has not run since it started to); raises ValueError for an
unknown process.
The latest plain COMPLETED execution of a process in FS_FOLLOW_UP (None when never).
COMPLETED,0 rows (an empty segment) is not one: an empty batch writes no refresh row.
The entities of process_id by last refresh time, and those its latest run did not confirm.
Returns a dict:
state:"tracked","not_tracked"(the process has COMPLETED runs but its refresh table holds no row of the latest one: an older client, a failed write, or an empty run),"never_run"or"no_table";message: the state in words ("not tracked (older client or failed write)"fornot_tracked: silence is never read as a dropped entity);table,n_entities,last_refresh,by_refresh_time(a frame:REFRESH_TIME,N_ENTITIES),n_not_confirmedandnot_confirmed(a sample frame): entities whose last confirmation is older than the latest run's. With a FilterManager the other segments are in that set until their own turn.
as_of=None reads every period (the last confirmation ever); a date reads the table as of
that valid time.
A comparable naive timestamp (the follow-up and the refresh table are in the same session zone).