tdfs4ds.execution¶
Feature ingestion and process execution for the tdfs4ds feature store.
Core execution call chain:
upload_features(df, entity_id, feature_names, ...)
→ register_process_view.__wrapped__(...)
→ execute_query(query_insert / query_insert_dist / query_insert_filtermanager)
→ [if optimize_query=True and instruct model available]
→ skill_optimize_query(process_id=process_id)
→ saves full Markdown report to tdfs4ds_optim_{timestamp}_{view}_{process_id}.md
→ if improvement found and update_view=True: REPLACE VIEW {view_name} AS {best_sql}
→ [if document=True and instruct model available]
→ document_process_incremental(process_id=process_id, display=False, upload=True)
→ run(process_id) # ingests from the (now-optimized) view; builds dataset view if dataset_view_name set
→ [if document=True and instruct model available and dataset_view_name is not None]
→ document_dataset_incremental(dataset_id) # bottom-up dataset docs after feature docs exist
→ SELECT * FROM process_catalog WHERE PROCESS_ID = ?
→ _upload_features(df, entity_id, feature_names, feature_versions=process_id, ...)
→ Gettdtypes / update_varchar_length / validate_feature_types
→ register_entity / register_features
→ _compute_features_batch(...) [once, or per-filter loop]
→ followup_open
→ prepare_feature_ingestion # unpivots df into volatile table
→ store_feature # MERGE/UPDATE_INSERT into feature tables
→ followup_close (COMPLETED)
→ apply_collect_stats(...) # called once after all batches
roll_out(process_list, time_manager, ...)
→ iterates time steps → run(process_id) per step
module-attribute
¶
ACC_6881_HINT = "Error 6881 (not eligible for the Analytical Compute Cluster): for a tdstone2-Apply view that references a TimeManager/FilterManager, TWO things are needed — (a) the manager table on OFS (storage='OFS'), so the ACC can read it, AND (b) the filter written as a JOIN to the manager view, NOT a correlated scalar subquery '(SELECT BUSINESS_DATE FROM <view>)', which the ACC rejects. OFS + JOIN scores; OFS + scalar subquery still 6881s. The filter shape is the user's responsibility. Also: run the model's score() BEFORE upload_features — roll_out/run() cannot drive an Apply model (its ingestion CTAS also hits 6881)."
module-attribute
¶
_INTO_VOLATILE_ART_RE = compile('INTO\\s+VOLATILE\\s+ART\\s*\\(\\s*([A-Za-z_\\"][\\w.\\"]*)', IGNORECASE)
module-attribute
¶
module-attribute
¶
A utility for managing dynamic, versioned filter sets as database-backed views.
The FilterManager enables lightweight scenario management by storing multiple
filter definitions in a hidden Teradata table and exposing a public view that
dynamically switches between them by filter_id. Each row in the hidden table
represents a complete filter configuration. The active configuration is
controlled by updating the view definition rather than rewriting table data.
Key Features
- Store multiple filter states (scenarios) indexed by
filter_id - Switch filter states instantly by updating a view
- Optionally include time-based slicing using a
BUSINESS_DATEcolumn - Clone filters between managers (soft or hard clone modes)
- Prune obsolete filters to control table size
- Retrieve current and historical filter definitions
Workflow Overview
- Create a
FilterManagerpointing to a target view name. - Load one or more filter definitions using
load_filter(). - Switch active filters using
update(filter_id). - Inspect the active filter via
display()or view DDL. - Optionally prune or clone filters as needed.
How It Works Internally
- A hidden table named
<view_name>_HIDDENstores filter definitions. - A Teradata view named
<view_name>exposes only the active filter row. - Each filter automatically receives a sequential
filter_id(ROW_NUMBER()ordering ensures deterministic assignment). - If time-based filtering is used via
time_column, aBUSINESS_DATEcolumn is added and projected in all operations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table_name
|
str
|
Public view name to manage or create. |
required |
schema_name
|
str
|
Teradata schema where artifacts will be created. |
required |
filter_id_name
|
str
|
Name of the filter ID column. Defaults to |
'filter_id'
|
time_column
|
str
|
Optional name of a timestamp column from input DataFrames
that maps to a |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
schema_name |
str
|
Target schema for view and hidden table. |
table_name |
str
|
Name of hidden table storing filters (auto-suffixed with |
view_name |
str
|
Name of public view pointing to current filter. |
filter_id_name |
str
|
Column containing filter ID. |
nb_filters |
int | None
|
Number of stored filters (None until initialized). |
col_names |
list[str] | None
|
Columns projected by the view (data columns only). |
time_filtering |
bool | None
|
True if time-based filtering enabled. |
Notes
- Database objects are only created when
load_filter()is first called. - Safe for iterative pipeline runs—auto-detects existing artifacts.
- Designed for large production tables and Teradata-native workflows.
instance-attribute
¶
__init__(table_name, schema_name, filter_id_name='filter_id', time_column=None, storage=None, ofs_map='TD_MAP2')
Initialize the FilterManager.
If the hidden table/view already exist, metadata (column names, maximum
filter id, and time filtering status) are detected and cached. If they do
not exist yet, attributes are initialized but no objects are created until
load_filter() is called.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage
|
Where to create the hidden backing table when this manager
(re)creates it. |
None
|
|
ofs_map
|
Map paired with |
'TD_MAP2'
|
Load a new filter set into the hidden table and (re)point the public view at filter_id=1.
Each row in df is assigned a deterministic filter_id based on ROW_NUMBER() over the
ordered set of its columns (plus BUSINESS_DATE when time filtering is enabled). If
time_column is provided, values from that column are copied into BUSINESS_DATE and the
view will include that time dimension.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Incoming filter definitions (one row per filter). |
required |
primary_index
|
list[str]
|
Primary index columns for the hidden table. Defaults to ['filter_id'] when omitted. |
None
|
time_column
|
str
|
Name of the time column in |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Repoint the public view to a different filter id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filter_id
|
int
|
Target filter id to apply. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If filter artifacts do not exist yet. |
Retrieve the current view contents as a teradataml.DataFrame.
Returns:
| Type | Description |
|---|---|
|
teradataml.DataFrame: Rows projected by the public view (current filter). |
Retrieve all filter rows from the hidden table.
Returns:
| Type | Description |
|---|---|
|
teradataml.DataFrame: Full set of stored filters. |
Return the earliest business date/time from the current view.
The method reads the first BUSINESS_DATE value from the current view
and normalizes it to a %Y-%m-%d %H:%M:%S string. Requires that time
filtering is enabled.
Returns:
| Name | Type | Description |
|---|---|---|
str |
Earliest datetime as formatted string ('YYYY-MM-DD HH:MM:SS'). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If time-based filtering is not enabled. |
Extract the currently active filter id from the view DDL.
Returns:
| Name | Type | Description |
|---|---|---|
int |
Filter id parsed from the view's definition. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the filter id cannot be parsed from the DDL. |
Remove all filters with ids lower than filter_id and renumber remaining ones.
If filter_id is omitted, the method uses the current filter id from the view.
After pruning, filter ids are normalized so the smallest remaining id becomes 1,
and the public view is repointed to filter_id=1.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filter_id
|
int
|
Threshold id; rows with |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
FilterManager |
Self, to allow method chaining. |
clone_filter(source_filtermanager, filter_id_to_apply=1, take_ownership=False, clone_mode='soft', if_exists='error')
Clone filter definitions from another FilterManager.
Supports: - soft clone (default): just point to source _HIDDEN table - hard clone: copy the source _HIDDEN table and own the copy
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_filtermanager
|
FilterManager
|
Source FilterManager to clone. |
required |
filter_id_to_apply
|
int
|
Filter ID to activate. Default: 1. |
1
|
take_ownership
|
bool
|
Whether this manager owns the cloned table (soft mode only). |
False
|
clone_mode
|
str
|
"soft" or "hard". Default: "soft". |
'soft'
|
if_exists
|
str
|
Behavior if target hidden table already exists - "error" (default): raise an exception - "replace": drop and recreate - "skip": reuse existing table |
'error'
|
Returns:
| Type | Description |
|---|---|
|
FilterManager |
Raises:
| Type | Description |
|---|---|
ValueError
|
On invalid clone_mode or missing source. |
Take ownership of the currently linked hidden filter table.
This enables this FilterManager instance to manage (and potentially drop)
the hidden table via _drop() or future maintenance methods.
Returns:
| Name | Type | Description |
|---|---|---|
FilterManager |
self (for chaining) |
Retrieve the currently applied filter condition from the database.
This method executes a SQL query that aggregates the filter columns from a filter manager view and returns the result either as a Python dictionary or a raw JSON string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_type
|
str
|
Format of the returned filter condition. - "dict": Returns a Python dictionary (default). - "str": Returns a raw JSON string. |
'str'
|
Returns:
| Type | Description |
|---|---|
|
dict | str | None: The current filter condition in the requested format. Returns None if no filter is present. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If the SQL execution fails. |
Example
filters = self.get_filter_condition(output_type="dict") print(filters)
Bases: ValueError
A detected TimeManager is refused (TIME_MANAGER_CHECK='error'): nothing is registered.
Bases: RuntimeError
Raised when filter-level ingestion failures are escalated by on_failure.
Carries the machine-readable detail so an orchestrator does not have to parse
the message: process_id, n_failed, n_total and failures
(a list of {filter_id, error}).
build_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, feature_store_time=False, join_type='INNER', grouped=False, document=False, vector_indexing=False, record_lineage=True, as_of=None, time_manager=None, feature_offsets=None, offset_strategy=None)
Generate a SQL query and create a Teradata view to retrieve records for the specified entity IDs based on selected features.
Parameters:¶
entity_id : list, dict, or str The entity IDs for which records are needed. Can be a list, dictionary, or string. - If list: contains multiple entity IDs. - If dict: keys are the entity IDs. - If str: a single entity ID.
list, str or dict
The feature names alone (['amount', 'nb_tx']) when each feature is
computed by one process: its version is found in the catalog. A version is
needed only for a feature several processes compute; write it in a dict,
with None for the others ({'amount': '<UUID>', 'nb_tx': None}).
Such a feature named without a version raises FeatureSelectionError
(a ValueError) naming its processes and the dict to write; so does a
name not registered for this entity. All features are resolved with one
catalog query. In the dict, version is:
- None: the feature's only version,
- {"alias": <column name>}: the only version under a name of its own,
- a process UUID string (the column takes the feature's name),
- {"process_id": <UUID>, "alias": <column name>}: one version under a name of its own, or
- a list of {"process_id": <UUID>, "process_view_name": <str>, "alias": <str, optional>}:
several versions side by side, one column each, named <feature>_<process view>
unless an alias is given.
An alias is a plain identifier (letters, digits, _) and may not repeat another column;
offsets (feature_offsets) are suffixed to it (spend_LAG_1D).
str
The name of the view to be created in the database.
str, optional
The name of the database schema where the view will be created.
Defaults to the global tdfs4ds.SCHEMA if not provided.
bool, optional
If True, the function will return the SQL query string instead of executing it. Defaults to False.
bool, optional
If True, the query will use a specific VALIDTIME based on tdfs4ds.FEATURE_STORE_TIME.
If False, the query will use CURRENT VALIDTIME. Defaults to False.
str, datetime, date or pandas.Timestamp, optional
Freeze the dataset at this point in time (VALIDTIME AS OF TIMESTAMP).
Any ISO date or timestamp is accepted — '2024-03-31',
'2024-03-31 23:59:59', '2024-03-31T23:59:59+02:00' — and
normalised to the literal Teradata needs (a bare date becomes midnight;
Teradata rejects a date-only TIMESTAMP literal). Without an offset the
value is read in the session time zone, like FEATURE_STORE_TIME.
TimeManager or str, optional
Make the dataset follow a TimeManager: every feature is read as of
the manager's current BUSINESS_DATE, so time_manager.update(i)
moves the dataset without rebuilding it. A 'database.view' string
naming the manager's public view is accepted too.
as_of, time_manager and feature_store_time=True are mutually
exclusive. With none of them the dataset reads CURRENT VALIDTIME (the
default, unchanged). The chosen mode is recorded in the dataset catalog:
DATASET_VALIDTIME holds the qualifier and the build table
(FS_V_<catalog>_BUILD) holds the mode, timestamp or TimeManager.
str, optional
Specifies the type of join to use between features: 'INNER' (default) or 'FULL OUTER'.
bool, optional
If False (default), uses one sub-query per feature joined together — the classic multi-join approach. Every individual feature sub-query can be targeted by a PI-equality filter pushed from the outer WHERE, making single-entity lookups single-AMP. If True, groups features that share the same storage table and process version into a single sub-query with MAX(CASE WHEN …) pivoting, reducing the number of JOINs from N_features to N_distinct_(table, version) pairs. A HAVING clause enforces non-NULL for every feature in the group when join_type='INNER', preserving the same missing-feature semantics as the classic approach. PI push-down is preserved for INNER JOIN (outer SELECT uses A1.entity); for FULL OUTER the COALESCE in the outer SELECT prevents push-down, same as the classic approach.
bool, optional
If True, generate and persist business-dictionary documentation for the
newly built dataset via document_dataset_incremental (LLM-powered).
Requires tdfs4ds.INSTRUCT_MODEL_PROVIDER and tdfs4ds.INSTRUCT_MODEL_MODEL
to be configured; silently skipped (with an info log) otherwise. Failures
are non-fatal — the dataset view is always created/returned regardless.
Defaults to False (opt-in). Mirrors upload_features. Leave False when
the dataset is built via upload_features/run() — that path already
documents it, so enabling it here would double-document.
bool, optional
If True (and document=True and an instruct model is configured), refresh
the consumer-agent vector index via build_vector_index(force_rebuild=False)
after documentation so the new dataset is searchable immediately. Non-fatal.
Defaults to False (opt-in).
bool, optional
Persist the dataset view's lineage subgraph (dataset -> process views ->
sources) into FS_LINEAGE_GRAPH via record_process_lineage. No LLM is
involved and the write is diff-based (an unchanged graph writes nothing).
Non-fatal. Defaults to True since 0.3.1.220: with the former opt-in
default no dataset reached the lineage graph unless
rebuild_lineage_graph() was run afterwards. Pass False to skip it.
dict, optional
Read features at other instants than the dataset's own (lags and leads),
e.g. {'amount': ['0', '-1 day', '-2 days'], 'churn': ['+1 month']}:
'-1 day'is the value one day BEFORE the dataset's instant (a lag, for autoregressive inputs),'+1 month'one month AFTER (a lead, e.g. a forecast target),'0'the instant itself.- A listed feature gets one column per offset (
amount,amount_LAG_1D,amount_LAG_2D,churn_LEAD_1MO); it keeps its plain column only when'0'is listed. - The instant is the dataset's: now,
as_of, or the TimeManager's business date (time_manager.update()moves every offset column). - A lead is NULL while its instant is later than the database's current
time: what is valid after now is not known yet. Under
CURRENTa lead is therefore always NULL (refused withjoin_type='INNER'). - Units: year, month, week, day, hour, minute, second; months are
calendar months (
ADD_MONTHS). No fraction of a second: valid times are stored to the second.
Without it the SQL is exactly what it was. See the user guide, "Lags and leads".
{'PIVOT', 'PER_OFFSET'}, optional
How the offset columns are read. 'PIVOT' (default) reads each
storage table (and process version) once for all its offset columns:
one aggregation step, local to the AMPs, partition elimination on the
FEATURE_IDs. 'PER_OFFSET' gives each offset column its own sub-query
and join; measured on Vantage 20 it parses far slower (32 columns: 31 s
against 0.35 s) and is kept for comparison.
Returns:¶
str or tdml.DataFrame
- If return_query is True, returns the generated SQL query as a string.
- Otherwise, returns a Teradata DataFrame containing the results.
Raises:¶
ValueError
If an invalid join_type is provided.
True if exc looks like a Teradata ACC-ineligibility error (6881).
The error message without the driver's stack trace.
Keeps EVERY message line — the half that names the fault is often not the first line — and drops only the Go frames teradatasql appends and the SQLAlchemy "Background on this error" link. Falls back to the raw text when nothing would be left.
Return name as a Teradata identifier, quoting it only if required.
Idempotent: an already-quoted name is returned unchanged rather than being
wrapped a second time. Embedded quotes are escaped as "".
Parameters¶
name : str
The raw object/database name.
force : bool, default False
Quote even when the name does not need it. Use only at the few sites
that have always emitted unconditional quotes and whose exact output is
already baked into persisted DDL (see
feature_store/feature_query_retrieval.py) — elsewhere forcing would
change the SQL of existing stores for no reason.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Return the volatile ART name declared by an INTO VOLATILE ART(<name>)
statement, or None when the statement is not of that shape / can't be parsed.
Used only in UAF ART view mode to DROP a stale volatile ART before recreating it (idempotent same-session re-run). A miss is harmless — the DROP is simply skipped.
Execute the UAF ART materialisation statements (art_sql) in list order.
For each statement: DROP the volatile ART it declares first (idempotent — a fresh
session has none; a re-run/next-partition has a stale one), then execute the
statement. All via tdml.execute_sql on the active session, so the resulting
session-scoped volatile ART is visible to the ingestion CTAS that follows.
art_sql order is authoritative and must NOT be sorted/deduped — a later
statement may read an ART an earlier one created. A no-op when art_sql is
falsy (the classic non-UAF path). Re-raises on statement failure so the caller's
followup_close(status='FAILED,...') records it.
Rows staged for ingestion, or None when the count could not be taken.
Used to tell an ingestion that produced nothing from one that produced real
values. A process reading an empty upstream view ingests zero rows and used to
record a plain COMPLETED — indistinguishable from a real result, and
permanently pinned by the skip predicate (STATUS == 'COMPLETED' at this
valid time), so the hollow values survived every later run until someone
deleted the row by hand.
Fail-open by design: a count is a diagnostic, never a reason to fail an
ingestion that already succeeded, so any error yields None = "unknown",
which the caller treats exactly as the historical behaviour.
{feature: sql type} of the feature columns of the source, or None (fail-open).
The catalog rows of the features of this result set that are already registered
(type_changes.stored_types), or None when the catalog could not be read.
At run time: warn about registered features whose source type no longer fits the stored type (the run writes with the stored type), and return the truncation checks. Never raises.
At registration (0.3.5.17): the refusal text when a feature has a type a value cannot have (CLOB, BLOB, JSON, XML), else None. Fail-open: unreadable types = no refusal (the run-time check still refuses CLOB/BLOB/JSON).
At registration: the refusal text when a pre-existing feature's source type is not held by its stored type, else None. Fail-open (a read failure = no refusal).
(explicit, value) for an upload_features argument: None = not given (the
recorded value, else 1024); 0 or 'none' = exact lengths (value None); a number = it.
The FORCE_VARCHAR_LENGTH recorded for the process already registered on this view
(a re-registration), else 1024. Fail-open to 1024.
Record FORCE_VARCHAR_LENGTH for the process. Never raises: a store without the
settings table runs with 1024 (the default every older process used); any other value is
then a warning.
One aggregate on the staged batch: for each registered VARCHAR feature whose source is longer than its stored length, the longest staged value and the rows above the stored length. Logs a WARNING; never raises (the batch writes as before).
'missing' / 'same' / 'current' / 'other' for an existing dataset view.
same: already built with this TimeManager. current: built at CURRENT VALIDTIME.
other: any other temporal mode, another manager, or a view that cannot be read as a
tdfs4ds dataset. missing: no such view. Never raises (an unreadable state is other:
the view is left alone).
The dataset build of upload_features / run.
Without dataset_time_manager this is exactly the historical call. With it, the dataset is
built with build_dataset(time_manager=...) so tm.update() moves it; a dataset that
already exists in another mode is not rebuilt silently (a warning names rebuild_dataset).
tdfs4ds.TIME_MANAGER_DETECT (default True): find the TimeManager a view reads
when upload_features is given none (0.3.6.0).
A TimeManager rebuilt (read-only) from a lineage node id DB.VIEW.
The TimeManager view_name reads, from its own lineage (deep), or None.
Returns (time_manager_or_None, graph). With a domain clock (0.3.6.0): a view on the
clock's dates gets the clock (or the one clone it reads); a view on other dates is a
warning (TIME_MANAGER_CHECK='error': _TimeManagerRefused). Without one: the
single manager the view reads; several are a warning and nothing is chosen. Fail-open:
a lineage that cannot be built chooses nothing.
The TimeManager a newly registered process should read (0.3.6.0).
soft_clone=None: when time_manager was found by detection (detected) and is the
clock of the data domain, the process gets its own soft clone of it and its view is
rewritten to read the clone (data_domain_settings.clone_for_process). A manager
passed explicitly is kept, as before 0.3.6.0, even the domain clock. True: a clone
for any manager (the domain clock or not, passed or found). Anything else, or a
failure, keeps time_manager.
'database.view' of the TimeManager the dataset must follow, or None.
dataset_time_manager: True / False override; None = the recorded setting (read
only when a dataset is built, wanted). The manager is the time_manager= argument, else
the one recorded for the process. Without one: a single warning and the historical build.
Why column cannot be the valid time of this process, or None.
A DATE / TIMESTAMP column of the source, neither an entity column nor a feature, and the settings table must exist (or be creatable) to record it.
_compute_features_batch(df, entity_id, feature_names, selected_features, primary_index, partitioning, entity_null_substitute, process_id, filtermanager, dataset_view_name, dataset_already_created, grouped=False, art_sql=None, valid_start=None, close_absent=False, track_refresh=False)
Execute one feature ingestion batch: followup_open → [materialise ART] → prepare → store → followup_close (COMPLETED) → optional dataset build.
Returns (features_infos, dataset_or_None) on success.
On failure: logs the error, attempts to record FAILED in the followup table, then re-raises.
Callers that want to continue after a failure (e.g. the filtermanager loop) should wrap
this call in their own try/except and use continue rather than re-raising.
Does NOT call apply_collect_stats — the caller is responsible for that.
grouped is forwarded to build_dataset — see upload_features for details.
art_sql (UAF ART view mode only): a list of INTO VOLATILE ART statements
materialised before ingestion, in this same session — so the session-scoped
volatile ART is visible to the ingestion CTAS. Runs after filtermanager.update()
(done by the caller), so the series-input view's scalar subquery already points at
the current partition. None/empty for the classic path (a no-op).
valid_start: the run's pinned valid start for a real-time run (no
FEATURE_STORE_TIME), so every feature of the run shares one start. None
keeps the per-statement CURRENT VALIDTIME (see run).
close_absent (process option close_absent_entities, 0.3.3.22): after the
values are stored, end the validity of the entities of this process (and of the
active FilterManager segment) that this batch no longer produces, at the same
valid time. Skipped when the batch staged 0 rows; a failure records the batch FAILED.
track_refresh (process option track_entity_refresh, 0.3.3.30): after the values (and
the close of absent entities) are stored, record in the process's admin FS_R_* table
that this batch confirmed each of its entities. Skipped when the batch staged 0 / an
unknown number of rows. Fail-open: a failed write is a warning, the batch stays COMPLETED.
_compute_features_batch_tagged(df, entity_id, feature_names, selected_features, primary_index, partitioning, entity_null_substitute, process_id, filtermanager, dataset_view_name, dataset_already_created, grouped=False, art_sql=None, valid_start=None, close_absent=False, track_refresh=False)
Body of :func:_compute_features_batch, run inside the execution's query band.
Resolve the effective failure policy from the two spellings.
on_failure is the real setting; raise_on_failure is the shorthand a
caller who only wants "make it fail loudly" would reach for. When both are
given the explicit on_failure wins and the disagreement is logged rather
than silently resolved — a caller who wrote both had a reason to expect each.
Returns one of ON_FAILURE_POLICIES.
{feature: {'type': ...}} for every feature of the batch, the new ones from new_types.
Gettdtypes returns only the features not registered yet; the storage tables of the
registered ones must exist too. With stored ({FEATURE upper: {"type": ...}}, the
catalog rows, 0.3.5.1) the registered features take their CATALOG type: re-deriving it
from the source with the current rounding created empty tables nothing writes to (seen
live). Fail-open: when the source types cannot be read, new_types alone (the historical
behaviour).
_upload_features(df, entity_id, feature_names, feature_versions=_FEATURE_VERSION_DEFAULT, primary_index=None, partitioning='', filtermanager=None, entity_null_substitute={}, process_id=None, force_compute=False, force_varchar_length=None, dataset_view_name=None, grouped=False, art_sql=None, on_failure=None, raise_on_failure=None, valid_start=None, close_absent=False, track_refresh=False)
Uploads a set of features into the Feature Store for a given entity.
This function registers an entity and its associated features in the feature catalog if they are not already defined, prepares the data for ingestion, and stores it in the feature store. It also supports incremental feature computation and conditional execution depending on prior runs.
Parameters¶
df : pandas.DataFrame
Input dataframe containing entity keys and feature columns to upload.
entity_id : str, list, or dict
Identifier(s) for the entity. List/str inputs have their SQL types inferred
from df with exact length + character set preserved (e.g. CHAR(14) keys are
not truncated); a dict is used verbatim.
feature_names : list of str
List of feature column names to upload from df.
feature_versions : dict or str, optional
Feature version(s). Default is _FEATURE_VERSION_DEFAULT ('dev.0.0').
primary_index : str or list, optional
Primary index for Teradata feature tables.
partitioning : str, optional
Partitioning clause. Default is ''.
filtermanager : FilterManager, optional
If provided, features are built iteratively per filter step.
entity_null_substitute : dict, optional
Replacement values for nulls in entity keys (applied via COALESCE). A NULL in
an entity key column not listed here raises a ValueError during ingestion
(NULL keys cause a cartesian-product MERGE).
process_id : str, optional
Process execution identifier for follow-up logging.
force_compute : bool, optional
If True, forces recomputation even if already completed. Default is False.
force_varchar_length : int, optional
If provided, all VARCHAR feature columns are resized to this length.
dataset_view_name : str, optional
Name of the output dataset view to create after ingestion.
grouped : bool, optional
Forwarded to build_dataset whenever a dataset view is built. See
upload_features for details. Default: False (classic multi-join).
art_sql : list of str, optional
UAF ART view mode. Forwarded to every _compute_features_batch call so the
volatile ART(s) are materialised per partition before ingestion. None for
the classic path. See upload_features / _materialise_art.
close_absent : bool, optional
Process option close_absent_entities: every batch (each FilterManager
segment) ends the validity of the stored entities it no longer produces.
track_refresh : bool, optional
Process option track_entity_refresh: every batch records, in the process's admin
FS_R_* table, that it confirmed each of its entities (see entity_refresh).
Returns¶
pandas.DataFrame or None If BUILD_DATASET_AT_UPLOAD is enabled, returns the built dataset. Otherwise None.
The database's current timestamp, as a FEATURE_STORE_TIME-style string.
Read from the database (not the client) so it agrees with CURRENT VALIDTIME.
None when it cannot be read: the caller then keeps the historical
per-statement CURRENT VALIDTIME.
(database, view) of the TimeManager recorded for process_id, or None.
Recorded in FS_TIME_MANAGER by roll_out / run(..., time_manager=)
and exposed as TIME_DATABASE_NAME / TIME_VIEW_NAME by the process
catalog view. Never raises.
Decide the valid time of one run(). Returns (source, value, detail).
Precedence:
'explicit'—tdfs4ds.FEATURE_STORE_TIMEis set (aroll_outstep or the caller's choice): used as is, nothing else consulted.'argument'—run(..., time_manager=tm): the manager's current business date. A manager that cannot give a date is an error (it was asked for explicitly).'recorded'— the TimeManager recorded for the process: its current business date. A broken association is a warning, never a failure.'realtime'— none of the above: the database time, read once, so that every feature of the run shares one valid start (valuemay beNoneif the clock cannot be read: the historical per-statement behaviour).
({pid: [manager node, ...]}, index): the TimeManagers each registered process reads,
from the lineage (deep); the recorded one (catalog TIME_*) when the lineage does not
know the process. Pure over its inputs.
_run_many(process_id=None, *, at=None, data_domain=None, order='dependencies', time_manager=None, return_dataset=False, on_failure=None, raise_on_failure=None, dataset_view_name=None, **run_kwargs)
Body of run([...]) / run(data_domain=...) / run(..., at=) (0.3.6.0).
Every process is computed at ONE time: at (a date or timestamp), else 'now' (the
database clock, read once), or the current date of time_manager. Each TimeManager view
the processes read (their soft clones, or a shared manager) is set to that date for the
computation and put back on its cursor afterwards (literal_time_managers): no cursor
moves. A process reading no TimeManager is computed on the current source data and
stored at the same time, with a warning. Processes run one after the other in
dependency order; a failed process's downstream processes are skipped.
run(process_id=None, return_dataset=False, force_compute=False, force_varchar_length=None, dataset_view_name=None, grouped=False, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, time_manager=None, close_absent_entities=None, _time_manager_checked=False, track_entity_refresh=None, dataset_time_manager=None, *, at=None, data_domain=None, order='dependencies')
Execute a specific process from the feature store identified by the process ID.
Uses global logger for diagnostics (gated by tdfs4ds.DISPLAY_LOGS).
Several processes at one time (0.3.6.0): process_id may be a list, or pass
data_domain='X' (True = tdfs4ds.DATA_DOMAIN) for every process of a data domain
(primary store only). They run one after the other in dependency order (from the
lineage graph; order='given' keeps yours), all at ONE time: at='2024-03-31', else
the database clock read once (at='now', the default for a list), or the current
date of time_manager=. Each TimeManager view the processes read (their soft clones)
is set to that date for the computation and put back on its cursor afterwards: no
cursor moves. A process reading no TimeManager is stored at the same time, with a
warning. A failed process's downstream processes are skipped; on_failure applies
per process ('raise' raises FeatureIngestionError after the list, 'stop' stops at
the first failure). Returns {process_id: 'COMPLETED' | 'FAILED: ...' |
'SKIPPED: ...'}. run('pid', at=...) computes one process at that date and returns
what run('pid') returns.
grouped is forwarded to build_dataset whenever a dataset view is (re)built.
See upload_features for details. Default: False (classic multi-join, unchanged
behaviour). Note: not persisted in the process catalog — re-running an existing
process_id without passing grouped=True again reverts to the classic join.
Valid time of the ingested values, decided once per run:
tdfs4ds.FEATURE_STORE_TIME, when set — whatroll_outdoes at each step.time_manager=— the manager's current business date (the pairing is recorded, asroll_outrecords it).- the TimeManager recorded for the process (by
roll_outor a previoustime_manager=), at its current business date. - otherwise real time: the database time, read once, shared by every feature and partition of the run.
For 2 and 3, FEATURE_STORE_TIME is set for the duration of the run (the
follow-up row records that date) and restored afterwards. Running without a
FEATURE_STORE_TIME or a TimeManager is a fallback, not a best practice.
A time_manager= the process view does not read (checked on the lineage graph,
deep) is governed by tdfs4ds.TIME_MANAGER_CHECK: 'warn' (default) runs as
before but records no pairing; 'error' raises ValueError before computing
anything; 'off' skips the check. A real-time run of a process whose view reads a
TimeManager that is not declared logs a warning naming link_time_managers.
close_absent_entities: None (default) applies the option recorded for the
process (upload_features(..., close_absent_entities=True) or
set_process_options); True / False override it for this call only. When on,
the entities the process no longer produces have their validity ended from the
run's valid time on (the MERGE's "from t onward" rule).
track_entity_refresh: None (default) applies the option recorded for the process
(upload_features(..., track_entity_refresh=True) or set_process_options); True /
False override it for this call only. When on, the run records in an admin table
(FS_R_*, read by tdfs4ds.entity_refresh) that it confirmed each entity of a batch,
whether or not the value changed. A failed write is a warning; the run stays COMPLETED.
dataset_time_manager: None (default) applies the setting recorded for the process
(upload_features(..., dataset_time_manager=True) or set_process_options); True / False
override it for this call only. When on, the dataset view built by this run (dataset_view_name)
follows the process's TimeManager (the time_manager= argument, else the recorded one): it is
built with build_dataset(time_manager=...), so tm.update() moves it. A dataset that already
exists in another mode is left as it is, with a warning naming rebuild_dataset.
Other parameters:
process_id: a PROCESS_ID, or a list of them (0.3.6.0).return_dataset: return the dataset view built by the run (withdataset_view_nameortdfs4ds.BUILD_DATASET_AT_UPLOAD) instead of None.force_compute: recompute even when the follow-up table records the run as COMPLETED at this valid time (default False: such a run is skipped).force_varchar_length: VARCHAR rounding of new text features; None = the value recorded for the process (seeupload_features).dataset_view_name,grouped: a dataset to (re)build after the run, and its strategy.on_failure('continue' | 'stop' | 'raise') /raise_on_failure(True = 'raise'): what a failed FilterManager segment does; 'raise' attempts every segment, then raisesFeatureIngestionError.check_upstream/strict_upstream: refuse the run when an upstream process (lineage graph) FAILED at this valid time / is also RUNNING.time_manager: compute at this manager's current date (see above); a process reading a soft clone of it gets the clone set to that date for the run (0.3.6.0).at,data_domain,order(keyword, 0.3.6.0): see "Several processes at one time".
_run(process_id, return_dataset=False, force_compute=False, force_varchar_length=None, dataset_view_name=None, grouped=False, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, valid_start=None, close_absent_entities=None, track_entity_refresh=None)
Body of :func:run, once the valid time is decided (valid_start: real-time pin).
Split schema.object into (schema, object) (both de-quoted, stripped).
Returns ("", object) when no schema qualifier is present — the caller
enforces the mandatory-schema rule on the empty schema.
Normalise an upload_features source to a teradataml DataFrame.
Accepts three input shapes and returns (df, view_name):
- a teradataml DataFrame → returned unchanged (back-compat);
- a fully-qualified
schema.view→ resolved to a DataFrame over that view; sql_query+target_view_name→ the view is CREATED from the SQL, then resolved to a DataFrame (the created view is permanent and becomes the process view re-run byrun(process_id)).
On any recoverable problem (missing schema, view already exists, SQL/view
fails to compile or is not executable) it logs a clear, prescriptive message
via logger_safe('error', …) and returns (None, None) so the caller
aborts without raising a traceback (the users are non-developers).
Effective value of one AI step of upload_features.
An explicit argument (True/False) always wins. None reads the configured
default tdfs4ds.<setting> (TDFS4DS_<setting> in a .env), unless the
master switch tdfs4ds.AGENTIC_FEATURES is off, which makes every
defaulted step False. The one rule for all five steps.
upload_features(df, entity_id, feature_names, metadata={}, primary_index=None, partitioning='', filtermanager=None, entity_null_substitute={}, force_compute=True, force_varchar_length=None, dataset_view_name=None, optimize_query=None, update_view=True, document=None, max_candidates=None, cascade_redocumentation=None, vector_indexing=None, sql_query=None, target_view_name=None, art_sql=None, art_layers=None, grouped=False, return_process_id=False, run_now=True, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, explain=None, time_manager=None, close_absent_entities=False, storage_tag=None, track_entity_refresh=False, dataset_time_manager=False, valid_time_column=None, soft_clone=None)
Upload feature data to the feature store for a specified entity.
All diagnostics go through logger_safe() which respects tdfs4ds.DISPLAY_LOGS.
The feature source can be supplied in three ways:
- A teradataml DataFrame (
df=...) — the classic path. - A fully-qualified view name (
df='SCHEMA.view') — for SQL users who already have a view. The schema is mandatory; a bare view name (no schema) is rejected. The view must already exist and be executable. - A SQL query (
sql_query='SELECT ...') together with a mandatorytarget_view_name='SCHEMA.view'. The query is validated withEXPLAIN, a view is created attarget_view_name(the schema is mandatory; the view must NOT already exist), and that view — which is permanent and becomes the process view re-executed byrun(process_id)— is used as the feature source. Whensql_queryis given it takes precedence overdf.
Recoverable input problems (missing schema, target view already exists, SQL
or view fails to compile / is not executable) do not raise a traceback:
a clear message with a fix is logged via logger_safe('error', …) and the
function returns None. Check the log when a call returns None.
Parameters¶
df : teradataml DataFrame or str
The feature source: a DataFrame over a permanent view, or 'SCHEMA.view'.
Pass None with sql_query.
feature_names : list of str
The columns of the source to register as features.
metadata : dict, optional
Free-form description stored with the process (METADATA JSON). Default: {}.
primary_index : list of str, optional
Primary index of the storage tables. Default: the entity columns.
partitioning : str, optional
Partitioning clause of the storage tables. Default: none.
filtermanager : FilterManager, optional
Segments the process runs one at a time (see the fs-filter skill).
force_compute : bool, optional
Recompute even when the follow-up table records the run as COMPLETED at this
valid time. Default: True.
dataset_view_name : str, optional
'SCHEMA.view' of a dataset built from the features after the run.
on_failure : {'continue', 'stop', 'raise'}, optional
What a failed FilterManager segment does: 'continue' (default) records it
and goes on, 'stop' stops at the first one, 'raise' attempts every
segment then raises FeatureIngestionError.
raise_on_failure : bool, optional
Shorthand: True = on_failure='raise'. An explicit on_failure wins.
check_upstream : bool, optional
Refuse the run when an upstream process (from the lineage graph) FAILED at this
valid time. Default: False.
strict_upstream : bool, optional
With check_upstream, also refuse when an upstream is RUNNING.
sql_query : str, optional
A SQL SELECT defining the feature source. Requires target_view_name.
The view is created from this query (after an EXPLAIN check) and kept.
target_view_name : str, optional
'SCHEMA.view' — where to create the view for sql_query (mandatory
when sql_query is given; schema required; must not already exist).
art_sql : list of str, optional
Opt-in UAF ART view mode. A list of solitary SQL statements that
materialise a Teradata UAF operator's Analytical Result Table (ART) using
EXECUTE FUNCTION ... INTO VOLATILE ART(<name>). When provided, the
process is registered with PROCESS_TYPE='uaf art view' and, on every
run(), these statements are executed in list order, per FilterManager
partition, immediately before the ingestion CTAS (same session, so the
session-scoped volatile ART is visible to the CTAS). This unlocks features
whose value lives in a secondary ART layer (e.g. ARIMA fit-metrics
R_SQUARE/MAPE/AIC in ARTFITMETADATA, residual tests over
ARTFITRESIDUALS, TD_PACF) — which a single ingestion statement
cannot reach. df/sql_query+target_view_name still define the
plain feature view (the one reading the layer via
TD_EXTRACT_RESULTS(ART_SPEC(..., LAYER(...)))); art_sql is stored in
the process METADATA and never turned into a view (a view cannot contain
INTO VOLATILE ART). Order matters — a later statement may read an ART
an earlier one created. For this mode optimize_query is force-disabled
and the view EXPLAIN in documentation is skipped (the ART does not exist at
optimize/document time). Requires art_layers. Default: None (classic
'denormalized view' path, unchanged).
art_layers : list of str, optional
Required when art_sql is given: the declared LAYER(...) name(s) the
feature view reads (e.g. ['ARTFITMETADATA']). Used for documentation and
lineage only — it is not validated against the view DDL (the view is
authoritative for what is computed). Default: None.
entity_id : str or list (recommended), or dict (advanced override)
Entity / primary-index key(s). Pass a string for a single-column key or a
list for a composite key, e.g. 'CUSTOMER_ID' or
['CUSTOMER_ID', 'EVENT_DATE'] — this is the recommended form. You do not
need to specify SQL data types: they are inferred from the source DataFrame
with their exact length and character set preserved (a CHAR(14) key stays
CHAR(14) in the created feature table, not truncated to CHAR(1)). Only pass a
dict (e.g. {'CUSTOMER_ID': 'BIGINT', 'EVENT_DATE': 'DATE'}) in the rare
case where you need to override an inferred type; the supplied type strings are
then used verbatim.
entity_null_substitute : dict, optional
Per-column replacement values for NULLs in entity keys (e.g.
{'CLI_CD': '-'}), applied via COALESCE. NULL values in an entity key column
that is NOT covered here will raise a ValueError during ingestion: a NULL key
produces a cartesian-product MERGE (severe skew / storage blow-up). Filter the
NULL rows upstream or provide a substitute. Default: {}.
optimize_query : bool, optional
If True and an instruct model is configured, run the query optimizer on the
registered process after ingestion. The full report is saved to a file named
tdfs4ds_optim_{timestamp}_{view}_{process_id}.md in the current working
directory. Only the score summary is emitted to the log.
Default: None = tdfs4ds.UPLOAD_OPTIMIZE_QUERY (False unless configured).
The AI steps (``optimize_query``, ``document``, ``explain``,
``cascade_redocumentation``, ``vector_indexing``) all default to None,
which reads ``tdfs4ds.UPLOAD_<NAME>`` (``TDFS4DS_UPLOAD_<NAME>`` in a .env).
``tdfs4ds.AGENTIC_FEATURES = False`` (``TDFS4DS_AGENTIC_FEATURES=false``)
turns all of them off at once. An explicit argument always wins.
update_view : bool, optional
If True and query optimization finds an improvement, replace the process view
DDL with the optimized SQL. Default: True.
document : bool, optional
If True and an instruct model is configured, run incremental documentation on
the registered process (and after any view update). If dataset_view_name is
also provided, document_dataset_incremental is called after run() so
that the dataset-level documentation can propagate the already-generated feature
descriptions bottom-up.
Default: None = tdfs4ds.UPLOAD_DOCUMENT (True unless configured).
explain : bool, optional
Part of document: also run the LLM analysis of the view's EXPLAIN plan
(scores, warnings, recommendations). False documents the process without it
and keeps any stored analysis. Ignored when document is False.
Default: None = tdfs4ds.UPLOAD_EXPLAIN (True unless configured).
max_candidates : int, optional
Maximum number of SQL rewrite candidates to generate during query optimization.
Overrides tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATES for this call only.
Default: None (uses the global value).
cascade_redocumentation : bool, optional
If True (and document=True with an instruct model configured), re-document
every downstream process/dataset that depends on this process — but only when
this process's documentation actually changed. The descendant set is read from
the persisted lineage graph (FS_LINEAGE_GRAPH). This propagates a changed
parent meaning down to everything that consumes it, at the cost of one extra LLM
pass per affected descendant; hence it is opt-in.
Default: None = tdfs4ds.UPLOAD_CASCADE_REDOCUMENTATION (False unless configured).
vector_indexing : bool, optional
If True (and document=True with an instruct model configured), rebuild the
vector index after documentation is written so the consumer agent's search
results reflect the new/updated business descriptions immediately. Uses an
additive upsert (force_rebuild=False) — fast but stale entries from
deleted/renamed objects persist until a full rebuild_vector_indexing() is
run. Non-fatal: a failure logs a warning without aborting the ingestion.
Default: None = tdfs4ds.UPLOAD_VECTOR_INDEXING (False unless configured).
grouped : bool, optional
Forwarded to build_dataset whenever a dataset view is built (i.e. when
dataset_view_name is set or tdfs4ds.BUILD_DATASET_AT_UPLOAD is True).
If False (default), uses one sub-query per feature joined together — the
classic multi-join approach; every feature sub-query can be targeted by a
PI-equality filter, keeping single-entity lookups single-AMP.
If True, groups features sharing the same storage table and process
version into one sub-query with MAX(CASE WHEN …) pivoting, reducing JOIN
count from N_features to N_distinct_(table, version) pairs — recommended for
wide datasets with many features from the same source table. See
build_dataset for the full explanation. Default: False (unchanged
behaviour). Not persisted in the process catalog — a later run(process_id)
without grouped=True reverts to the classic join for any dataset rebuild.
return_process_id : bool, optional
If True, return the PROCESS_ID (a UUID string) of the process registered by
this call instead of the dataset — so the caller can hold on to it for a later
run(process_id) / roll_out([process_id], ...) without having to look it
up in the process catalog. Mirrors run(return_dataset=...): the flag toggles
what is returned, it does not wrap it in a tuple. Note the default return
(dataset) is already None unless tdfs4ds.BUILD_DATASET_AT_UPLOAD is
True, so in the common case nothing is given up by setting this. The recoverable
input-error paths still return None regardless (check the log). Default:
False (unchanged behaviour — the dataset is returned).
run_now : bool, optional
If False, register the process without executing it — the process catalog row,
the data-distribution row and any filtermanager row are written and the view is
created, but run() is not called, so no feature values are computed. Use it
for a lazy "declare now, compute later" flow: the returned PROCESS_ID (pair with
return_process_id=True) can be executed at any later point via run(process_id)
or roll_out([process_id], time_manager), which drives every vintage including the
first. A dataset_view_name given with it is recorded as a process setting and built
by each later run() that passes none (so by roll_out, at each date; 0.3.5.14).
Caveat: register_entity / register_features and the FS_T_*
table creation happen inside run(), so until the first run the feature
catalog has no entry for these features and their storage tables do not exist —
get_feature_versions and friends return nothing for them. That is the intended
lazy state, not an error. Dataset documentation is also skipped (the dataset view
does not exist yet).
**Validation.** Because the run is what normally proves the source executes,
``run_now=False`` first validates it with an ``EXPLAIN`` — **before** any catalog
row is written, so an invalid source never leaves a registered process behind. On
failure nothing is registered: a prescriptive message is logged and ``None`` is
returned (no traceback), like the other recoverable paths. For a **UAF ART view**
only the **first** ``art_sql`` statement is EXPLAINed and the ART is **not**
materialised — later statements may read an ART an earlier one creates, so without
materialisation they would fail for a reason unrelated to their own validity; the
feature view is skipped for the same reason (it reads the ART → Error 3807).
**What EXPLAIN does and does not prove.** It proves the statement compiles and its
objects are reachable. It does **not** prove the process will ingest: ``run()`` also
performs type resolution (``Gettdtypes`` / ``validate_feature_types``) and rejects
NULL entity keys not covered by ``entity_null_substitute``, and the real ingestion
statement is an unpivot CTAS rather than ``SELECT *``. So a lazily-declared process
can EXPLAIN clean and still fail on its first run. Default: True (unchanged
behaviour — register **and** run).
storage_tag : str, optional
Store this process's features in a table of their own (the table of their entity and type with
__<TAG> appended, same primary index and partitioning), recorded in FS_FEATURE_PLACEMENT
before the first run. Two processes of one entity under different tags write different tables, so
a parallel run does not serialise them. A feature already registered in another table, or shared
with another process, is refused: relocate_features moves features with their history.
Default None: nothing changes.
track_entity_refresh : bool, optional
Process option (0.3.3.30), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out() of the process. The ingestion MERGE writes nothing when a value is
unchanged, so a feature row only tells when a value last changed. When True, each batch also
records, in an admin table of its own (FS_R_*, never a feature, exposed by no catalog or
view), that the process confirmed each entity it produced, with a valid-time history. Read it
with tdfs4ds.entity_refresh(process_id) and tdfs4ds.entity_refresh_summary(process_id)
(entities by last refresh, and those the latest run did not confirm). Cost: one
SELECT DISTINCT over the staged batch and one MERGE per batch. A failed write is a warning;
the ingestion stays COMPLETED. A batch that staged 0 rows records nothing. Default False: the
ingestion statements are unchanged.
dataset_time_manager : bool, optional
Setting (Part D), recorded in FS_PROCESS_SETTINGS and applied by every later run() /
roll_out(). When True and the process has a TimeManager (time_manager= here, else the
recorded one), the dataset of dataset_view_name is built with build_dataset(time_manager=...)
(grouped or not): it shows the business date of the manager, so tm.update(i) moves it without a
re-run. Without a TimeManager: a warning and the usual current dataset. A dataset that already
exists at CURRENT VALIDTIME (or in another mode) is not rebuilt silently: a warning names
tdfs4ds.rebuild_dataset(<name>, time_manager=..., replace=True). Default False: unchanged.
force_varchar_length : int, optional
Process setting (0.3.5.1), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out(). The VARCHAR features being registered take one type per character
set: the stored type of a registered text feature of the process that holds them, else
VARCHAR(ceil(longest / N) * N) over the new features. None (default): the value
recorded for this view's process, else 1024 (a recorded value is never reset by the default).
0 or 'none': exact lengths. A registered feature keeps its stored type; a source type
it does not hold refuses the registration with the alternatives (cast, or
tdfs4ds.plan_type_change).
valid_time_column : str, optional
Process setting (0.3.5.0), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out(). A DATE or TIMESTAMP column of the source (neither an entity column
nor a feature) giving each row its own valid time: the row's values are valid from it until
the entity's next row (or 9999-01-01), and one run writes a whole history. The source may
then hold several rows per entity, one per valid time. Equal consecutive values are merged;
each entity's stored history is replaced from its earliest valid time in the load onward and
kept before it, so a re-run is a no-op and an incremental load rewrites only from its own
dates. FEATURE_STORE_TIME and the TimeManager date no longer set the values' valid time
(a TimeManager can still bound the rows the view returns). With close_absent_entities, an
entity absent from a load has no value from the load's earliest valid time on; with
track_entity_refresh, the load's latest valid time is recorded. Not for a UAF ART view.
A tdfs4ds older than 0.3.5.0 running the process ignores the setting. Default None: unchanged.
time_manager : TimeManager, optional
The TimeManager the process view reads. The pairing is recorded at registration
(FS_TIME_MANAGER, as roll_out records it), so run(process_id) later
needs no argument and computes at the manager's business date; with
run_now=True the first run already does. The view is checked first: a
manager it does not read is a warning under tdfs4ds.TIME_MANAGER_CHECK='warn'
(default; the pairing is not recorded) and refused under 'error' (nothing
registered, None returned). An explicit manager is paired as before 0.3.6.0,
even when it is the clock of the data domain (no soft clone).
None (default, 0.3.6.0): the view's lineage is walked and the manager it reads
is recorded: the clock of the data domain (set_domain_time_manager), which
then gets its own soft clone (see soft_clone), or the one manager it reads
(several: a warning, nothing recorded). tdfs4ds.TIME_MANAGER_DETECT = False
restores the pre-0.3.6.0 behaviour (nothing recorded until a roll_out).
soft_clone : bool, optional
(0.3.6.0) None (default): a process whose view reads the clock of its data
domain, found by detection, gets its own soft clone <view>_TM (same dates, own
cursor) and its view is rewritten to read it. True: a soft clone of any
manager, passed or found. False: never.
close_absent_entities : bool, optional
Process option (0.3.3.22), recorded in FS_PROCESS_OPTIONS and applied by
every later run() / roll_out() of the process. When True, each run
ends the validity of the stored entities the process no longer produces, from
the run's valid time on (the TimeManager date under roll_out) — the same
"from t onward" rule as the MERGE for the entities still produced. A re-run at
the same date is a no-op, and an entity that comes back gets a new period (the
absence stays a gap). Replay a history in date order, from its first date. A
batch that staged 0 rows closes nothing; a failed close records the batch FAILED. With a
filtermanager, every filter column must be an entity column (a segment
closes only its own entities), else nothing is registered and None is
returned. False (default): registration unchanged, and an option recorded
by an earlier registration of the same view is switched off.
tdfs4ds.set_process_options changes it without re-registering.
Returns¶
teradataml DataFrame or str or None
By default the dataset view (None unless tdfs4ds.BUILD_DATASET_AT_UPLOAD
is True); the PROCESS_ID string when return_process_id=True; None on
a recoverable input error, with a prescriptive message logged.
Upload features from a tdstone2 model's predictions to the feature store.
Handles the full workflow: extract feature names/types, register in the feature catalog, prepare for ingestion, store in the feature store, and build a dataset view.
Parameters: - model: The tdstone2 model object whose predictions contain features to upload.
Precondition — the model must expose one of two output accessors. This function
dispatches on whether the model has a score member: if it does, it reads
model.get_model_predictions(); if it does not, it reads
model.get_computed_features(). A model class exposing neither is not
supported by this bridge and will fail with an AttributeError. The entity is
likewise taken purely from the model's mapper (tdstone2_entity_id reads
mapper_scoring.id_row / mapper.id_partition / mapper.id_row) — tdfs4ds
injects no origin/as-of column, so a contract keyed on an origin date only gets one
if the model's own mapper already carries it. For a model that does not fit this
shape, register a permanent view over its output table with upload_features
instead — that path has no such requirement.
Returns: - DataFrame: A DataFrame representing the dataset view created in the feature store.
The TimeManager that drives process_list when roll_out is given none (0.3.5.15).
Read from the managers attached to the processes (process catalog TIME_*). They must
share one table of dates: one manager, or a manager and its soft clones (whose cursors
roll_out then moves with it). Raises ValueError naming what does not fit.
Refuse a roll-out whose first date is before the stored values of a process (0.3.5.16),
or, with rebuild_history, delete those values first. Returns the force_compute to use.
roll_out(process_list=None, time_manager=None, time_id_start=1, time_id_end=None, force_compute=False, force_display_logs=False, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, close_absent_entities=None, track_entity_refresh=None, dataset_time_manager=None, rebuild_history=False, *, data_domain=None, order='dependencies')
Execute a series of processes for each date in a time manager, managing time, computation settings, and logging.
Iterates over a range of time steps, updating the TimeManager with each step, and executes all processes in process_list for that time step.
Parameters:
- process_list (list): List of process IDs to execute for each time step.
- time_manager (TimeManager, optional): the manager whose dates are rolled out.
None (0.3.5.15) = the TimeManager attached to the processes; they must share one
table of dates (one manager, or a manager and its soft clones), else ValueError.
- time_id_start (int, optional): Starting time step ID. Default is 1.
- time_id_end (int, optional): Ending time step ID. If None, runs until last step.
- force_compute (bool, optional): Force recomputation even if results exist. Default False.
- force_display_logs (bool, optional): Force log display during rollout. Default False.
Each process is first checked against the lineage graph, before any step runs:
- its view reads the manager: fine;
- its view reads a soft clone of the manager (same table of dates, own cursor): at each step the clone's cursor is moved to the same step before the processes run, so the values stay consistent (0.3.5.15); its own pairing is kept;
- its view reads another manager (other dates):
ValueError, also underTIME_MANAGER_CHECK='warn'(0.3.5.15; it used to warn and write values stamped with this manager's dates but computed at the other one's cursor); -
its view reads no manager: a warning under
'warn'(default; the pairing is not recorded), aValueErrorunder'error'.'off'skips the check (and the alignment). -
close_absent_entities (bool, optional):
None(default) applies each process's recorded option;True/Falseoverride it for this roll-out. With it on, each date ends the validity of the entities a process no longer produces at that date; a later date where they come back opens a new period. Replay a history in date order (from its first date). - track_entity_refresh (bool, optional):
None(default) applies each process's recorded option;True/Falseoverride it for this roll-out. With it on, each date records which entities each process confirmed (tdfs4ds.entity_refresh). - rebuild_history (bool, optional): before step 1,
roll_outreads where each process's stored values start (0.3.5.16). A run writes from its date onward but never before the first value already stored, so when they start after the first date rolled out (a first run in real time, or at a later date), the roll-out would add nothing: it is refused withValueErrorand nothing runs.Truedeletes the stored values of those processes from the first date rolled out onward (logged inFS_REMOVAL_LOG) and replays every date withforce_compute=True; with nothing to delete it only forces the recomputation. A process whose valid time comes from a source column is not checked; a start that cannot be read is a warning and the roll-out goes on. - on_failure / raise_on_failure: what a failed FilterManager segment does in each run
('continue' default, 'stop', 'raise' = attempt every segment then raise
FeatureIngestionError;raise_on_failure=Trueis the shorthand). A raised failure stops the roll-out at that date. - check_upstream / strict_upstream: refuse a process when an upstream process (lineage graph) FAILED at that date / is also RUNNING.
- dataset_time_manager (bool, optional):
None(default) applies each process's recorded setting;True/Falseoverride it: the dataset a run builds follows the process's TimeManager. - data_domain (str or True, keyword, 0.3.6.0): roll out every process of a data domain
(
True=tdfs4ds.DATA_DOMAIN; primary store only), driven by the domain clock (set_domain_time_manager) whentime_manageris not given.process_list=Nonemeans the processes oftdfs4ds.DATA_DOMAIN. Passing both a list anddata_domainis aValueError. - order (keyword, 0.3.6.0):
'dependencies'(default) runs the processes of each date in dependency order, from the lineage graph; the sort is stable, so a list already in a valid order is unchanged.'given'keeps the list's order (the behaviour before 0.3.6.0).
str, optional
A comment to attach to the created view for documentation purposes. Defaults to None.