tdfs4ds.dataset.builder¶
Dataset construction functions for the tdfs4ds feature store.
Provides build_dataset(), build_dataset_opt(), and augment_source_with_features() for generating Teradata views that join feature tables with source data.
property
¶
True when the build table and its view exist (checked once, then cached).
property
¶
True when the window table and its view exist (checked once, then cached).
property
¶
True when the offsets table and its view exist (checked once, then cached).
__init__(schema_name=None, name='DATASET', existing_objects=None, schema_name_view=None, existing_view_objects=None)
Parameters¶
existing_objects : set, optional
A pre-computed set of lower-cased table/view names already present in
schema_name. When provided, the constructor skips its own
db_list_tables() call (saves one round-trip). Pass the set
returned by connect()'s DBC.TablesV query.
schema_name_view : str, optional
Database that holds the catalog VIEW objects. When None it defaults to
tdfs4ds.SCHEMA_VIEW if a dedicated view database is configured, else
to schema_name (co-located with the tables — byte-identical to the
historical single-database behaviour).
existing_view_objects : set, optional
Pre-computed set of lower-cased view names present in schema_name_view.
Only consulted when the view database differs from the table database.
Create whatever part of the catalog is missing. Idempotent.
Creates only the absent objects and leaves the existing ones alone, returning the list of names it created (empty when there was nothing to do). Creation order is preserved, so a table is always created before the view that reads it.
It used to refuse — logging an ERROR and raising — as soon as any
of the six objects existed, while _exists only reported True when
all six did. Those two predicates disagree on every partial catalog,
which left it permanently unrepairable: the constructor sees "not all
present", calls this, and this rejects "some present". Because the
constructor self-heals on every instantiation, that fired on ordinary
READ paths (dataset_catalog(), the agent's _resolve_dataset) and
on connect()/setup() — an ERROR-level log and an exception for a
condition the caller cannot act on. Self-healing is this constructor's
long-standing contract; it was simply incomplete.
Register (or update) dataset in the catalog. Returns its DATASET_ID.
Parameters¶
build : dict, optional
How the view was built — {'strategy', 'join_type', 'time'
(DatasetTime), 'source_dataset_id'} — written to the build table
when it exists. Omitted: the build table is left untouched.
dataset_id : str, optional
Force the DATASET_ID to write. Used by restore_dataset_catalog
when a dataset was registered several times and one id must be kept;
ordinary callers let it be looked up by name.
offsets : list of dict, optional
One row per column read at an offset (lags and leads), written to the
offsets table when it exists; [] closes the rows of an earlier
build. None (default) leaves the table untouched.
Build details (strategy, join type, temporal mode) — None when unavailable.
Offset columns (one row per lag / lead column) — None when unavailable.
Join it to get_dataset_features on (DATASET_ID, FEATURE_NAME): a
column without offset has no row.
History windows (one row per historical dataset) — None when unavailable.
Join it to the catalog on DATASET_ID with a LEFT OUTER JOIN: a
snapshot dataset has no row. WINDOW_START / WINDOW_END are the
bounds at query time (WINDOW_START NULL = from the earliest stored
value); the other columns are the definition.
Remove a dataset (catalog + entity + feature rows) by DATASET_ID.
Parameters¶
dataset_id : str
The DATASET_ID to remove.
purge : bool, optional
When False (default), a CURRENT VALIDTIME DELETE valid-time-closes
the currently-valid rows (temporal soft delete; history is retained).
When True, a NONSEQUENCED VALIDTIME DELETE physically removes every
temporal row for the dataset — use this to fully reset a dataset that
carries stale/closed history before re-registering it.
close_lineage : bool, optional
True (default): the dataset view's edges in FS_LINEAGE_GRAPH are
closed as well. False when the view itself stays (a duplicate
registration removed by restore_dataset_catalog).
Raises¶
ValueError
If no dataset with this dataset_id currently exists in the catalog.
instance-attribute
¶
Parameters¶
entity, features : optional
The AUTHORITATIVE entity/feature split, supplied by build_dataset
(which knows it at build time). When BOTH are provided they are used
verbatim and the fragile DDL-regex re-derivation
(_retrieve_entities_and_features) is skipped. When omitted (e.g.
an ad-hoc Dataset(view_name=...) for inspection) the regex path is
used as before — preserving backward compatibility.
entity is {col: sql_type}; features is
{NAME: {id, version, type, database, view}}.
valid_time : str, optional
The governing temporal qualifier, supplied by build_dataset which
chose it. When omitted it is read back from the view's DDL.
Everything needed to (re)generate a dataset view.
__init__(entity, features, join_type='INNER', strategy=MULTIJOIN, time=None, schema_name=None, view_name=None, offset_strategy=None)
Column names of a dataset, in the order they are asked for.
<feature>, or <feature>_<process view> when several versions are
requested, or the version's own alias when given; then _<offset
suffix> (_LAG_1D), then _2, _3... for a repeat of a generated
name. A given alias is never renamed: two equal aliases are refused.
_resolve_feature_specs and preview_column_names both use it.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Read a DatasetSpec back out of a dataset view's DDL.
Understands both strategies build_dataset emits (multi-join and grouped),
both join types and the three temporal modes, including views written by
earlier tdfs4ds versions (the generated shape has not changed). Returns
None when the text is not a tdfs4ds dataset view — e.g. a view from
augment_source_with_features or any user view — rather than guessing.
Features come back in the view's own column order; the entity is every other output column.
The REPLACE VIEW statement for spec. None when it has no feature.
resolve_dataset_time(feature_store_time=False, as_of=None, time_manager=None, current_feature_store_time=None) -> DatasetTime
Resolve build_dataset's temporal arguments into one DatasetTime.
- nothing set →
CURRENT(the default, unchanged). as_of=<literal>→AS_OFthat literal, normalised.time_manager=<TimeManager or 'db.view'>→TIME_MANAGER.feature_store_time=True(historical flag) →AS_OFtdfs4ds.FEATURE_STORE_TIMEwhen it is set, elseCURRENT— exactly as before. Its value is now normalised too, which leaves a well-formed timestamp untouched and turns a bare date (rejected by Teradata) into midnight.
The three are mutually exclusive: asking for two different points in time is a mistake to report, not to arbitrate silently.
get_available_entity_id_records(entity_id, selected_features, other=None, time_column=None, filtermanager=None, filter_conditions=None)
Generate a SQL query to retrieve records for available entity IDs based on selected features.
Parameters: entity_id (list, dict, or str): The entity IDs for which records are needed. Can be a list, dictionary, or string. selected_features (dict): A dictionary of selected features with their versions. other (optional): An additional DataFrame object for more complex queries. Defaults to None. time_column (str, optional): The name of the time column to use for temporal conditions. Defaults to None.
tuple: A tuple containing: - str: A SQL query string to retrieve records for available entity IDs. - dict: A dictionary of the list of features and their locations.
Retrieve the location of selected features for given entity IDs.
Parameters: entity_id (list, dict, or str): The entity IDs for which feature locations are needed. Can be a list, dictionary, or string. selected_features (dict): A dictionary of selected features with their versions.
Returns: DataFrame: A pandas DataFrame containing the feature locations.
Normalise entity_id (list / dict / str) to a sorted list of column names.
Compute the AUTHORITATIVE entity/feature split a dataset view will have,
using exactly the column aliases _build_dataset_multijoin generates.
This is what build_dataset already knows at build time — the role of a
column (entity vs feature) is dataset-specific and must NOT be re-derived
from the global catalog (the same name can be a feature in one process and
an entity in another) nor from fragile DDL regex.
Returns (entity_cols, feature_map) where:
* entity_cols — list of entity column names (the dataset grain),
* feature_map — {ALIAS_UPPER: {id, version, database, view}} keyed
by the feature's final column alias in the view (uppercased). type is
filled in later from the built view's column types.
{ALIAS_UPPER: {id, version, database, view}} for registration.
_dataset_with_known_split(view_name, schema_name, entity_id, selected_features, valid_time=None, feature_offsets=None, specs=None)
Construct a Dataset using the build-time entity/feature split.
Types are sourced from the built view's columns (authoritative for what the
view actually emits). Falls back to the regex-based Dataset(view_name=...)
if the split can't be computed, so registration never hard-fails. specs:
the build's own feature specs (nothing is resolved again).
A Dataset whose entity/feature split comes from a parsed DatasetSpec.
Types are read from the view itself (what it actually emits). Unlike
_dataset_with_known_split nothing is re-resolved against the feature
catalog: the view text says which table, id and version each column reads.
The DatasetSpec of an existing dataset view, read from its own DDL.
view_name may be 'database.view'. Returns None (with an error
logged) when the view is missing or is not a tdfs4ds dataset view.
(database, view) of a dataset to create; view_name may be 'database.view'.
upload_features documents dataset_view_name as 'database.view', but the
builders took the whole string as the view name: the view was created as
DB."DB.view" (one name containing a dot) and registered as DB.DB.view.
A database given twice and differently is refused rather than guessed. With
no database at all, the view database (tdfs4ds.view_schema()) is used:
a dataset is a view, and its feature sources are fully qualified anyway.
Write the dataset's lineage subgraph to FS_LINEAGE_GRAPH. Never raises.
Without it a dataset appears in the lineage graph only after a full
rebuild_lineage_graph(). The write is diff-based, so re-recording an
unchanged dataset touches no row.
rebuild_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, grouped=None, join_type=None, as_of=None, time_manager=None, current=False, replace=False, comment=None, return_query=False, start=None, end=None, duration=None, delay=None, history=False)
Rebuild an existing dataset with another strategy or at another point in time.
The source view's own DDL is the specification: the same storage tables,
feature ids, process versions and column names are kept. Nothing is
re-resolved against the feature catalog, so the result does not depend on
the session's DATA_DOMAIN nor on which versions are registered today.
Parameters¶
view_name, schema_name :
The source dataset (view_name may be 'database.view').
target_view_name, target_schema_name :
Where to write the result. Default: the source itself, which requires
replace=True.
grouped : bool, optional
True → one pivoted sub-query per (storage table, version);
False → one sub-query per feature. None keeps the source's.
join_type : {'INNER', 'FULL OUTER'}, optional
None keeps the source's.
as_of, time_manager, current :
New temporal mode — a literal point in time, a TimeManager, or
current=True for CURRENT VALIDTIME. None of them keeps the
source's mode. Given on a historical dataset, they turn it back into a
snapshot.
start, end, duration, delay, history :
Rebuild as a HISTORICAL dataset over that window (see
build_historical_dataset); history=True alone means the full
history. On a historical source they replace its window.
replace : bool
Allow overwriting a view that already exists. Without it the call
refuses (REPLACE VIEW would silently overwrite it).
return_query : bool
Return the DDL instead of executing it.
Returns the new dataset as a teradataml DataFrame (or the DDL), None on
a recoverable error (logged with the fix).
build_grouped_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, replace=False, comment=None, return_query=False)
Build the grouped version of an existing (non-grouped) dataset.
The grouped strategy reads each storage table once per process version —
features sharing a table are pivoted with MAX(CASE WHEN …) — instead of
once per feature, so a wide dataset needs far fewer joins. Same rows, same
columns, same temporal mode as the source.
target_view_name defaults to <view_name>_GROUPED (a new dataset, the
source is left untouched). Pass the source's own name with replace=True
to convert it in place.
(items, several): the versions asked for one feature, as dicts.
'<pid>' (one version), {'process_id': pid, 'alias': ...} (one version
with its own column name) or a list of {'process_id', 'process_view_name',
'alias'} (several versions side by side; alias optional).
The view name alone: the catalog stores process views qualified and often
quoted ("DB"."V"), which made <feature>__DB___V_ columns. The app's
builder already used the bare name (panels.dataset_versions_columns).
{feature: [column names]} a dataset will have, without reading the catalog.
selection per feature: None (one version), {'alias': ...} (one
version with its own name), or a list of several versions side by side, each
a process view name or {'view', 'alias'}. feature_offsets as in
build_dataset. The admin app's builder shows these names while the form
is filled in. Raises ValueError on an invalid or repeated alias.
Resolve selected_features into (entity_cols, [FeatureSpec]).
The single place where column aliases are minted: a plain feature keeps its
name, a feature requested under several process versions gets
<feature>_<process view>, a feature read at offsets gets one column per
offset (<name>_LAG_1D, <name>_LEAD_1MO; '0' keeps the plain name),
and a repeated alias gets _2, _3… (compute_dataset_split uses
these specs for registration). locations: already resolved by
selection.resolve_for_build (else read with get_feature_location).
The builders' historical feature_store_time flag, unless a mode is given.
The strategy the view text really has (what restore_dataset_catalog will read).
It differs from the one requested only with offsets: a grouped request where each offset feature is alone in its table has nothing to group.
(grouped, offset_pivot) the features' value types allow (see
tdfs4ds.feature_store.value_types.fit_strategy). Reads nothing for a plain multi-join
dataset without lag/lead pivots.
Rows for the dataset catalog's optional offsets table: one per offset column.
_build_dataset_ddl(entity_id, selected_features, view_name, schema_name, join_type, strategy, dataset_time, feature_offsets=None, offset_strategy=None, specs=None)
_build_dataset_multijoin(entity_id, selected_features, view_name, schema_name, feature_store_time, join_type, dataset_time=None, feature_offsets=None, offset_strategy=None, specs=None)
Build the REPLACE VIEW DDL using one sub-query per feature (classic approach).
Each feature gets its own sub-query; the outer query chains them with JOINs. Teradata pushes a PI-equality filter on the entity column directly into every sub-query, making single-entity lookups single-AMP.
Returns the full DDL string, or None when no features are available.
_build_dataset_grouped(entity_id, selected_features, view_name, schema_name, feature_store_time, join_type, dataset_time=None, feature_offsets=None, offset_strategy=None, specs=None)
Build the REPLACE VIEW DDL using one sub-query per (table, version) group.
Features sharing the same storage table and process version are pivoted into a single sub-query with MAX(CASE WHEN FEATURE_ID=X THEN FEATURE_VALUE END) per feature and GROUP BY on the entity columns. This reduces the number of JOINs from N_features down to N_distinct_(table, version) pairs.
For INNER JOIN a HAVING clause keeps only entities that carry a row for every feature in the group, the same semantics as the per-feature approach (a row whose stored value is NULL counts, as it does there). For FULL OUTER no HAVING is added so partial entities survive with NULLs.
PI-equality filters on the entity column are pushed through the GROUP BY into the base-table scan when the outer WHERE references the entity column directly. For INNER JOIN the outer SELECT uses A1.entity (enables push-down); for FULL OUTER it uses COALESCE (push-down is not possible, same as classic approach).
Returns the full DDL string, or None when no features are available.
build_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, feature_store_time=False, join_type='INNER', grouped=False, document=False, vector_indexing=False, record_lineage=True, as_of=None, time_manager=None, feature_offsets=None, offset_strategy=None)
Generate a SQL query and create a Teradata view to retrieve records for the specified entity IDs based on selected features.
Parameters:¶
entity_id : list, dict, or str The entity IDs for which records are needed. Can be a list, dictionary, or string. - If list: contains multiple entity IDs. - If dict: keys are the entity IDs. - If str: a single entity ID.
list, str or dict
The feature names alone (['amount', 'nb_tx']) when each feature is
computed by one process: its version is found in the catalog. A version is
needed only for a feature several processes compute; write it in a dict,
with None for the others ({'amount': '<UUID>', 'nb_tx': None}).
Such a feature named without a version raises FeatureSelectionError
(a ValueError) naming its processes and the dict to write; so does a
name not registered for this entity. All features are resolved with one
catalog query. In the dict, version is:
- None: the feature's only version,
- {"alias": <column name>}: the only version under a name of its own,
- a process UUID string (the column takes the feature's name),
- {"process_id": <UUID>, "alias": <column name>}: one version under a name of its own, or
- a list of {"process_id": <UUID>, "process_view_name": <str>, "alias": <str, optional>}:
several versions side by side, one column each, named <feature>_<process view>
unless an alias is given.
An alias is a plain identifier (letters, digits, _) and may not repeat another column;
offsets (feature_offsets) are suffixed to it (spend_LAG_1D).
str
The name of the view to be created in the database.
str, optional
The name of the database schema where the view will be created.
Defaults to the global tdfs4ds.SCHEMA if not provided.
bool, optional
If True, the function will return the SQL query string instead of executing it. Defaults to False.
bool, optional
If True, the query will use a specific VALIDTIME based on tdfs4ds.FEATURE_STORE_TIME.
If False, the query will use CURRENT VALIDTIME. Defaults to False.
str, datetime, date or pandas.Timestamp, optional
Freeze the dataset at this point in time (VALIDTIME AS OF TIMESTAMP).
Any ISO date or timestamp is accepted — '2024-03-31',
'2024-03-31 23:59:59', '2024-03-31T23:59:59+02:00' — and
normalised to the literal Teradata needs (a bare date becomes midnight;
Teradata rejects a date-only TIMESTAMP literal). Without an offset the
value is read in the session time zone, like FEATURE_STORE_TIME.
TimeManager or str, optional
Make the dataset follow a TimeManager: every feature is read as of
the manager's current BUSINESS_DATE, so time_manager.update(i)
moves the dataset without rebuilding it. A 'database.view' string
naming the manager's public view is accepted too.
as_of, time_manager and feature_store_time=True are mutually
exclusive. With none of them the dataset reads CURRENT VALIDTIME (the
default, unchanged). The chosen mode is recorded in the dataset catalog:
DATASET_VALIDTIME holds the qualifier and the build table
(FS_V_<catalog>_BUILD) holds the mode, timestamp or TimeManager.
str, optional
Specifies the type of join to use between features: 'INNER' (default) or 'FULL OUTER'.
bool, optional
If False (default), uses one sub-query per feature joined together — the classic multi-join approach. Every individual feature sub-query can be targeted by a PI-equality filter pushed from the outer WHERE, making single-entity lookups single-AMP. If True, groups features that share the same storage table and process version into a single sub-query with MAX(CASE WHEN …) pivoting, reducing the number of JOINs from N_features to N_distinct_(table, version) pairs. A HAVING clause enforces non-NULL for every feature in the group when join_type='INNER', preserving the same missing-feature semantics as the classic approach. PI push-down is preserved for INNER JOIN (outer SELECT uses A1.entity); for FULL OUTER the COALESCE in the outer SELECT prevents push-down, same as the classic approach.
bool, optional
If True, generate and persist business-dictionary documentation for the
newly built dataset via document_dataset_incremental (LLM-powered).
Requires tdfs4ds.INSTRUCT_MODEL_PROVIDER and tdfs4ds.INSTRUCT_MODEL_MODEL
to be configured; silently skipped (with an info log) otherwise. Failures
are non-fatal — the dataset view is always created/returned regardless.
Defaults to False (opt-in). Mirrors upload_features. Leave False when
the dataset is built via upload_features/run() — that path already
documents it, so enabling it here would double-document.
bool, optional
If True (and document=True and an instruct model is configured), refresh
the consumer-agent vector index via build_vector_index(force_rebuild=False)
after documentation so the new dataset is searchable immediately. Non-fatal.
Defaults to False (opt-in).
bool, optional
Persist the dataset view's lineage subgraph (dataset -> process views ->
sources) into FS_LINEAGE_GRAPH via record_process_lineage. No LLM is
involved and the write is diff-based (an unchanged graph writes nothing).
Non-fatal. Defaults to True since 0.3.1.220: with the former opt-in
default no dataset reached the lineage graph unless
rebuild_lineage_graph() was run afterwards. Pass False to skip it.
dict, optional
Read features at other instants than the dataset's own (lags and leads),
e.g. {'amount': ['0', '-1 day', '-2 days'], 'churn': ['+1 month']}:
'-1 day'is the value one day BEFORE the dataset's instant (a lag, for autoregressive inputs),'+1 month'one month AFTER (a lead, e.g. a forecast target),'0'the instant itself.- A listed feature gets one column per offset (
amount,amount_LAG_1D,amount_LAG_2D,churn_LEAD_1MO); it keeps its plain column only when'0'is listed. - The instant is the dataset's: now,
as_of, or the TimeManager's business date (time_manager.update()moves every offset column). - A lead is NULL while its instant is later than the database's current
time: what is valid after now is not known yet. Under
CURRENTa lead is therefore always NULL (refused withjoin_type='INNER'). - Units: year, month, week, day, hour, minute, second; months are
calendar months (
ADD_MONTHS). No fraction of a second: valid times are stored to the second.
Without it the SQL is exactly what it was. See the user guide, "Lags and leads".
{'PIVOT', 'PER_OFFSET'}, optional
How the offset columns are read. 'PIVOT' (default) reads each
storage table (and process version) once for all its offset columns:
one aggregation step, local to the AMPs, partition elimination on the
FEATURE_IDs. 'PER_OFFSET' gives each offset column its own sub-query
and join; measured on Vantage 20 it parses far slower (32 columns: 31 s
against 0.35 s) and is kept for comparison.
Returns:¶
str or tdml.DataFrame
- If return_query is True, returns the generated SQL query as a string.
- Otherwise, returns a Teradata DataFrame containing the results.
Raises:¶
ValueError
If an invalid join_type is provided.
build_historical_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, join_type='INNER', start=None, end=None, duration=None, delay=None, record_lineage=True, document=False, vector_indexing=False, process_version=False, feature_offsets=None)
Create a HISTORICAL dataset view: one row per entity per period of constant values.
Where build_dataset returns a snapshot (each entity's feature values at
one valid instant), this returns their history over a window::
<entity columns>, ValidPeriod, ValidStart, ValidEnd, <feature columns>
Each row says: from ValidStart (included) to ValidEnd (excluded),
this entity had exactly these feature values. Consecutive rows of an
entity always differ in at least one value (NORMALIZE ON MEETS OR
OVERLAPS merges equal neighbours, should the stored history hold any). ValidPeriod is the same interval as a Teradata
PERIOD(TIMESTAMP(0) WITH TIME ZONE), for CONTAINS / OVERLAPS.
The result is a plain view: nothing is materialised, new ingestions show up immediately, and a relative window moves by itself.
Parameters¶
entity_id, selected_features, view_name, schema_name, comment, return_query :
As in build_dataset (feature names alone when each has one version).
join_type : {'INNER', 'FULL OUTER'}
'INNER' keeps a period only where EVERY feature has a value — an
entity's history starts when its last feature appears, and an entity
whose features never overlap is absent. 'FULL OUTER' keeps every
period where AT LEAST ONE feature has a value, with NULLs elsewhere.
Either way, at any instant t of the window the history equals
build_dataset(..., as_of=t, join_type=...).
Example — feature A from January (10, then 20 from July), feature B
only from April::
FULL OUTER: [Jan, Apr) A=10 B=NULL | [Apr, Jul) 10 x | [Jul, ...) 20 x
INNER: [Apr, Jul) 10 x | [Jul, ...) 20 x
start, end : str, date, datetime or pandas.Timestamp, optional
Fixed window bounds (ISO strings accepted, a bare date is midnight).
The window is half-open [start, end): end='2024-04-01' covers
all of March. Without end the window ends at now.
duration : str, tuple or timedelta, optional
Window length: '3 months', '90 days', '12 hours',
'1 year 2 months', ISO 8601 'P3M', (3, 'months') or a
timedelta. Combined with start → [start, start + duration);
with end → [end - duration, end); alone → [now - duration, now).
Months use ADD_MONTHS (31 May minus 3 months is 29 Feb).
delay : same forms as duration, optional
Shift now back: duration='12 months', delay='1 day' is
[now - 1 day - 12 months, now - 1 day). Not combinable with
start/end.
``now`` is the current valid time when the view is QUERIED, so any
window without a fixed end slides with time. No argument at all means
the full history up to now. A period still open in the store therefore
ends at the window's end, never at 9999-12-31.
record_lineage, document, vector_indexing :
As in build_dataset.
process_version : bool, default False
Also create <view_name>_PV: the same history with a
<feature>_PROCESS_VERSION column per feature (the version of the
process SQL that wrote each period, NULL before versioning). The dataset
view itself is never changed. Whatever this flag, a history that spans
several process versions is reported in the log
(tdfs4ds.history_versions).
feature_offsets : dict, optional
Lags and leads, as in build_dataset: each instant of the history
reads amount_LAG_1D one day earlier, churn_LEAD_1MO one month
later (NULL until that date is reached). A history with offsets is
computed with elementary intervals for both join types (a shifted period
cannot go through the native sequenced join). Not combinable with
process_version=True yet.
Returns¶
tdml.DataFrame on the new view, the DDL when return_query=True, or
None when no feature could be resolved.
Notes¶
INNER uses Teradata's native SEQUENCED VALIDTIME join (each storage
table read once, rowkey merge joins, the window folded into the join).
FULL OUTER cuts each entity's timeline at every change point and attaches
each feature on containment; it reads each storage table more than once.
See tdfs4ds.dataset.history.
Optional governance steps after a dataset view is registered. Never raises.
Mirrors the upload_features post-build block. Order: lineage -> documentation -> vector index. Lineage is on by default (no LLM, diff-based write). When build_dataset is called internally from upload_features/run()/the consumer agent, document/vector_indexing stay False so the dataset is NOT double-documented (those paths own their single doc pass).
build_dataset_opt(entity_id, selected_features, view_name=None, schema_name=None, comment='dataset', no_temporal=False, time_manager=None, query_only=False, entity_null_substitute={}, other=None, time_column=None, filtermanager=None, filter_conditions=None)
Experimental: a dataset view built by augment_source_with_features (one
LEFT JOIN per feature onto the entity list), named view_name in schema_name
(default the view database). Only entity_id, selected_features, view_name
and schema_name are used; the other arguments are accepted for the signature of
build_dataset and ignored. Prefer build_dataset.
augment_source_with_features(source_schema, source_name, entity_id, feature_selection, output_view_schema, output_view_name, join_type='LEFT JOIN')
Augment a source table with selected features by creating a view that joins the source data with feature tables based on specified entity identifiers and feature selections.
The join type can be specified as either 'LEFT JOIN' or 'INNER JOIN': - 'LEFT JOIN': Retains all rows from the source table, with missing values for unmatched features. - 'INNER JOIN': Returns only rows where all specified features have matching, non-missing values.
Parameters: source_schema (str): The schema of the source table to augment. source_name (str): The name of the source table to augment. entity_id (str or list of str): The entity ID(s) for joining the source data with feature tables. feature_selection (list): A list of features to select and join with the source data. output_view_schema (str): The schema in which to create the output view. output_view_name (str): The name of the output view to create with augmented features. join_type (str, optional): 'LEFT JOIN' or 'INNER JOIN'. Defaults to 'LEFT JOIN'.
Returns: tdml.DataFrame: A DataFrame object that references the newly created view.
str, optional
A comment to attach to the created view for documentation purposes. Defaults to None.