tdfs4ds.dataset¶
build_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, feature_store_time=False, join_type='INNER', grouped=False, document=False, vector_indexing=False, record_lineage=True, as_of=None, time_manager=None, feature_offsets=None, offset_strategy=None)
Generate a SQL query and create a Teradata view to retrieve records for the specified entity IDs based on selected features.
Parameters:¶
entity_id : list, dict, or str The entity IDs for which records are needed. Can be a list, dictionary, or string. - If list: contains multiple entity IDs. - If dict: keys are the entity IDs. - If str: a single entity ID.
list, str or dict
The feature names alone (['amount', 'nb_tx']) when each feature is
computed by one process: its version is found in the catalog. A version is
needed only for a feature several processes compute; write it in a dict,
with None for the others ({'amount': '<UUID>', 'nb_tx': None}).
Such a feature named without a version raises FeatureSelectionError
(a ValueError) naming its processes and the dict to write; so does a
name not registered for this entity. All features are resolved with one
catalog query. In the dict, version is:
- None: the feature's only version,
- {"alias": <column name>}: the only version under a name of its own,
- a process UUID string (the column takes the feature's name),
- {"process_id": <UUID>, "alias": <column name>}: one version under a name of its own, or
- a list of {"process_id": <UUID>, "process_view_name": <str>, "alias": <str, optional>}:
several versions side by side, one column each, named <feature>_<process view>
unless an alias is given.
An alias is a plain identifier (letters, digits, _) and may not repeat another column;
offsets (feature_offsets) are suffixed to it (spend_LAG_1D).
str
The name of the view to be created in the database.
str, optional
The name of the database schema where the view will be created.
Defaults to the global tdfs4ds.SCHEMA if not provided.
bool, optional
If True, the function will return the SQL query string instead of executing it. Defaults to False.
bool, optional
If True, the query will use a specific VALIDTIME based on tdfs4ds.FEATURE_STORE_TIME.
If False, the query will use CURRENT VALIDTIME. Defaults to False.
str, datetime, date or pandas.Timestamp, optional
Freeze the dataset at this point in time (VALIDTIME AS OF TIMESTAMP).
Any ISO date or timestamp is accepted — '2024-03-31',
'2024-03-31 23:59:59', '2024-03-31T23:59:59+02:00' — and
normalised to the literal Teradata needs (a bare date becomes midnight;
Teradata rejects a date-only TIMESTAMP literal). Without an offset the
value is read in the session time zone, like FEATURE_STORE_TIME.
TimeManager or str, optional
Make the dataset follow a TimeManager: every feature is read as of
the manager's current BUSINESS_DATE, so time_manager.update(i)
moves the dataset without rebuilding it. A 'database.view' string
naming the manager's public view is accepted too.
as_of, time_manager and feature_store_time=True are mutually
exclusive. With none of them the dataset reads CURRENT VALIDTIME (the
default, unchanged). The chosen mode is recorded in the dataset catalog:
DATASET_VALIDTIME holds the qualifier and the build table
(FS_V_<catalog>_BUILD) holds the mode, timestamp or TimeManager.
str, optional
Specifies the type of join to use between features: 'INNER' (default) or 'FULL OUTER'.
bool, optional
If False (default), uses one sub-query per feature joined together — the classic multi-join approach. Every individual feature sub-query can be targeted by a PI-equality filter pushed from the outer WHERE, making single-entity lookups single-AMP. If True, groups features that share the same storage table and process version into a single sub-query with MAX(CASE WHEN …) pivoting, reducing the number of JOINs from N_features to N_distinct_(table, version) pairs. A HAVING clause enforces non-NULL for every feature in the group when join_type='INNER', preserving the same missing-feature semantics as the classic approach. PI push-down is preserved for INNER JOIN (outer SELECT uses A1.entity); for FULL OUTER the COALESCE in the outer SELECT prevents push-down, same as the classic approach.
bool, optional
If True, generate and persist business-dictionary documentation for the
newly built dataset via document_dataset_incremental (LLM-powered).
Requires tdfs4ds.INSTRUCT_MODEL_PROVIDER and tdfs4ds.INSTRUCT_MODEL_MODEL
to be configured; silently skipped (with an info log) otherwise. Failures
are non-fatal — the dataset view is always created/returned regardless.
Defaults to False (opt-in). Mirrors upload_features. Leave False when
the dataset is built via upload_features/run() — that path already
documents it, so enabling it here would double-document.
bool, optional
If True (and document=True and an instruct model is configured), refresh
the consumer-agent vector index via build_vector_index(force_rebuild=False)
after documentation so the new dataset is searchable immediately. Non-fatal.
Defaults to False (opt-in).
bool, optional
Persist the dataset view's lineage subgraph (dataset -> process views ->
sources) into FS_LINEAGE_GRAPH via record_process_lineage. No LLM is
involved and the write is diff-based (an unchanged graph writes nothing).
Non-fatal. Defaults to True since 0.3.1.220: with the former opt-in
default no dataset reached the lineage graph unless
rebuild_lineage_graph() was run afterwards. Pass False to skip it.
dict, optional
Read features at other instants than the dataset's own (lags and leads),
e.g. {'amount': ['0', '-1 day', '-2 days'], 'churn': ['+1 month']}:
'-1 day'is the value one day BEFORE the dataset's instant (a lag, for autoregressive inputs),'+1 month'one month AFTER (a lead, e.g. a forecast target),'0'the instant itself.- A listed feature gets one column per offset (
amount,amount_LAG_1D,amount_LAG_2D,churn_LEAD_1MO); it keeps its plain column only when'0'is listed. - The instant is the dataset's: now,
as_of, or the TimeManager's business date (time_manager.update()moves every offset column). - A lead is NULL while its instant is later than the database's current
time: what is valid after now is not known yet. Under
CURRENTa lead is therefore always NULL (refused withjoin_type='INNER'). - Units: year, month, week, day, hour, minute, second; months are
calendar months (
ADD_MONTHS). No fraction of a second: valid times are stored to the second.
Without it the SQL is exactly what it was. See the user guide, "Lags and leads".
{'PIVOT', 'PER_OFFSET'}, optional
How the offset columns are read. 'PIVOT' (default) reads each
storage table (and process version) once for all its offset columns:
one aggregation step, local to the AMPs, partition elimination on the
FEATURE_IDs. 'PER_OFFSET' gives each offset column its own sub-query
and join; measured on Vantage 20 it parses far slower (32 columns: 31 s
against 0.35 s) and is kept for comparison.
Returns:¶
str or tdml.DataFrame
- If return_query is True, returns the generated SQL query as a string.
- Otherwise, returns a Teradata DataFrame containing the results.
Raises:¶
ValueError
If an invalid join_type is provided.
build_dataset_opt(entity_id, selected_features, view_name=None, schema_name=None, comment='dataset', no_temporal=False, time_manager=None, query_only=False, entity_null_substitute={}, other=None, time_column=None, filtermanager=None, filter_conditions=None)
Experimental: a dataset view built by augment_source_with_features (one
LEFT JOIN per feature onto the entity list), named view_name in schema_name
(default the view database). Only entity_id, selected_features, view_name
and schema_name are used; the other arguments are accepted for the signature of
build_dataset and ignored. Prefer build_dataset.
augment_source_with_features(source_schema, source_name, entity_id, feature_selection, output_view_schema, output_view_name, join_type='LEFT JOIN')
Augment a source table with selected features by creating a view that joins the source data with feature tables based on specified entity identifiers and feature selections.
The join type can be specified as either 'LEFT JOIN' or 'INNER JOIN': - 'LEFT JOIN': Retains all rows from the source table, with missing values for unmatched features. - 'INNER JOIN': Returns only rows where all specified features have matching, non-missing values.
Parameters: source_schema (str): The schema of the source table to augment. source_name (str): The name of the source table to augment. entity_id (str or list of str): The entity ID(s) for joining the source data with feature tables. feature_selection (list): A list of features to select and join with the source data. output_view_schema (str): The schema in which to create the output view. output_view_name (str): The name of the output view to create with augmented features. join_type (str, optional): 'LEFT JOIN' or 'INNER JOIN'. Defaults to 'LEFT JOIN'.
Returns: tdml.DataFrame: A DataFrame object that references the newly created view.
rebuild_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, grouped=None, join_type=None, as_of=None, time_manager=None, current=False, replace=False, comment=None, return_query=False, start=None, end=None, duration=None, delay=None, history=False)
Rebuild an existing dataset with another strategy or at another point in time.
The source view's own DDL is the specification: the same storage tables,
feature ids, process versions and column names are kept. Nothing is
re-resolved against the feature catalog, so the result does not depend on
the session's DATA_DOMAIN nor on which versions are registered today.
Parameters¶
view_name, schema_name :
The source dataset (view_name may be 'database.view').
target_view_name, target_schema_name :
Where to write the result. Default: the source itself, which requires
replace=True.
grouped : bool, optional
True → one pivoted sub-query per (storage table, version);
False → one sub-query per feature. None keeps the source's.
join_type : {'INNER', 'FULL OUTER'}, optional
None keeps the source's.
as_of, time_manager, current :
New temporal mode — a literal point in time, a TimeManager, or
current=True for CURRENT VALIDTIME. None of them keeps the
source's mode. Given on a historical dataset, they turn it back into a
snapshot.
start, end, duration, delay, history :
Rebuild as a HISTORICAL dataset over that window (see
build_historical_dataset); history=True alone means the full
history. On a historical source they replace its window.
replace : bool
Allow overwriting a view that already exists. Without it the call
refuses (REPLACE VIEW would silently overwrite it).
return_query : bool
Return the DDL instead of executing it.
Returns the new dataset as a teradataml DataFrame (or the DDL), None on
a recoverable error (logged with the fix).
build_grouped_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, replace=False, comment=None, return_query=False)
Build the grouped version of an existing (non-grouped) dataset.
The grouped strategy reads each storage table once per process version —
features sharing a table are pivoted with MAX(CASE WHEN …) — instead of
once per feature, so a wide dataset needs far fewer joins. Same rows, same
columns, same temporal mode as the source.
target_view_name defaults to <view_name>_GROUPED (a new dataset, the
source is left untouched). Pass the source's own name with replace=True
to convert it in place.
build_historical_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, join_type='INNER', start=None, end=None, duration=None, delay=None, record_lineage=True, document=False, vector_indexing=False, process_version=False, feature_offsets=None)
Create a HISTORICAL dataset view: one row per entity per period of constant values.
Where build_dataset returns a snapshot (each entity's feature values at
one valid instant), this returns their history over a window::
<entity columns>, ValidPeriod, ValidStart, ValidEnd, <feature columns>
Each row says: from ValidStart (included) to ValidEnd (excluded),
this entity had exactly these feature values. Consecutive rows of an
entity always differ in at least one value (NORMALIZE ON MEETS OR
OVERLAPS merges equal neighbours, should the stored history hold any). ValidPeriod is the same interval as a Teradata
PERIOD(TIMESTAMP(0) WITH TIME ZONE), for CONTAINS / OVERLAPS.
The result is a plain view: nothing is materialised, new ingestions show up immediately, and a relative window moves by itself.
Parameters¶
entity_id, selected_features, view_name, schema_name, comment, return_query :
As in build_dataset (feature names alone when each has one version).
join_type : {'INNER', 'FULL OUTER'}
'INNER' keeps a period only where EVERY feature has a value — an
entity's history starts when its last feature appears, and an entity
whose features never overlap is absent. 'FULL OUTER' keeps every
period where AT LEAST ONE feature has a value, with NULLs elsewhere.
Either way, at any instant t of the window the history equals
build_dataset(..., as_of=t, join_type=...).
Example — feature A from January (10, then 20 from July), feature B
only from April::
FULL OUTER: [Jan, Apr) A=10 B=NULL | [Apr, Jul) 10 x | [Jul, ...) 20 x
INNER: [Apr, Jul) 10 x | [Jul, ...) 20 x
start, end : str, date, datetime or pandas.Timestamp, optional
Fixed window bounds (ISO strings accepted, a bare date is midnight).
The window is half-open [start, end): end='2024-04-01' covers
all of March. Without end the window ends at now.
duration : str, tuple or timedelta, optional
Window length: '3 months', '90 days', '12 hours',
'1 year 2 months', ISO 8601 'P3M', (3, 'months') or a
timedelta. Combined with start → [start, start + duration);
with end → [end - duration, end); alone → [now - duration, now).
Months use ADD_MONTHS (31 May minus 3 months is 29 Feb).
delay : same forms as duration, optional
Shift now back: duration='12 months', delay='1 day' is
[now - 1 day - 12 months, now - 1 day). Not combinable with
start/end.
``now`` is the current valid time when the view is QUERIED, so any
window without a fixed end slides with time. No argument at all means
the full history up to now. A period still open in the store therefore
ends at the window's end, never at 9999-12-31.
record_lineage, document, vector_indexing :
As in build_dataset.
process_version : bool, default False
Also create <view_name>_PV: the same history with a
<feature>_PROCESS_VERSION column per feature (the version of the
process SQL that wrote each period, NULL before versioning). The dataset
view itself is never changed. Whatever this flag, a history that spans
several process versions is reported in the log
(tdfs4ds.history_versions).
feature_offsets : dict, optional
Lags and leads, as in build_dataset: each instant of the history
reads amount_LAG_1D one day earlier, churn_LEAD_1MO one month
later (NULL until that date is reached). A history with offsets is
computed with elementary intervals for both join types (a shifted period
cannot go through the native sequenced join). Not combinable with
process_version=True yet.
Returns¶
tdml.DataFrame on the new view, the DDL when return_query=True, or
None when no feature could be resolved.
Notes¶
INNER uses Teradata's native SEQUENCED VALIDTIME join (each storage
table read once, rowkey merge joins, the window folded into the join).
FULL OUTER cuts each entity's timeline at every change point and attaches
each feature on containment; it reads each storage table more than once.
See tdfs4ds.dataset.history.
The DatasetSpec of an existing dataset view, read from its own DDL.
view_name may be 'database.view'. Returns None (with an error
logged) when the view is missing or is not a tdfs4ds dataset view.
restore_dataset_catalog(databases=None, discover=True, dry_run=True, drop_orphans=False, display=True, datasets=None)
List every dataset and restore the dataset catalog from the views themselves.
Parameters¶
datasets : list of str, optional
Restrict the check (and any repair) to these datasets, given as
'view' or 'database.view'. Default: every dataset found.
databases : list of str, optional
Databases to scan for dataset views. Default: the view database
(tdfs4ds.view_schema()), the table database (tdfs4ds.SCHEMA) and
every database the catalog already references.
discover : bool
Also look for dataset views the catalog does not know (DBC.TablesV
scan) and offer to register them. False checks catalogued datasets
only.
dry_run : bool
True (default) changes nothing and reports what would be done.
False applies it.
drop_orphans : bool
When applying, also close the catalog rows of datasets whose view no
longer exists (CURRENT VALIDTIME DELETE — history kept). Off by
default: a missing view may be a permission problem, not a deletion.
Returns¶
pandas.DataFrame
One row per dataset: STATUS is one of OK, REPAIR (catalog
disagrees with the view), REGISTER (view not catalogued),
DUPLICATE (registered several times), ORPHAN (view gone) or
UNPARSEABLE (catalogued view that is not a build_dataset view — left
alone); ISSUES says what disagrees, ACTION what was (or would
be) done. STRATEGY/JOIN_TYPE/TIME_MODE describe the view.
retrieve_batch(dataset, list_entity_keys, entity_cols=None, *, columns=None, method='auto', volatile_threshold=DEFAULT_VOLATILE_THRESHOLD, conn=None, dedup=True, pushdown='auto')
Retrieve rows of dataset for a list of entity-key tuples.
Parameters¶
dataset : str
Fully-qualified DB.dataset_view to read from.
list_entity_keys : list[tuple]
Entity-key tuples in the order of entity_cols (each tuple one row's key).
entity_cols : list[str] | None
Grain/entity key columns. When None they (and their SQL types) are
auto-derived from the dataset catalog (needs an active teradataml context).
columns : list[str] | None
Columns to project; None selects all.
method : {'auto','in_list','volatile_join'}
'in_list' builds a WHERE predicate (single-col IN / composite
OR-of-ANDs); 'volatile_join' loads the keys into a volatile
table co-located on the feature-store primary index and joins; 'auto'
(default) uses the volatile join when the unique-key count reaches
volatile_threshold OR when the in_list predicate would exceed the
parser limit, else the in_list predicate.
volatile_threshold : int
Key count at/above which 'auto' switches to the volatile join.
conn : teradatasql connection | None
Optional raw connection (benchmark reuse). None uses the teradataml
context.
dedup : bool
Drop duplicate key tuples first (required for the volatile join; also
shrinks the in_list predicate).
pushdown : {'auto', True, False}
'auto' (default) regenerates the dataset's query from its DDL with
the keys inside every storage-table read, so each FS_T_* table is
read by primary index and only in the partitions the keys fall in -
filtering the view from outside reaches neither, and under FULL OUTER
nothing can be pushed through the view at all. Falls back to the view
for a historical dataset or a view the parser does not understand.
False keeps the previous SQL exactly. Same rows either way.
Returns¶
list[tuple] The fetched rows.
Resolve the REAL primary index / partitioning of a dataset's feature tables.
Returns a dict::
{"primary_index": [...], "partitioning": "...",
"source": "DBC.IndicesV" | "fallback_entity_cols",
"feature_tables": ["db.FS_T_…", …]}
The volatile key table should use primary_index so the join co-locates
with the feature storage tables. Fail-open: returns entity_cols when the
real distribution can't be read (restricted DBC, view-only, no match).
Return the dataset grain as [(column_name, sql_type), …].
Reads ENTITY (name) + ENTITY_TYPE (type) from the dataset catalog's
entity table. Requires an active teradataml context. Returns [] when the
dataset can't be resolved.
str, optional
A comment to attach to the created view for documentation purposes. Defaults to None.