tdfs4ds.dataset.ddl_spec¶

tdfs4ds.dataset.ddl_spec ¶

Dataset view DDL <-> specification, both directions. Pure (no database access).

A dataset view built by build_dataset is fully described by:

its entity columns (the grain),
one FeatureSpec per exposed feature column — storage table, FEATURE_ID, FEATURE_VERSION (the producing process) and the column alias,
the join type, the build strategy (multi-join or grouped),
its temporal mode (DatasetTime).

render_dataset_ddl turns that into the REPLACE VIEW statement; parse_dataset_ddl reads it back out of a view's own DDL. The round trip is what makes three operations possible WITHOUT re-resolving features against the catalog (which depends on the session's DATA_DOMAIN and on what is registered today, and could silently pick a different version):

restoring a dataset catalog from the views that actually exist,
rebuilding a non-grouped dataset as a grouped one,
rebuilding a dataset at another point in time.

The view text is authoritative: it is what Teradata executes.

CURRENT `module-attribute` ¶

CURRENT = 'CURRENT'

AS_OF `module-attribute` ¶

AS_OF = 'AS_OF'

TIME_MANAGER `module-attribute` ¶

TIME_MANAGER = 'TIME_MANAGER'

HISTORY `module-attribute` ¶

HISTORY = 'HISTORY'

MULTIJOIN `module-attribute` ¶

MULTIJOIN = 'MULTIJOIN'

GROUPED `module-attribute` ¶

GROUPED = 'GROUPED'

_IDENT `module-attribute` ¶

_IDENT = '(?:"[^"]+"|[A-Za-z_$#@][\\w$#@]*)'

_SUBQUERY_END_RE `module-attribute` ¶

_SUBQUERY_END_RE = compile('\\)\\s*A(\\d+)\\b')

_TABLE_RE `module-attribute` ¶

_TABLE_RE = compile(f'\bFROM\s+(?P<table>{_IDENT}\s*\.\s*{_IDENT})\s+B1\b', IGNORECASE)

_VERSION_RE `module-attribute` ¶

_VERSION_RE = compile("\\bFEATURE_VERSION\\s*=\\s*'(?P<v>[^']*)'", IGNORECASE)

_PLAIN_ALIAS_RE `module-attribute` ¶

_PLAIN_ALIAS_RE = compile('\\bB1\\.FEATURE_VALUE\\s+AS\\s+(?P<alias>\\w+)', IGNORECASE)

_PLAIN_ID_RE `module-attribute` ¶

_PLAIN_ID_RE = compile('\\bFEATURE_ID\\s*=\\s*(?P<id>\\d+)', IGNORECASE)

_PIVOT_RE `module-attribute` ¶

_PIVOT_RE = compile('MAX\\s*\\(\\s*CASE\\s+WHEN\\s+FEATURE_ID\\s*=\\s*(?P<id>\\d+)\\s+THEN\\s+FEATURE_VALUE\\s+END\\s*\\)\\s+AS\\s+(?P<alias>\\w+)', IGNORECASE)

_MULTIJOIN_WHERE_RE `module-attribute` ¶

_MULTIJOIN_WHERE_RE = compile('\\(\\s*FEATURE_ID\\s*=\\s*\\d+\\s+AND\\s+FEATURE_VERSION\\s*=', IGNORECASE)

_AS_OF_LITERAL_RE `module-attribute` ¶

_AS_OF_LITERAL_RE = compile("\\bVALIDTIME\\s+AS\\s+OF\\s+(?:TIMESTAMP|DATE)\\s*'(?P<ts>[^']*)'", IGNORECASE)

_TM_PERIOD_RE `module-attribute` ¶

_TM_PERIOD_RE = compile(f'\bB1\.ValidStart\s*<=\s*\(\s*SELECT\s+CAST\s*\(\s*BUSINESS_DATE\s+AS\s+TIMESTAMP\s*\(\s*\d\s*\)\s+WITH\s+TIME\s+ZONE\s*\)\s+FROM\s+(?P<view>{_IDENT}(?:\s*\.\s*{_IDENT})?)\s*\)', IGNORECASE)

_AS_OF_TM_RE `module-attribute` ¶

_AS_OF_TM_RE = compile(f'\bVALIDTIME\s+AS\s+OF\s*\(\s*SELECT\s+BUSINESS_DATE\s+FROM\s+(?P<view>{_IDENT}(?:\s*\.\s*{_IDENT})?)\s*\)', IGNORECASE)

_OUTER_CURRENT_RE `module-attribute` ¶

_OUTER_CURRENT_RE = compile('\\bCURRENT\\s+VALIDTIME\\b', IGNORECASE)

_VIEW_NAME_RE `module-attribute` ¶

_VIEW_NAME_RE = compile(f'\b(?:REPLACE|CREATE)\s+VIEW\s+(?P<name>{_IDENT}(?:\s*\.\s*{_IDENT})?)\s+AS\b', IGNORECASE)

_SELECT_RE `module-attribute` ¶

_SELECT_RE = compile('\\bSELECT\\b', IGNORECASE)

_NORMALIZE_RE `module-attribute` ¶

_NORMALIZE_RE = compile('\\bNORMALIZE\\s+ON\\s+MEETS\\s+OR\\s+OVERLAPS\\b', IGNORECASE)

_INSTANT_RE `module-attribute` ¶

_INSTANT_RE = compile('B1\\.ValidStart\\s*<=\\s*(?P<t>.+?)\\s+AND\\s+B1\\.ValidEnd\\s*>\\s*(?P=t)\\s*(?:AND\\b|\\)|$)', IGNORECASE | DOTALL)

_OFFSET_PIVOT_RE `module-attribute` ¶

_OFFSET_PIVOT_RE = compile('MAX\\s*\\(\\s*CASE\\s+WHEN\\s+(?P<cond>.+?)\\s+THEN\\s+B1\\.FEATURE_VALUE\\s+END\\s*\\)\\s+AS\\s+(?P<alias>\\w+)', IGNORECASE | DOTALL)

_TS_LITERAL_RE `module-attribute` ¶

_TS_LITERAL_RE = compile("^CAST\\s*\\(\\s*TIMESTAMP\\s*'(?P<ts>[^']*)'\\s+AS\\s+TIMESTAMP\\s*\\(\\s*0\\s*\\)\\s+WITH\\s+TIME\\s+ZONE\\s*\\)$", IGNORECASE)

_NOW_SQL_RE `module-attribute` ¶

_NOW_SQL_RE = compile('^CAST\\s*\\(\\s*CURRENT_TIMESTAMP\\s*\\(\\s*0\\s*\\)\\s+AS\\s+TIMESTAMP\\s*\\(\\s*0\\s*\\)\\s+WITH\\s+TIME\\s+ZONE\\s*\\)$', IGNORECASE)

_TM_SCALAR_RE `module-attribute` ¶

_TM_SCALAR_RE = compile(f'^\(\s*SELECT\s+(?P<e>.+?)\s+FROM\s+(?P<view>{_IDENT}(?:\s*\.\s*{_IDENT})?)\s*\)$', IGNORECASE | DOTALL)

_BD_RE `module-attribute` ¶

_BD_RE = compile('^CAST\\s*\\(\\s*BUSINESS_DATE\\s+AS\\s+TIMESTAMP\\s*\\(\\s*\\d\\s*\\)\\s+WITH\\s+TIME\\s+ZONE\\s*\\)$', IGNORECASE)

DatasetTime ¶

The resolved temporal mode of one dataset view.

Holds what the SQL needs (outer_qualifier, sub_qualifier) and what the catalog records (mode, as_of, the TimeManager's location).

mode `instance-attribute` ¶

mode = mode

as_of `instance-attribute` ¶

as_of = as_of

tm_database `instance-attribute` ¶

tm_database = tm_database

tm_view `instance-attribute` ¶

tm_view = tm_view

tm_table `instance-attribute` ¶

tm_table = tm_table

window `instance-attribute` ¶

window = window

tm_view_sql `property` ¶

tm_view_sql

outer_qualifier `property` ¶

outer_qualifier: str

sub_qualifier `property` ¶

sub_qualifier: str

business_date_sql `property` ¶

business_date_sql: str

The manager's business date as a scalar sub-query (TIME_MANAGER only).

period_predicate `property` ¶

period_predicate: str

Leading WHERE conjuncts selecting the valid instant, or ''.

Only the TIME_MANAGER mode needs one (see the module docstring for why it is not an AS OF sub-query); for the other modes the qualifier alone selects the instant, and the SQL is byte-identical to before.

init ¶

__init__(mode=CURRENT, as_of=None, tm_database=None, tm_view=None, tm_table=None, window=None)

repr ¶

__repr__()

eq ¶

__eq__(other)

instant_sql ¶

instant_sql(offset=None) -> str

The dataset's valid instant moved by offset, as a constant expression.

CURRENT: the query time; AS_OF: the literal; TIME_MANAGER: the manager's business date, the shift inside the scalar sub-query so that it stays one single-row read resolved once. Shifting the constant (never the ValidStart / ValidEnd columns) keeps partition elimination: the arithmetic is folded into a literal in the plan (verified on Vantage 20).

catalog_validtime ¶

catalog_validtime() -> str

The value written to DATASET_VALIDTIME (VARCHAR(255)).

CURRENT VALIDTIME or VALIDTIME AS OF TIMESTAMP '...' — the qualifier the view carries — or TIME_MANAGER <db>.<view> for a dataset following a TimeManager. A very long quoted manager name could overflow the column, so it then degrades to the mode name; the build table holds the full location anyway.

describe ¶

describe() -> str

FeatureSpec ¶

One feature column of a dataset view.

slots `class-attribute` `instance-attribute` ¶

__slots__ = ('table', 'feature_id', 'version', 'alias', 'from_version_list', 'offset', 'base_name')

table `instance-attribute` ¶

table = table

feature_id `instance-attribute` ¶

feature_id = int(feature_id)

version `instance-attribute` ¶

version = version

alias `instance-attribute` ¶

alias = alias

from_version_list `instance-attribute` ¶

from_version_list = from_version_list

offset `instance-attribute` ¶

offset = offset

base_name `instance-attribute` ¶

base_name = base_name

init ¶

__init__(table, feature_id, version, alias, from_version_list=False, offset=None, base_name=None)

key ¶

key()

eq ¶

__eq__(other)

repr ¶

__repr__()

DatasetSpec ¶

Everything needed to (re)generate a dataset view.

entity `instance-attribute` ¶

entity = list(entity)

features `instance-attribute` ¶

features = list(features)

join_type `instance-attribute` ¶

join_type = join_type

strategy `instance-attribute` ¶

strategy = strategy

time `instance-attribute` ¶

time = time if time is not None else DatasetTime(CURRENT)

schema_name `instance-attribute` ¶

schema_name = schema_name

view_name `instance-attribute` ¶

view_name = view_name

offset_strategy `instance-attribute` ¶

offset_strategy = offset_strategy or PIVOT

grouped `property` ¶

grouped

has_offsets `property` ¶

has_offsets

init ¶

__init__(entity, features, join_type='INNER', strategy=MULTIJOIN, time=None, schema_name=None, view_name=None, offset_strategy=None)

repr ¶

__repr__()

_outer_entity ¶

_outer_entity(entity, n_sub_queries, join_type)

_wrap_view ¶

_wrap_view(schema_name, view_name, outer_qualifier, select_query, from_query)

_key_parts ¶

_key_parts(key_filter, table)

(join_sql, where_sql) a key filter adds to one sub-query; both '' without one.

_render_multijoin ¶

_render_multijoin(spec, sub_q, key_filter=None)

One sub-query per feature, chained with JOINs (the classic strategy).

_feature_groups ¶

_feature_groups(features)

Features grouped by (table, version), in first-appearance order.

_render_grouped ¶

_render_grouped(spec, sub_q, key_filter=None)

One sub-query per (storage table, version), pivoted with MAX(CASE WHEN).

check_offsets ¶

check_offsets(spec)

Refuse what cannot give a row; warn about what gives NULLs by construction.

_containment ¶

_containment(t, offset)

B1 holds the value valid at instant t; a lead also needs t reached.

_offset_groups ¶

_offset_groups(features, by_table=False)

Offset columns grouped by (table, FEATURE_ID, version), first appearance order.

by_table (the grouped strategy): by (table, version), so one read serves every offset column of every feature stored in that table.

_pivot_subquery ¶

_pivot_subquery(spec, table, feature_id, version, feats, key_filter, pad)

One read for several offset columns (PIVOT).

The WHERE keeps only the rows valid at one of the instants (constants: the plan reads the FEATURE_ID partition(s)); each column takes the row valid at its own instant. Grouping by the entity (the primary index of the storage table) aggregates locally, without redistribution. With feature_id None the columns come from several features of one table (the grouped strategy): each column then names its FEATURE_ID inside its CASE.

_offset_subquery ¶

_offset_subquery(spec, f, key_filter, pad)

One offset column on its own (PER_OFFSET): the multi-join shape at its instant.

_render_with_offsets ¶

_render_with_offsets(spec, sub_q, key_filter=None)

A dataset with at least one offset column.

Columns without an offset keep the sub-query they have without offsets (the multi-join one, or the grouped pivot by table and version), so the only text that changes is the offset columns'. These are read with NONSEQUENCED VALIDTIME and an explicit instant, which the outer CURRENT VALIDTIME (if any) does not reach. The outer select lists the columns in the spec's order.

render_dataset_ddl ¶

render_dataset_ddl(spec, schema_name=None, view_name=None)

The REPLACE VIEW statement for spec. None when it has no feature.

_renderer ¶

_renderer(spec)

render_retrieval_sql ¶

render_retrieval_sql(spec, key_filter)

The dataset's own query, with a key filter pushed into every sub-query.

Same select list, same joins and same temporal predicates as the view; the only addition is key_filter's join and predicate inside each storage-table read, next to its FEATURE_ID / FEATURE_VERSION filter. Filtering the view from the outside cannot get there: under FULL OUTER the view's entity is a COALESCE over every sub-query, and a grouped sub-query aggregates before an outer predicate applies, so each FS_T_* table is read whole - no primary-index access, no partition elimination.

Returns the same rows as SELECT * FROM <view> WHERE <keys>:

the key columns are the entity, i.e. the grouping columns, so filtering before GROUP BY / HAVING drops whole groups and changes none that survive;
under FULL OUTER every side joins on the entity, so restricting every side to the key set equals restricting COALESCE(entity) to it.

None for a historical dataset (its own renderer, not supported here) or a spec with no feature.

_instant_of ¶

_instant_of(expr)

(DatasetTime, Offset) of an instant expression an offset column reads, or None.

_parse_offset_body ¶

_parse_offset_body(body, table, version)

Offset columns of one sub-query: (features, time, pivot) or None.

Recognised only in a NONSEQUENCED VALIDTIME sub-query whose instant is a shifted constant tdfs4ds writes (the query time, a literal, a TimeManager scalar) — so the historical shapes, which read the instant through their qualifier or the TimeManager predicate, are never taken for one.

_skip_literal ¶

_skip_literal(text, i, step)

Index past the quoted literal that text[i] opens/closes, walking by step.

_matching_open ¶

_matching_open(text, close_idx)

Index of the '(' matching the ')' at close_idx (quotes respected).

_split_top_level ¶

_split_top_level(text, sep=',')

_column_name ¶

_column_name(item)

Output column name of an outer select item: 'A1.x' -> x, '... AS e' -> e.

_sub_query_time ¶

_sub_query_time(body)

parse_dataset_ddl ¶

parse_dataset_ddl(ddl)

Read a DatasetSpec back out of a dataset view's DDL.

Understands both strategies build_dataset emits (multi-join and grouped), both join types and the three temporal modes, including views written by earlier tdfs4ds versions (the generated shape has not changed). Returns None when the text is not a tdfs4ds dataset view — e.g. a view from augment_source_with_features or any user view — rather than guessing.

Features come back in the view's own column order; the entity is every other output column.

storage_table_sql ¶

storage_table_sql(database, table)

A storage table as build_dataset writes it into a view: "DB"."TABLE".

_plain_table ¶

_plain_table(name)

swap_feature_tables ¶

swap_feature_tables(spec, mapping)

A copy of spec in which some features read another storage table. Pure.

mapping maps what to move to where:

{feature_id: 'DB.TABLE'} (int key): that feature, whichever table it is read from;
{'OLDDB.OLDTABLE': 'DB.TABLE'} (str key): every feature read from that table.

Only the features matched move; the others keep their table (the old table still serves them - a rewrite of every occurrence of its name would break them). Strategy, join type, time mode, aliases, offsets and window are carried by the copy, so render_dataset_ddl writes the same view with the new tables. Returns (new_spec, moved) where moved lists (alias, old table, new table).

tdfs4ds.dataset.ddl_spec¶