tdfs4ds.dataset.restore¶
List the datasets of a feature store and restore the dataset catalog from them.
The catalog (FS_<name>_CATALOG / _ENTITY / _FEATURES) is a
description of dataset VIEWS that exist on their own. Earlier tdfs4ds versions
wrote parts of that description wrongly, and nothing read those parts back, so
they went unnoticed:
DATASET_VALIDTIMEheld a fragment of the select list (SELECTorSELECT A1.CUSTOMERID) instead of the temporal qualifier (fixed 0.3.1.214 for new registrations, existing rows untouched);FEATURE_ID— aVARCHARcolumn — received a bare integer, which Teradata pads on the implicit cast (' 6');- before 0.3.1.79 the entity/feature split was re-derived by regex and could register a view with zero features and every column as entity;
- a view registered twice makes
add_datasetrefuse every later update.
Each view's own DDL is authoritative — it is what Teradata executes — so the
catalog can be rebuilt from it: restore_dataset_catalog parses every
dataset view (ddl_spec.parse_dataset_ddl), compares what the catalog says
with what the view does, and, when asked, rewrites the rows that disagree.
DATASET_IDs are preserved (the MERGE is keyed on them), so references to a
dataset survive the repair.
module-attribute
¶
_COLUMNS = ['DATASET_DATABASE', 'DATASET_NAME', 'DATASET_ID', 'STATUS', 'ISSUES', 'ACTION', 'STRATEGY', 'JOIN_TYPE', 'TIME_MODE', 'N_ENTITY', 'N_FEATURES']
property
¶
True when the build table and its view exist (checked once, then cached).
property
¶
True when the window table and its view exist (checked once, then cached).
property
¶
True when the offsets table and its view exist (checked once, then cached).
__init__(schema_name=None, name='DATASET', existing_objects=None, schema_name_view=None, existing_view_objects=None)
Parameters¶
existing_objects : set, optional
A pre-computed set of lower-cased table/view names already present in
schema_name. When provided, the constructor skips its own
db_list_tables() call (saves one round-trip). Pass the set
returned by connect()'s DBC.TablesV query.
schema_name_view : str, optional
Database that holds the catalog VIEW objects. When None it defaults to
tdfs4ds.SCHEMA_VIEW if a dedicated view database is configured, else
to schema_name (co-located with the tables — byte-identical to the
historical single-database behaviour).
existing_view_objects : set, optional
Pre-computed set of lower-cased view names present in schema_name_view.
Only consulted when the view database differs from the table database.
Create whatever part of the catalog is missing. Idempotent.
Creates only the absent objects and leaves the existing ones alone, returning the list of names it created (empty when there was nothing to do). Creation order is preserved, so a table is always created before the view that reads it.
It used to refuse — logging an ERROR and raising — as soon as any
of the six objects existed, while _exists only reported True when
all six did. Those two predicates disagree on every partial catalog,
which left it permanently unrepairable: the constructor sees "not all
present", calls this, and this rejects "some present". Because the
constructor self-heals on every instantiation, that fired on ordinary
READ paths (dataset_catalog(), the agent's _resolve_dataset) and
on connect()/setup() — an ERROR-level log and an exception for a
condition the caller cannot act on. Self-healing is this constructor's
long-standing contract; it was simply incomplete.
Register (or update) dataset in the catalog. Returns its DATASET_ID.
Parameters¶
build : dict, optional
How the view was built — {'strategy', 'join_type', 'time'
(DatasetTime), 'source_dataset_id'} — written to the build table
when it exists. Omitted: the build table is left untouched.
dataset_id : str, optional
Force the DATASET_ID to write. Used by restore_dataset_catalog
when a dataset was registered several times and one id must be kept;
ordinary callers let it be looked up by name.
offsets : list of dict, optional
One row per column read at an offset (lags and leads), written to the
offsets table when it exists; [] closes the rows of an earlier
build. None (default) leaves the table untouched.
Build details (strategy, join type, temporal mode) — None when unavailable.
Offset columns (one row per lag / lead column) — None when unavailable.
Join it to get_dataset_features on (DATASET_ID, FEATURE_NAME): a
column without offset has no row.
History windows (one row per historical dataset) — None when unavailable.
Join it to the catalog on DATASET_ID with a LEFT OUTER JOIN: a
snapshot dataset has no row. WINDOW_START / WINDOW_END are the
bounds at query time (WINDOW_START NULL = from the earliest stored
value); the other columns are the definition.
Remove a dataset (catalog + entity + feature rows) by DATASET_ID.
Parameters¶
dataset_id : str
The DATASET_ID to remove.
purge : bool, optional
When False (default), a CURRENT VALIDTIME DELETE valid-time-closes
the currently-valid rows (temporal soft delete; history is retained).
When True, a NONSEQUENCED VALIDTIME DELETE physically removes every
temporal row for the dataset — use this to fully reset a dataset that
carries stale/closed history before re-registering it.
close_lineage : bool, optional
True (default): the dataset view's edges in FS_LINEAGE_GRAPH are
closed as well. False when the view itself stays (a duplicate
registration removed by restore_dataset_catalog).
Raises¶
ValueError
If no dataset with this dataset_id currently exists in the catalog.
Read a DatasetSpec back out of a dataset view's DDL.
Understands both strategies build_dataset emits (multi-join and grouped),
both join types and the three temporal modes, including views written by
earlier tdfs4ds versions (the generated shape has not changed). Returns
None when the text is not a tdfs4ds dataset view — e.g. a view from
augment_source_with_features or any user view — rather than guessing.
Features come back in the view's own column order; the entity is every other output column.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Node ids ('DB.OBJECT') of the persisted lineage graph, or None when it cannot be read or is empty (then nothing is reported as missing).
(ddl, None) or (None, error) — a missing view is a finding, not a crash.
(db, view) pairs whose text looks like a build_dataset view. Fail-open.
Human-readable differences between the catalog and the view.
build_row / window_row: None when the optional table does not exist
(nothing to compare), False when the dataset has no row, else the row.
offset_rows: None when the offsets table does not exist, else the
dataset's rows (possibly none).
restore_dataset_catalog(databases=None, discover=True, dry_run=True, drop_orphans=False, display=True, datasets=None)
List every dataset and restore the dataset catalog from the views themselves.
Parameters¶
datasets : list of str, optional
Restrict the check (and any repair) to these datasets, given as
'view' or 'database.view'. Default: every dataset found.
databases : list of str, optional
Databases to scan for dataset views. Default: the view database
(tdfs4ds.view_schema()), the table database (tdfs4ds.SCHEMA) and
every database the catalog already references.
discover : bool
Also look for dataset views the catalog does not know (DBC.TablesV
scan) and offer to register them. False checks catalogued datasets
only.
dry_run : bool
True (default) changes nothing and reports what would be done.
False applies it.
drop_orphans : bool
When applying, also close the catalog rows of datasets whose view no
longer exists (CURRENT VALIDTIME DELETE — history kept). Off by
default: a missing view may be a permission problem, not a deletion.
Returns¶
pandas.DataFrame
One row per dataset: STATUS is one of OK, REPAIR (catalog
disagrees with the view), REGISTER (view not catalogued),
DUPLICATE (registered several times), ORPHAN (view gone) or
UNPARSEABLE (catalogued view that is not a build_dataset view — left
alone); ISSUES says what disagrees, ACTION what was (or would
be) done. STRATEGY/JOIN_TYPE/TIME_MODE describe the view.