tdfs4ds.dataset.dataset_catalog¶
instance-attribute
¶
Parameters¶
entity, features : optional
The AUTHORITATIVE entity/feature split, supplied by build_dataset
(which knows it at build time). When BOTH are provided they are used
verbatim and the fragile DDL-regex re-derivation
(_retrieve_entities_and_features) is skipped. When omitted (e.g.
an ad-hoc Dataset(view_name=...) for inspection) the regex path is
used as before — preserving backward compatibility.
entity is {col: sql_type}; features is
{NAME: {id, version, type, database, view}}.
valid_time : str, optional
The governing temporal qualifier, supplied by build_dataset which
chose it. When omitted it is read back from the view's DDL.
property
¶
True when the build table and its view exist (checked once, then cached).
property
¶
True when the window table and its view exist (checked once, then cached).
property
¶
True when the offsets table and its view exist (checked once, then cached).
__init__(schema_name=None, name='DATASET', existing_objects=None, schema_name_view=None, existing_view_objects=None)
Parameters¶
existing_objects : set, optional
A pre-computed set of lower-cased table/view names already present in
schema_name. When provided, the constructor skips its own
db_list_tables() call (saves one round-trip). Pass the set
returned by connect()'s DBC.TablesV query.
schema_name_view : str, optional
Database that holds the catalog VIEW objects. When None it defaults to
tdfs4ds.SCHEMA_VIEW if a dedicated view database is configured, else
to schema_name (co-located with the tables — byte-identical to the
historical single-database behaviour).
existing_view_objects : set, optional
Pre-computed set of lower-cased view names present in schema_name_view.
Only consulted when the view database differs from the table database.
Create whatever part of the catalog is missing. Idempotent.
Creates only the absent objects and leaves the existing ones alone, returning the list of names it created (empty when there was nothing to do). Creation order is preserved, so a table is always created before the view that reads it.
It used to refuse — logging an ERROR and raising — as soon as any
of the six objects existed, while _exists only reported True when
all six did. Those two predicates disagree on every partial catalog,
which left it permanently unrepairable: the constructor sees "not all
present", calls this, and this rejects "some present". Because the
constructor self-heals on every instantiation, that fired on ordinary
READ paths (dataset_catalog(), the agent's _resolve_dataset) and
on connect()/setup() — an ERROR-level log and an exception for a
condition the caller cannot act on. Self-healing is this constructor's
long-standing contract; it was simply incomplete.
Register (or update) dataset in the catalog. Returns its DATASET_ID.
Parameters¶
build : dict, optional
How the view was built — {'strategy', 'join_type', 'time'
(DatasetTime), 'source_dataset_id'} — written to the build table
when it exists. Omitted: the build table is left untouched.
dataset_id : str, optional
Force the DATASET_ID to write. Used by restore_dataset_catalog
when a dataset was registered several times and one id must be kept;
ordinary callers let it be looked up by name.
offsets : list of dict, optional
One row per column read at an offset (lags and leads), written to the
offsets table when it exists; [] closes the rows of an earlier
build. None (default) leaves the table untouched.
Build details (strategy, join type, temporal mode) — None when unavailable.
Offset columns (one row per lag / lead column) — None when unavailable.
Join it to get_dataset_features on (DATASET_ID, FEATURE_NAME): a
column without offset has no row.
History windows (one row per historical dataset) — None when unavailable.
Join it to the catalog on DATASET_ID with a LEFT OUTER JOIN: a
snapshot dataset has no row. WINDOW_START / WINDOW_END are the
bounds at query time (WINDOW_START NULL = from the earliest stored
value); the other columns are the definition.
Remove a dataset (catalog + entity + feature rows) by DATASET_ID.
Parameters¶
dataset_id : str
The DATASET_ID to remove.
purge : bool, optional
When False (default), a CURRENT VALIDTIME DELETE valid-time-closes
the currently-valid rows (temporal soft delete; history is retained).
When True, a NONSEQUENCED VALIDTIME DELETE physically removes every
temporal row for the dataset — use this to fully reset a dataset that
carries stale/closed history before re-registering it.
close_lineage : bool, optional
True (default): the dataset view's edges in FS_LINEAGE_GRAPH are
closed as well. False when the view itself stays (a duplicate
registration removed by restore_dataset_catalog).
Raises¶
ValueError
If no dataset with this dataset_id currently exists in the catalog.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
SQL for expr ± (months, seconds) read from columns.
Same order as Duration.apply (months first, with ADD_MONTHS and its
month-end clamping, then the exact part), so the bounds equal the ones the
history view computes. The seconds are split into DAY(4) / HOUR / MINUTE /
SECOND intervals: a single INTERVAL SECOND holds at most 9999 seconds,
and a column cannot feed a DAY TO SECOND literal. Verified live,
including 9999 days 23:59:59 and 31 May minus 3 months -> 29 Feb.
The public view of the window table, with the bounds at query time.
WINDOW_END / WINDOW_START re-evaluate a sliding window each time
the view is read, exactly as the historical dataset does. WINDOW_START
is NULL for EARLIEST (from the first stored value).
One query level only: CURRENT VALIDTIME over a derived table is
refused (Error 9330, the temporal table being one level down), so the end
expression is written out again inside the start expression.
drop_dataset(dataset_id: str, schema_name: str = None, purge: bool = False, close_lineage: bool = True) -> None
Remove a dataset entry (catalog + entity + feature rows) by DATASET_ID.
purge=True physically removes all temporal rows (history included);
the default soft-deletes by valid-time-closing the currently-valid rows.
close_lineage=False keeps the view's lineage edges (the view stays).
To also remove its documentation and index entries, use
:func:tdfs4ds.removal.remove_dataset.