Temporal model & time travel¶
Every catalog and every feature value is VALIDTIME temporal. That gives you four capabilities for free:
- Audit — every change is preserved with a validity range.
- Reproducibility — build a dataset
as_ofa past timestamp and obtain bit-for-bit identical features. - What-if — register a new process today, ingest yesterday's data with
FEATURE_STORE_TIME, compare. - Recovery — undelete by re-querying at a timestamp before the corruption.
The global handle¶
tdfs4ds.FEATURE_STORE_TIME = '2024-06-01 00:00:00' # point-in-time
tdfs4ds.FEATURE_STORE_TIME = None # current
Every read goes through CURRENT VALIDTIME when FEATURE_STORE_TIME is None
and VALIDTIME AS OF <ts> otherwise. The catalogs, the consumer agent and the
query optimizer honour this global directly.
Datasets are the exception: a dataset is a permanent view, so its point in
time is part of its definition and is passed to build_dataset explicitly —
as_of= for a fixed date, time_manager= to follow a TimeManager (see
below). Setting FEATURE_STORE_TIME alone
builds a current dataset.
Set it before reading, not before writing
FEATURE_STORE_TIME controls reads. Writes always reflect the current
transaction time. To replay yesterday's ingestion at yesterday's state,
use TimeManager and roll_out instead (see User guide / TimeManager).
Two clocks¶
tdfs4ds distinguishes two notions of time:
| Clock | What it represents | How to drive it |
|---|---|---|
| System time | When a row was inserted / updated | Implicit — Teradata's VALIDTIME |
| Business time | When the event happened in the real world | Explicit — TimeManager advances a "current business date" view that drives source queries |
For nightly ingestion, advance both: use TimeManager to step through
business dates, and set FEATURE_STORE_TIME = business_time.get_date_in_the_past()
so reads stay aligned with writes.
Reading at a past timestamp¶
tdfs4ds.FEATURE_STORE_TIME = '2024-01-01 00:00:00'
tdfs4ds.process_catalog() # processes registered on or before that date
tdfs4ds.feature_catalog() # features that were live then
tdfs4ds.FEATURE_STORE_TIME = None
Datasets at a point in time¶
A dataset view embeds its temporal qualifier in every feature sub-query.
Three modes, all compatible with grouped=True:
| Mode | build_dataset argument |
Sub-query qualifier |
|---|---|---|
| Current (default) | — | SEQUENCED VALIDTIME (outer CURRENT VALIDTIME) |
| Fixed date | as_of='2024-01-01' |
VALIDTIME AS OF TIMESTAMP '2024-01-01 00:00:00' |
| TimeManager | time_manager=bt |
NONSEQUENCED VALIDTIME, filtered by ValidStart <= (SELECT … BUSINESS_DATE …) AND ValidEnd > (…) |
# Point-in-time-correct training set, frozen at a date
past = tdfs4ds.build_dataset(
entity_id = ['CUSTOMER_ID'],
selected_features = selected_features,
view_name = 'DS_CUSTOMER_2024_01_01',
grouped = True,
as_of = '2024-01-01',
)
# A dataset that moves with the business clock: bt.update(i) (or roll_out)
# shifts it to step i without rebuilding the view
rolling = tdfs4ds.build_dataset(
entity_id = ['CUSTOMER_ID'],
selected_features = selected_features,
view_name = 'DS_CUSTOMER_ROLLING',
time_manager = bt,
)
The historical form, feature_store_time=True with FEATURE_STORE_TIME set,
still builds a fixed-date dataset. An existing dataset can be re-timed with
rebuild_dataset(view, as_of=…) or rebuild_dataset(view, time_manager=…).
See Datasets.
How a write works¶
upload_featuresopens a follow-up row (STATUS=RUNNING).- The feature engineering view is queried.
- Values are MERGE'd into
FS_T_<ENT>_<TYPE>(or UPDATE+INSERT ifSTORE_FEATURE='UPDATE_INSERT'). - The follow-up row is closed (
STATUS=COMPLETED) — or remainsRUNNINGif the process crashed. A staleRUNNINGrow is the canonical signal that an ingestion run is incomplete.
See the Inspect & monitor page for the follow-up table and recovery patterns.
End period¶
tdfs4ds.END_PERIOD (default 'UNTIL_CHANGED') controls the validity
end-date inserted when a feature value first appears. The default sentinel
keeps the row open until a newer value supersedes it, mirroring slowly
changing dimension semantics.