Skip to content

Process versions

Since 0.3.3.12.

A process is identified by its PROCESS_ID, which belongs to its view: registering the same view again keeps it, and every value the process writes carries it as its feature version. The SQL beneath the view can still change: someone edits a view the process reads, the query optimizer rewrites the process view, a migration recreates it. The process version tracks that SQL.

What is stored

The SQL of a process is its chain: the process view, then every view it reads, down to the source tables. tdfs4ds stores the chain of each version in a temporal table, FS_PROCESS_SQL (public view FS_V_PROCESS_SQL): one row per view, the text in VARCHAR(32000) chunks, a hash per view and one for the whole chain.

upload_features and run() read the chain and compare its hash with the stored one:

Situation What happens
Same SQL (whitespace, case and comments do not count) nothing is written
Nothing stored yet (first run after an upgrade) the chain is stored under the catalog's current PROCESS_VERSION, which does not change
SQL changed a new version: the chain is stored, and the process catalog's PROCESS_VERSION moves to it
SQL back to an earlier version that version is in force again; no new number

In a roll_out, only the first date checks the SQL. The check costs about one second per run when nothing changed (a SHOW VIEW per view of the chain, one hash read).

TimeManagers and FilterManagers

Since 0.3.3.13 (0.3.3.12 could record a spurious version 2 for a process whose manager was attached at registration).

A manager moves by replacing its own view at every update(): its SQL changes at each business date or segment, which is no change of the process. So:

  • a manager view is never part of a version (the chain stops at it, as an input);
  • every manager the chain reads is replaced by a placeholder before hashing, so repointing the process onto another manager (link_time_managers, unify_time_managers, split_time_manager, ...) does not make a version either. Which manager a process uses is recorded, in time, by FS_TIME_MANAGER / FS_FILTER_MANAGER.

The stored SQL keeps the real manager names; only the comparison ignores them.

Which version computed a value

A version is stamped at the valid time the process writes its values at: the TimeManager's date (or FEATURE_STORE_TIME), else the run's own timestamp. The process catalog row and the values carry the same valid time. So the version that computed a stored value is the one in force at the value's ValidStart.

tdfs4ds.process_sql_versions(process_id)          # one row per version: validity, changed views, who
tdfs4ds.process_sql(process_id, version=2)         # the stored chain of version 2
tdfs4ds.process_sql(process_id, as_of='2024-03-31')  # the version in force at that date
print(tdfs4ds.process_sql_diff(process_id, 1, 2))  # a unified diff, view by view

The feature version of the stored values does not change: FEATURE_VERSION in the storage tables stays the PROCESS_ID, so datasets, histories and plots read exactly the rows they read before.

Histories

When the window of a historical dataset spans several process versions of a feature, build_historical_dataset says so in its log, and tdfs4ds.history_versions('DB.HIST_VIEW') lists them per feature. With process_version=True, a companion view <view>_PV adds a <feature>_PROCESS_VERSION column to each period (NULL before versioning). See Historical datasets.

In the admin app

Feature catalog, process detail:

  • the versions, with their validity, the views that changed, the runs that used each and who captured it;
  • a version opens its stored SQL, and two versions can be compared;
  • a SQL changed since the last capture is flagged, with Capture now;
  • at a past valid time, the SQL shown is the version in force then.

The export (XLSX / HTML) adds SQL_VERSIONS and LAST_SQL_CHANGE per process and, at a past valid time, the stored SQL.

Upgrading an existing store

  • connect() creates FS_PROCESS_SQL and its view when they are missing. A user who may not create them gets one warning; versioning is then off for the session and nothing else changes. No existing table is altered.
  • Existing processes have no stored SQL until their first capture: the next run() or upload_features, or the backfill below. Runs before that show as "before versioning".
tdfs4ds.capture_process_sql(all_processes=True)                # dry run: what would be stored
tdfs4ds.capture_process_sql(all_processes=True, dry_run=False) # store the current SQL of every process

The same backfill is in the admin app under Maintenance > Process SQL versions.

  • An older tdfs4ds client sharing the store ignores the table. A SQL change made through it is recorded at the next capture by a newer client.
  • A migrated process starts with its own first capture in the target store; the source's versions are not copied.

Switching it off

tdfs4ds.PROCESS_SQL_VERSIONING = False   # or TDFS4DS_PROCESS_SQL_VERSIONING=false

upload_features and run() then send exactly the statements of 0.3.3.11.