tdfs4ds.migration.executor¶
Cross-store migration executor (v0.3.1.150) — the write phase.
migrate_objects(plan, approvals) takes an approved manifest from
:func:tdfs4ds.plan_migration and recreates the selected objects in the
primary store, wave by wave (upstream-first), bringing along:
- non-FS upstream objects (intermediate views recreated from DDL; raw source tables copied or referenced cross-DB per choice),
- the filter/time managers (recreate or unify onto an equivalent primary one),
- the processes (re-registered with fresh primary ids; a source→primary id map is threaded everywhere),
- the feature-value data (default: recompute via
run(); fallback: copy theFS_T_*rows remapping the store-localFEATURE_ID), - the datasets (view recreated with
FEATURE_VERSIONrefs remapped), - the business dictionary (rows re-homed under the primary
DATABASE_NAME), - the lineage graph (re-derived from the primary DDL — primary-scoped), and
- the Teradata vector index (rebuilt so the migrated docs are searchable).
The executor refuses to run unless every conflict is approved and no approved
replace is still blocking (the lab→prod downstream-safety gate). Every
auxiliary step (BD / lineage / vector) is fail-open — a doc/index failure never
corrupts an otherwise-migrated feature (same discipline as upload_features).
module-attribute
¶
module-attribute
¶
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Uppercase, de-quoted, stripped — the canonical comparison form.
Split 'db.obj' / '"db"."obj"' / 'obj' into (db, obj) (raw,
de-quoted but original case preserved). db is '' when unqualified.
Return (skip_keys, replace_keys) from the approval map.
Source→primary placement for reference rewriting.
- copied tables / recreated views → their primary placement (rewrite refs),
reference-mode source tables → identity (keep the cross-DB source ref),skip(already-present) objects → their primary placement (refs point at the existing primary object).
Recreate a view in the primary: SHOW VIEW → rewrite refs + remap pids → REPLACE VIEW.
Copy a raw source table into the primary (WITH DATA) or leave it cross-DB.
Upstream non-FS objects (tables + intermediate views), dependencies first.
Entity columns -> {col: 'BIGINT'} (types are placeholders; run() recomputes the real SQL types from the live view — registration only uses the keys).
Re-register a migrated process view in the primary; return the primary PID.
Carry the source process's execution options over: FS_PROCESS_OPTIONS
(close_absent_entities) and FS_PROCESS_SETTINGS (track_entity_refresh).
Read from the source store by explicit schema (never through federation).
Fail-open: a source without the table, or a refused write, only loses the
option, said in the progress stream. The FS_R_* refresh tables are not copied
(a recompute regenerates them).
Record, for a migrated process, the TimeManager it reads in the primary.
The association drives run(): without it the recompute below ran in real
time, wrote today's value, and a later roll_out over past dates could no
longer insert anything before it (its merges only update the overlap). The
manager's primary location is the placement of the source manager's view.
Fail-open: a failure only loses the association, logged.
Recreate the clock of each migrated data domain in the primary and declare it (0.3.6.0).
The processes' soft clones are migrated as views over the clock's table of dates, which
Phase A copies (it is a source table of every clone); the clock view and the domain's
FS_DATA_DOMAIN_SETTINGS row were not, so the domain lost its clock. The clock view
is made over the placed table (or the table is copied first, as a hard clone, when no
clone brought it), at the source clock's cursor, and declared for the target domain.
Fail-open: a failure is reported, the processes keep their clones.
Recreate or unify a process's filter manager; return a FilterManager for registration (or None). Time managers are metadata-only (recorded via roll_out) and repointed if unified.
{(DB_UP, OBJ_UP)} (source keys) of objects that already exist in the
target and are kept: skipped by approval, or a reused source table. Their
target documentation is never overwritten; the source's only fills gaps.
{source: {key tuple}} of the target's current business-dictionary rows for
names ({(DB_UP, OBJ_UP)}): what a kept object already has documented.
Re-home the migrated objects' BD rows under their target DATABASE_NAME.
placement (_effective_placement): a source table left in its database
(reference) keeps its documentation there. keep_target: objects the
target already has (skipped, reused) -- their documentation there wins, the
source only fills parts the target lacks.
process_doc_copy_sql(src_db: str, dst_db: str, table: str, columns, pid_map: Dict[str, str]) -> List[str]
[DELETE, INSERT] copying the current rows of a process-documentation table
from the source store to the target, PROCESS_ID re-keyed. Pure. The DELETE ends
the target's current rows for those processes (a replace keeps its id).
Copy the business logic and EXPLAIN documentation of the processes this run
registered, then link their feature descriptions (needs the features in the
catalog: a process registered with data_strategy='none' gets them at its
first run, with tdfs4ds.link_feature_documentation(pid)).
migrate_objects(plan: Dict[str, Any], approvals: Optional[Dict[str, str]] = None, *, dry_run: bool = False, rebuild_vector_index: bool = True, progress: Optional[Callable[[str], None]] = None) -> Dict[str, Any]
Execute an approved migration manifest against the primary store.
Parameters¶
plan : dict
The manifest from :func:tdfs4ds.plan_migration.
approvals : dict[str, str] | None
{conflict_key: 'skip'|'replace'} for every conflict. Required whenever
the plan has conflicts; a replace that is still blocking (the
downstream-safety gate) is rejected.
dry_run : bool
Validate approvals and return the plan of record without writing.
rebuild_vector_index : bool
Rebuild the Teradata vector index after BD migration (default True; skipped
unless VECTOR_STORE_BACKEND == 'teradata').
progress : callable(str) | None
Per-step progress hook (for the SSE panel).
Returns¶
dict
Execution report: pid_map (source→primary), per-object outcomes,
skipped/replaced, and any post-migration downstream compile breakages.
_run_phases(plan, src, target_domain, objects, placement, pid_map, report, skip_keys, _obj_skipped, rebuild_vector_index, progress)
Execute the migration phases (primary-only; see migrate_objects).
Objects whose lineage Phase F refreshes: only those this run created.
A dataset counts only when Phase D built it (ok=True): a deferred one
(data_strategy='none'), one whose processes did not migrate, or a
failed build does not exist in the primary, and naming it used to abort
the whole refresh. A view whose creation failed is left out the same way.
PROCESS_ID of a process view already registered in the primary (or None).
{entity_col: type} for a source dataset (from its FS_V_*_ENTITY view).
build_dataset keyword arguments reproducing the source view's build.
The strategy (grouped or not), the join type and the point in time are read
from the source view's own DDL. A plain build_dataset call would silently
turn a grouped, fixed-date or TimeManager dataset into a current multi-join
one in the target store.
A TimeManager dataset is re-pointed to the manager's migrated copy when the plan placed one; otherwise it keeps referencing the source store's manager (cross-database), and the second value returned says so for the report.
{base feature: [offset ISO, ...]} of a parsed dataset, {} without offsets.
{column alias (upper): base feature (upper)} and {base: process id} of the offset columns.
(selected_features, feature_offsets) rebuilding spec's columns under their names. Pure.
base_names {FEATURE_ID: catalog name} (source store), pid_map
{source PROCESS_ID: target PROCESS_ID}. A column named like its feature
stays a plain process id (the call of a dataset without alias is unchanged);
any other name travels as the version's alias. None when a feature
name is unknown or a process was not migrated (the caller falls back).
{FEATURE_ID: FEATURE_NAME} of spec's features in the source store's catalog.
Rebuild a migrated dataset in the primary via build_dataset — NOT a view
copy. The source dataset view refs the store-local FEATURE_ID /
FEATURE_VERSION of the source; build_dataset regenerates them against
the primary's freshly-registered features and registers the primary dataset
catalog rows. The source feature→process map (from the source DDL) has its
process ids remapped source→primary. The strategy, join type and point in
time are carried over from the source view (_dataset_build_options).
Copy the FS_T_* feature-value rows for a process from the source store into the primary, remapping the store-local FEATURE_ID and preserving VALIDTIME history.
Best-effort fallback (used when recompute is not possible). Registers the entity + features in the primary first (assigning fresh FEATURE_IDs), then INSERTs the source values with each source FEATURE_ID mapped to its primary FEATURE_ID.