tdfs4ds.migration.planner¶
Cross-store migration planner (v0.3.1.150) — the read-only phase.
plan_migration(seeds, source) resolves a mixed list of process / dataset /
feature names in a distant (lab) store, expands the upstream process
closure (the self-contained migration unit — the same closure the parallel
scheduler computes), inventories every object the closure touches (process views,
datasets, intermediate views, raw source tables), classifies them, detects
name conflicts against the primary (production) store, runs the replace
downstream-safety gate, and inventories the filter/time managers — returning
a JSON-serialisable manifest that requires approval before anything is written.
This phase never writes. It reuses the pure graph core of
:mod:tdfs4ds.scheduling.parallel_plan (_resolve_seeds / _build_nodeid_to_pid
/ _process_dependency_edges / _build_process_dag / _upstream_closure /
_compute_waves) — only the catalog reads are re-scoped to the source store.
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Uppercase, de-quoted, stripped — the canonical comparison form.
Split 'db.obj' / '"db"."obj"' / 'obj' into (db, obj) (raw,
de-quoted but original case preserved). db is '' when unqualified.
build_placement_map(objects, source_desc, primary_schema, primary_view_schema, renames=None, database_map=None) -> Dict[Tuple[str, str], Tuple[str, str]]
Build the source→primary placement map for a list of classified objects.
Parameters¶
objects : iterable of dict
Each {'database','object','object_type'} where object_type is one
of 'table' | 'view' | 'dataset' | 'process_view' | 'intermediate_view'
| 'source_table' (anything containing 'table' lands in the table
schema, everything else in the view schema).
source_desc : dict
The read-only descriptor {'schema','schema_view',...} of the source
store (used only as the default source DB when an object's DB is blank).
primary_schema : str
Primary table database.
primary_view_schema : str
Primary view database (view_schema()).
renames : dict, optional
{(SRC_DB_UP, SRC_OBJ_UP): NEW_OBJ_NAME} for objects renamed to avoid a
conflict. The destination DB is still chosen by object type.
database_map : dict, optional
Output of :func:normalise_database_map: the target database per kind
(and source database), over the defaults above.
Returns¶
dict
{(SRC_DB_UP, SRC_OBJ_UP): (DST_DB, DST_OBJ)}.
A source store's public view as a pandas frame (fail-open to empty).
A PRIMARY store public view (NOT federated) — used for conflict checks.
Walk every closure process view's DDL and inventory + classify all objects it
reaches: process_view / dataset / intermediate_view / source_table.
build_teradata_dependency_graph already skips FS-internal technical tables
and stops at dataset / process-view level, so the remaining tables are raw
sources and the remaining views are intermediate.
(DB_UP, OBJ_UP) of objects already present in the primary schema/view schema
and in the target databases of the plan (extra_dbs, from database_map).
Mark mapped source tables found at their target data_choice='reuse' and
drop their object conflict. Kept as a conflict (with a reuse_refused note)
when the target lacks a source column: the migrated views would not compile.
_build_downstream_gate(dst_db, dst_obj, otype, incoming_names, primary_graph, primary_proc_by_key, primary_ds_features_by_node) -> Dict[str, Any]
Compute the downstream feature-loss verdict for replacing a primary object.
{data_domain: clock} read from the SOURCE store's FS_DATA_DOMAIN_SETTINGS
(explicit schema, read-only), with the clock's current cursor. Fail-open: a store
without the table (before 0.3.6.0) or a domain without a clock gives nothing.
plan_migration(seeds=None, source=None, *, all_domains: bool = True, target_data_domain: Optional[str] = None, data_strategy: str = 'recompute', attach: bool = True, database_map: Optional[Dict[str, Any]] = None) -> Dict[str, Any]
_plan_migration(seeds=None, source=None, *, all_domains: bool = True, target_data_domain: Optional[str] = None, data_strategy: str = 'recompute', attach: bool = True, database_map: Optional[Dict[str, Any]] = None) -> Dict[str, Any]
Plan a cross-store migration (READ-ONLY — nothing is written).
Parameters¶
seeds : list[str] | None
Mixed process (id/view) / dataset / feature names in the source store.
None = the whole source store.
source : str | dict
The source store (a schema string or a {'schema','schema_view',...}
descriptor). Attached read-only for the duration when attach=True.
all_domains : bool
Consider all source data domains (default).
target_data_domain : str | None
Re-route every migrated feature under this DATA_DOMAIN in the primary
(avoids feature-name collisions with existing production domains). None
keeps each feature's source domain.
data_strategy : 'recompute' | 'copy' | 'none'
Default per-process data strategy (recompute recreates + re-runs, at
the business date of the process's TimeManager when it has one;
copy snapshots the FS_T_ values; none registers the processes
without computing, for a primary that replays its history itself with
roll_out - a value computed first cannot be preceded by earlier
valid times afterwards). Per-process overrides can be applied at
execution.
attach : bool
Attach source read-only if not already attached (default True).
database_map : dict | None
Where the migrated objects go, per kind: source_table,
intermediate_view, process_view, dataset. A value is a
database name, or {source_db: target_db} ('*' = any other).
Default: tables to the primary's table database, views to its view
database. A source table mapped to a database that already holds it
(same name, every source column present) is reused* there: nothing is
copied and the migrated views read it. The feature store's own tables
and views are not migrated and cannot be mapped.
Returns¶
dict
The migration manifest (see module docstring). Pass it to
:func:tdfs4ds.migrate_objects with an approval map.