Skip to content

fs-migrate

Promote work from one feature store (a lab) into another (production): a read-only plan, an approval, the migration, then the production history.

name: "Feature Store: Migrate" description: "Generate code to promote work from one feature store (a lab) into another (production): plan_migration (read-only plan with every object, manager and conflict), an approval report, migrate_objects (the only writer), then rebuild the production history with roll_out. Invoke when the user wants to move or promote processes, features or datasets between feature stores. SKIP when the user only wants to READ another store's features without copying anything — use fs-federation." argument-hint: "[source-store][seeds]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

Generate Python code that promotes selected processes and datasets from a source feature store (the lab) into the store the session is connected to (production).

User context: $ARGUMENTS

Step Skill Why
Before fs-setup Production is its own feature store: setup(database=<PROD>) once.
After fs-rollout Rebuild the history in production with roll_out from the first date.
After fs-dataset Datasets of processes that were not computed are built afterwards.
Alternative fs-federation Read the other store's features in place instead of copying.

Demos: demos/notebooks/P1 - Production feature store.ipynb, P2 - Promote lab work to production.ipynb.

The model

Two governed phases:

  1. plan_migration(seeds, source=...) only reads. It attaches the source store read-only, resolves the seeds (process view names, dataset names or feature names), adds everything they depend on (the closure: upstream processes, intermediate views, source tables, TimeManagers / FilterManagers), orders the processes in waves, and lists the conflicts with what production already holds (feature names, object names). Every conflict needs a decision: skip or replace; a replace that would drop a feature a production consumer needs is blocking.
  2. migrate_objects(plan, approvals) is the only writer. It refuses to run while a conflict has no decision or a replace is blocking. It re-creates the objects, registers each process in production with a new process id (report['pid_map']: source id → production id), migrates the managers (recreated, or unified onto an equivalent production one), rebuilds the datasets, the business dictionary and the lineage.

Ask for any missing values before generating code: - Source store (database of the lab) and production database (the session connects to it). - Seeds — the processes / datasets / features to promote; their dependencies are added automatically. - Target data domain in production (recommended: its own, so promoted features never collide with others). - Source tables: copy them into production, read them in place (production needs SELECT on them), or map their database to production's own source database (the same tables already there are reused). - Where the objects go (database_map=): per kind (source_table, intermediate_view, process_view, dataset), a database name or {lab_db: prod_db}. Default: tables in the store's database, views in its view database. Feature store tables and views are never migrated: they belong to the production store. - Data: compute now (data_strategy='recompute') or register only and rebuild a history ('none' + roll_out).

Emit the following pattern, substituting real values for every <PLACEHOLDER>:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # as the production owner (TDFS4DS_TD_*)
tdfs4ds.connect(database='<PROD_DATABASE>')     # the store that receives the work
import pandas as pd
from IPython.display import HTML

# 1. Plan (read-only)
plan = tdfs4ds.plan_migration(
    seeds=['<PROCESS_VIEW_1>', '<PROCESS_VIEW_2>', '<DATASET_NAME>'],
    source='<LAB_DATABASE>',
    target_data_domain='<PROD_DATA_DOMAIN>',
    data_strategy='none',                       # register only; the history is rebuilt below
    database_map={                              # optional; omitted kinds keep the store's databases
        'source_table': {'<LAB_SOURCE_DB>': '<PROD_SOURCE_DB>'},   # same tables there: reused, not copied
        'process_view': '<PROD_VIEW_DB>',
        'dataset': '<PROD_DATASET_DB>',
    },
)
print({k: plan[k] for k in ('n_processes', 'n_waves', 'n_objects', 'n_conflicts', 'n_blocking', 'unresolved_seeds')})

# Source tables are COPIED by default. To read a shared source in place instead:
for o in plan['objects'].values():
    if o['object_type'] == 'source_table' and o['database'].upper() == '<SOURCE_DATABASE>'.upper():
        o['data_choice'] = 'reference'

pd.DataFrame(plan['managers'])                  # TimeManagers / FilterManagers: recreate or unify
HTML(tdfs4ds.render_migration_report(plan, fmt='html'))   # the approval report a reviewer reads

# 2. Approve every conflict (none on a first promotion), then execute
approvals = {c['key']: 'skip' for c in plan['conflicts']}  # or 'replace' where intended (never a blocking one)
report = tdfs4ds.migrate_objects(plan, approvals=approvals, rebuild_vector_index=False)
assert not report.get('errors'), report.get('errors')
print(report['pid_map'])                         # lab process id -> production process id
print(report.get('deferred'))                    # e.g. datasets to build once their processes have run

# 3. Rebuild the history in production, from the first business date
from tdfs4ds.utils.time_management import TimeManager
tm = TimeManager(table_name='<TIME_TABLE_NAME>', schema_name='<PROD_DATABASE>')   # recreated by the migration
tdfs4ds.roll_out(list(report['pid_map'].values()), time_manager=tm)
# a domain with a clock (0.3.6.0): the clock comes with it (plan['domain_clocks'], report['domain_clocks'])
# tdfs4ds.roll_out(data_domain='<TARGET_DATA_DOMAIN>')

# 4. Same promotion from a scheduler: a script with the approvals and data choices written in it
path = tdfs4ds.write_migration_script(plan, approvals=approvals)

Rules

  • Plan, review, approve, then execute. Never call migrate_objects without reading the plan; it refuses to run while a conflict has no decision or a replace is blocking ({'error': 'approval_required', ...}).
  • Promote again = conflicts. A second plan of the same seeds reports every promoted feature and object as a conflict (17 in the demo); nothing is written without approvals.
  • Prefer data_strategy='none' + roll_out for time-managed processes. A history is built forward only: values computed now would be valid from today and could not be extended back to the business dates. Register only, then replay the whole timeline (see fs-rollout). 'recompute' runs each process once at the migrated TimeManager's current date. 'copy' is not implemented.
  • A data domain with a clock (set_domain_time_manager, 0.3.6.0): the processes' soft clones and their table of dates come with the objects; the clock view is recreated over that table at its cursor and declared for the target domain, so roll_out(data_domain=...) / run(data_domain=...) work in production. Before 0.3.6.0 the clock was not carried. A TimeManager that came with the objects is not copied again (one copy per clone would split the calendar).
  • Give production its own data domain (target_data_domain=): a feature is (data domain, entity, name), so the promoted features then never collide with the source's.
  • Source tables: data_choice='reference' keeps the production views reading the shared source in place (the production database needs SELECT ... WITH GRANT OPTION on it); the default copies the table.
  • Mapped source tables already in production are reused (same name, every lab column present): no copy, no conflict, the migrated views read them there (data_choice='reuse' in the plan). A target missing a column stays a conflict, named in the report (not reused: ... lacks column(s) ...).
  • Documentation comes with the objects: descriptions, sections and column descriptions (re-homed to each object's target database; a table read in place keeps its own), who wrote them, and the processes' business logic and EXPLAIN analysis (re-keyed to the production process ids). Feature descriptions are linked after the run (report['documentation']); for data_strategy='none', call tdfs4ds.link_feature_documentation(pid) after the first roll_out. An object kept in production (skipped, or a reused table) keeps its documentation; the lab's only fills what is missing.
  • Production process ids differ from the lab's: use report['pid_map'], never the lab ids.
  • rebuild_vector_index=False unless production has its own Vector Store prefix: Teradata Vector Store collections are named per prefix for the whole service, so two stores on the default prefix overwrite each other's collections.
  • The writer needs SELECT on the lab's catalogs and objects; it writes only in production. Verify a promotion by comparing a production dataset with the lab's at the same business date (the demo: 0 rows differ).