Relocating features¶

Since 0.3.3.30.

By default every feature of one (data domain, entity, value type, primary index, partitioning) lives in the same FS_T_* table, whichever process computes it. That has two consequences:

two processes of one entity write the same table, and a MERGE locks it, so the parallel planner runs them one after the other;
a table cannot be reorganised for one process only (another primary index, say) without touching the others.

A storage tag gives a feature (or all the features of a process) a table of its own. The table is the usual one with __<TAG> appended: FS_T_<hash>__SALES, with its FS_V_ view. Value type, primary index and partitioning stay encoded in the name as before: relocating features of three types creates three tagged tables. A type is converted only when you ask for it (see Changing the type of a feature).

Why the catalog is not edited by hand¶

Editing FEATURE_TABLE moves no data: the rows stay where they are, and the next time register_features registers the feature (a new feature, or a type change) it recomputes the table name and writes it over the catalog row. Any dataset built from the catalog meanwhile reads a table that holds nothing. The tag is therefore recorded in an optional table, FS_FEATURE_PLACEMENT (valid time, one row per (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME)), and the naming rule follows it. A store without the table, or a feature without a row, behaves exactly as before: the names and statements are byte-identical, and the table is checked once per store.

A new process in its own table¶

tdfs4ds.upload_features(df=df, entity_id=['CUSTOMER_ID'], feature_names=['amount_sum'],
                        storage_tag='SALES')

The placement is written before the first run, so the process starts in its own table. A feature that is already registered in another table, or shared with another process, is refused with the fix (relocate it first), because its history would stay behind.

Moving features that already have history¶

plan = tdfs4ds.plan_relocation(process_id='<pid>', table_tag='SALES')     # read only
print(plan['blocking'], plan['cost'])                                     # nothing is applied if blocking
tdfs4ds.relocate_features(plan, dry_run=True)                             # what each step would do
tdfs4ds.relocate_features(plan, dry_run=False)                            # apply
tdfs4ds.verify_relocation(plan)                                           # read only
tdfs4ds.write_relocation_script(plan)                                     # a script: dry run unless --apply

The unit is the feature triplet (all its versions); process_id= means all the features of that process. A feature another process also computes is refused unless include_shared=True (it moves for both). A producer with a follow-up row in RUNNING blocks the plan (force=True overrides that one, and unparseable datasets or views reading the old per-table view; never the others).

The steps, in order. A failure before the last one leaves the old table intact, and the plan says what to drop:

create the table and its view (a clone of the source table: same columns, same primary index and partitioning, so rows land on the same AMPs and partitions);
copy the rows with their history (NONSEQUENCED VALIDTIME INSERT ... SELECT);
check that the row counts per (FEATURE_ID, FEATURE_VERSION) are equal;
write the placement rows and point the catalog FEATURE_TABLE / FEATURE_VIEW at the new table (in place: a closed version of the row would be returned by FS_V_FEATURE_CATALOG to a session whose temporal qualifier is ANSIQUALIFIER, and the next run would write to both tables);
rewrite the datasets that read the features (below), each checked with EXPLAIN and a row count equal to the count before (the view is put back if it differs);
record the lineage, add the expected statistics to the new table, clear the batch-retrieval layout cache;
delete the moved rows from the old table: the only irreversible step, last.

What the plan states before anything runs¶

the rows and perm of the moved features (a share of DBC.TableSizeV by row count);
the free perm of the database: both copies exist until the delete, and a copy that does not fit is blocking;
whether the copy redistributes (the primary index or the partitioning changes) or stays AMP-local;
the datasets that will be rewritten, the views that read the old per-table view (blocking: point them at the new view), and the precondition below.

Changing the primary index or the partitioning¶

plan_relocation(..., primary_index=['CUSTOMER_ID'], partitioning='PARTITION BY ...') creates the tagged table with that distribution instead of the source table's. The unit is then the whole process: its staging table uses the process primary index, so a MERGE stays AMP-local only when all its features share the target index. The process's FOR_PRIMARY_INDEX / FOR_DATA_PARTITIONING are updated with the catalog, which is what the next run reads; registering the view again keeps them. A feature shared with a process that keeps another index is refused: one feature has one table. This replaces the "remove, register again, replay" the workflow assessment used to suggest for skew, and keeps the history.

Changing the type of a feature¶

A feature has one catalog entry, so one stored type and one table, whatever the number of processes computing it. upload_features refuses a re-registration whose source type no longer fits the stored one (see upload_features), and the message points here. The migration is a relocation with a new FEATURE_VALUE type:

plan = tdfs4ds.plan_type_change(process_id=pid)                       # the features of this process that no longer fit
plan = tdfs4ds.plan_type_change(types={'name': 'VARCHAR(2048) CHARACTER SET UNICODE'},
                                entity=['CUSTOMER_ID'], data_domain='SALES')
# or, inside a relocation: tdfs4ds.plan_relocation(process_id=pid, value_types={'name': 'VARCHAR(2048)'})
tdfs4ds.write_relocation_script(plan, 'type_change.py')               # dry run unless --apply

Same steps as a relocation: new FS_T_ table with the new type (same primary index and partitioning), the full history copied and verified, catalog FEATURE_TABLE / FEATURE_VIEW / FEATURE_TYPE updated, datasets rewritten and checked, statistics, old rows deleted last. verify_relocation(plan) checks the catalog type.
With process_id, the target is the narrowest type holding the stored type and every producer's type, so the processes that already fit keep fitting. plan["type_changes"] lists the casts the other processes need before their next run.
value_conversion is blocking when the stored values do not convert (a text narrowing is allowed when the longest stored value fits).
A tag is not needed; an existing tag is kept.

Rewriting a dataset's table references¶

tdfs4ds.rewrite_dataset_tables('DB.MY_DATASET')                          # dry run: shows the new DDL
tdfs4ds.rewrite_dataset_tables('DB.MY_DATASET', dry_run=False)           # points each feature at the catalog's table
tdfs4ds.rewrite_dataset_tables('DB.MY_DATASET', {12: 'DB.FS_T_x__SALES'}, dry_run=False)

The view is read back into its specification (storage table, feature id, process version, alias, join type, strategy, offsets, history window, TimeManager), the table of the features concerned is swapped and the view is written again under the same name; the catalog row and DATASET_ID are kept. A text replacement of the old table name would be wrong, because the old table still serves the features that do not move. A view that does not parse back to the same specification is refused and left alone. This is also the repair of dataset_reads_stale_table (a feature whose type changed moved to another table with no history copy).

Older clients (precondition)¶

A client older than 0.3.3.30 knows no placement. Running a feature that is already registered leaves the catalog alone (ingestion reads FEATURE_TABLE from it), so its values keep going to the tagged table. But when it registers the feature again (a type change, or the feature removed and registered anew), its register_features computes the untagged name, sets the catalog back to the old table and the values land there. This cannot be prevented from the new side, so every client that runs a relocated process must be at least this version. Detection and repair:

a new client compares the placement with the catalog whenever it registers features, warns naming the feature and the old table, and puts the tag back;
the workflow assessment's storage-level check reports relocation_reverted (the rows an older client wrote to the old table, or a catalog row pointing back at it);
running relocate_features on the same plan again moves those rows with a sequenced merge, so the periods of the rows already moved do not overlap.

Readers on older clients are not affected: build_dataset, federation and the exports read FEATURE_TABLE / FEATURE_VIEW from the catalog, and the rewritten dataset views keep their name, DATASET_ID and columns. Migration does not copy placements (the physical layout is a decision of each store).

In the admin app¶

The workflow assessment links the relevant findings (relocation_reverted, dataset_reads_stale_table) to a plan, a dry run and the generated script. The app never executes a relocation, the same rule as Remove objects: run the script with --apply.