Skip to content

fs-upload

Engineer features, persist them as a permanent view, and register them in the feature store with upload_features.

Source file

Lives at tdfs4ds/skills/fs-upload/SKILL.md.

name: "Feature Store: Upload" description: "Generate Python code to engineer features from a Teradata source, persist as a permanent view, and register in the tdfs4ds feature store with upload_features. Invoke when the user wants to create or update a feature engineering process." argument-hint: "[source-table][entity-column] [feature-columns]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

Generate Python code to engineer features from a Teradata source, persist the logic as a permanent view, and register + upload the features to the tdfs4ds feature store.

User context: $ARGUMENTS

Before / After Skill Why
Before fs-setup Open the Teradata context and activate a data domain.
Optional fs-filter Segment a large source into iterative passes with FilterManager.
After fs-document LLM-generated business description + EXPLAIN score 1–5 for the registered process.
After fs-analyze If USER_SCORE is 1–3, run the full scalability analysis and propose a rewrite.
After fs-rollout Backfill features across a date range with TimeManager.

Demos: demos/notebooks/01 - Feature Engineering.ipynb, 02 - Register Your Features.ipynb.

Ask for any missing values before generating code: - Source database and table — where the raw data lives. - Entity column(s) — the primary key. A list of column names, e.g. ['CUSTOMER_ID'] or ['CUSTOMER_ID', 'EVENT_DATE']; the SQL types are read from the source. Do not ask for types. - Feature columns — list of derived columns to register as features. - View name — name for the permanent Teradata view. - Metadata — optional dict with descriptive tags. - Dataset view name — optional; if provided, also creates a ready-to-use dataset view.

Pattern A — SQL query, the view created by tdfs4ds (preferred)

Use for any feature logic written in SQL (aggregations, PERCENTILE_CONT, window functions). upload_features compiles the query (EXPLAIN), refuses a target view that already exists, creates the view and registers it.

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # teradataml context from TDFS4DS_TD_HOST / _USERNAME / _PASSWORD
tdfs4ds.connect(database='<DATABASE>')
tdfs4ds.select_data_domain('<DATA_DOMAIN>')   # or set TDFS4DS_DATA_DOMAIN in .env
import teradataml as tdml

feature_sql = """
LOCKING ROW FOR ACCESS
SELECT
    <ENTITY_COLUMN>
,   <AGGREGATION_EXPRESSION> AS <FEATURE_NAME>
FROM <SOURCE_DATABASE>.<SOURCE_TABLE>
GROUP BY <ENTITY_COLUMN>"""

process_id = tdfs4ds.upload_features(
    df=None,                                    # the source is the query
    sql_query=feature_sql,
    target_view_name='<DATABASE>.<VIEW_NAME>',   # must not exist yet
    entity_id=[<ENTITY_COLUMNS>],    # list of key columns — SQL types read from the source
    feature_names=['<FEATURE_NAME>'],
    metadata={'project': '<PROJECT_NAME>'},
    return_process_id=True,                     # keep the id for run() / roll_out()
    # run_now=False,                            # register only; compute later under a TimeManager (fs-rollout)
    # time_manager=tm,                          # attach the TimeManager the view reads: run(pid) then uses it
    # close_absent_entities=True,               # entities the view no longer returns lose their validity
    # dataset_time_manager=True,                # with time_manager=tm + dataset_view_name: the dataset follows tm (tm.update(i) moves it)
    # track_entity_refresh=True,                # admin record of when each entity was last confirmed (fs-inspect: entity_refresh)
    # dataset_view_name='<DATASET_VIEW>',       # also build a dataset of these features
    # filtermanager=fm,                         # segmented ingestion (fs-filter)
    # document=False, explain=False,            # override the .env defaults of the AI steps for this call
)

The source can also be the name of an existing view (upload_features('<DATABASE>.<VIEW_NAME>', ...)) or a teradataml DataFrame (Pattern B).

Pattern B — teradataml DataFrame methods + crystallize_view (fallback)

Use only when the feature transformation is expressed through the teradataml DataFrame API. Those build a temporary view that disappears when the context is removed; crystallize_view persists it as a permanent database object.

# ... connection setup same as Pattern A ...

from tdfs4ds.utils.lineage import crystallize_view
from tdfs4ds import upload_features

# Chained teradataml DataFrame methods produce a *temporary* Teradata view.
# crystallize_view persists it as a permanent database object before upload.
source_data = tdml.DataFrame(tdml.in_schema('<SOURCE_DATABASE>', '<SOURCE_TABLE>'))
df_features = source_data.groupby('<ENTITY_COLUMN>').agg({
    '<VALUE_COLUMN>': ['sum', 'mean'],
})
feature_names = ['sum_<VALUE_COLUMN>', 'mean_<VALUE_COLUMN>']

df_features_proc = crystallize_view(
    df_features,
    view_name='<VIEW_NAME>',
    schema_name='<DATABASE>',
)

upload_features(
    df_features_proc,
    entity_id=[<ENTITY_COLUMNS>],
    feature_names=feature_names,
    metadata={'project': '<PROJECT_NAME>'},
    dataset_view_name='<DATASET_VIEW>',  # ds_<base_name>
)

History in the source: valid_time_column

When the source already holds one row per entity and valid time (several rows per entity), give the process the column that carries that time. One run then writes the whole history; no TimeManager and no roll_out are needed.

upload_features(
    '<DATABASE>.<HISTORY_VIEW>',
    entity_id=['<ENTITY_COLUMN>'],
    feature_names=['<FEATURE_1>', '<FEATURE_2>'],
    valid_time_column='<EVENT_TS>',   # DATE or TIMESTAMP; not an entity column, not a feature
)
# change or switch off later: tdfs4ds.set_process_options(pid, valid_time_column='<COL>')  /  valid_time_column=False
  • A row's value is valid from its time until the entity's next row (the last one until 9999-01-01). Equal consecutive values are merged.
  • (entity, column) must be unique; a NULL or duplicated valid time refuses the batch and the error names the column.
  • Each entity's history is replaced from its earliest valid time in the load onward and kept before it: a re-run is a no-op, an incremental load rewrites only its own tail, a backdated correction needs a full load.
  • FEATURE_STORE_TIME and the TimeManager date no longer set the values' valid time. Not for a UAF ART view.
  • Prefer it to roll_out when the dates are in the data. Keep a TimeManager when the logic must be recomputed as of each business date (see fs-rollout).

Types: force_varchar_length and a refused re-registration

  • force_varchar_length is a process setting (not given = as recorded for the view's process, else 1024; 0 or 'none' = exact lengths). A new text feature joins the stored type of a registered text feature of the process that holds it, else the new text features share one VARCHAR(ceil(longest / N) * N) per character set (at most two text tables). run() / roll_out() take force_varchar_length=None = as recorded. Change it: tdfs4ds.set_process_options(pid, force_varchar_length=N) (or 'none'); read it in get_process_info(pid)['FORCE_VARCHAR_LENGTH'].
  • A registered feature keeps its stored type. The new source type is compared by whether the values fit (shorter VARCHAR, int into a wider int / FLOAT / DECIMAL, smaller DECIMAL, lower TIMESTAMP precision): a fit changes nothing.
  • When it does not fit, upload_features registers nothing and returns None: read the log. It offers (1) CAST the column in this view to the stored type; (2) widen the stored type for all producers with plan = tdfs4ds.plan_type_change(types={'<FEATURE>': '<TYPE>'}, entity=['<ENTITY_COLUMN>'], data_domain='<DOMAIN>'), tdfs4ds.write_relocation_script(plan, 'type_change.py'), a dry run, then --apply (see fs-relocate); (3) for incompatible types, another feature name. run() only warns and writes with the stored type.
  • A VARCHAR source longer than its stored type is reported as a warning per batch (the batch still completes).

Rules

  • Value types: a feature keeps the type the view computes (exact length, precision, character set); text is rounded up to force_varchar_length. CLOB, BLOB, JSON and XML are refused at registration (0.3.5.17: nothing is registered, the call returns None): they cannot be compared, so an unchanged value would open a new period at every run. Extract scalar fields in the view (JSONExtractValue, XMLEXTRACT, SUBSTR) or CAST short text to VARCHAR. PERIOD and ST_GEOMETRY are accepted but cannot be aggregated (see fs-dataset).

  • Prefer Pattern A — SQL passed as sql_query= with target_view_name=: tdfs4ds checks it and creates the view. A failed check is logged with its fix and the call returns None (no traceback): read the log.

  • df is the first, required parameter even with sql_query=: pass df=None there (omitting it is a TypeError).
  • Re-running a process is cheap and safe: the temporal MERGE writes nothing for a value that did not change (no new row, the period in force stands) and closes/opens a period only where a value moved, so a 15-minute schedule does not inflate the history. "Unchanged" means bit-identical: a float aggregate recomputed in another order can differ in its last digits and then opens a period.
  • Only use crystallize_view (Pattern B) when the teradataml DataFrame API is unavoidable and produces a temporary view.
  • entity_id is a list of the key columns, e.g. ['CUSTOMER_ID'] or ['CUSTOMER_ID', 'EVENT_DATE']. The SQL types are read from the source with their exact length and character set: never write them. (A dict of types is an early-version form, still accepted only to override an inferred type.)
  • feature_names must be a subset of the view's columns, excluding the entity column(s).
  • Never register a 1:1 copy of a raw source column (e.g. uploading claims_raw.amount unchanged). That is pure data duplication with no governance value. Join the source table in the training query instead and append the column to column_names_X.
  • upload_features registers then calls run(process_id) by default. Pass run_now=False to register only: the source is still checked with EXPLAIN before anything is written, and the process is computed later — under a TimeManager with run(..., time_manager=tm) or roll_out (see fs-rollout). Use it for any time-managed process: an immediate real-time run would store values valid from today, after the business dates.
  • The AI steps are configuration. With an instruct model configured, upload_features documents the process and analyses its EXPLAIN plan by default; the defaults come from .env: TDFS4DS_AGENTIC_FEATURES (master switch), TDFS4DS_UPLOAD_DOCUMENT, TDFS4DS_UPLOAD_EXPLAIN, TDFS4DS_UPLOAD_OPTIMIZE_QUERY (off), TDFS4DS_UPLOAD_VECTOR_INDEXING (off). An argument in the call (document=, explain=, optimize_query=, vector_indexing=) always wins. Without an instruct model they are skipped silently.
  • Under a scheduler pass on_failure='raise' (to upload_features, run, roll_out): a failed ingestion then raises FeatureIngestionError. The default 'continue' records the failure in the follow-up table and returns normally.
  • time_manager=tm attaches the TimeManager the view reads (recorded in FS_TIME_MANAGER, like filtermanager=), so run(process_id) computes at its business date with no argument. The view's lineage is checked first: under tdfs4ds.TIME_MANAGER_CHECK='error' a manager it does not read is refused and nothing is registered (a view created from sql_query is dropped again); under the default 'warn' it is a warning and the pairing is not recorded. Without time_manager, registration is unchanged before 0.3.6.0.
  • From 0.3.6.0 the manager is found without time_manager=: upload_features walks the view's lineage and records the one TimeManager it reads (several: a warning, none recorded; tdfs4ds.TIME_MANAGER_DETECT = False turns it off). When the data domain has a clock (tdfs4ds.set_domain_time_manager(tm)), a process reading it gets its own soft clone <view>_TM and its view is rewritten to read it (soft_clone=False keeps the clock, soft_clone=True clones any manager). See fs-rollout for roll_out(data_domain=) and run([...], at=).
  • storage_tag='SALES' for a process that must not share a storage table (the parallel planner serialises processes writing the same FS_T_* table; also lets its primary index change alone later). Recorded in FS_FEATURE_PLACEMENT before the first run. A feature already registered elsewhere is refused: see fs-relocate.
  • close_absent_entities=True when entities can leave the source (closed accounts, delisted products). By default a run writes only the entities its view returns: one that disappears keeps its last values, and current datasets keep showing it. With the option (a process property in FS_PROCESS_OPTIONS, applied by every later run / roll_out), each run ends their validity from its valid time on; history before is kept and an entity that comes back gets a new period. A run that returned 0 rows closes nothing. With a filtermanager, every filter column must be an entity column (else the registration is refused). Change it later with tdfs4ds.set_process_options(pid, close_absent_entities=...); run/roll_out(..., close_absent_entities=...) override it for one call.
  • Manager-driven processes (TimeManager, FilterManager) read the manager's one-row view with a scalar subquery (WHERE col = (SELECT ... FROM <manager view>)), never a join: see fs-filter for what the execution plan reads.
  • Pass return_process_id=True when the generated script needs the PROCESS_ID afterwards — for a later run(process_id) or to assemble the process_list for roll_out. It returns the UUID string instead of the dataset, so the caller does not have to look it up in the process catalog: 
    process_id = upload_features(..., return_process_id=True)
roll_out([process_id], time_manager=tm)
    Default is False (returns the dataset, unchanged).
  • Always pass dataset_view_name to build a denormalised dataset view alongside feature registration. Name it by replacing the vw_feat_ prefix with ds_ (e.g. vw_feat_savings_dist → ds_savings_dist).
  • upload_features always uses the classic multi-join (grouped=False) by default when it builds a dataset view — pass grouped=True at upload time for wide datasets (many features sharing the same source table) to reduce JOIN count via MAX(CASE WHEN) pivoting instead of building the dataset separately afterward with build_dataset(..., grouped=True).
  • Always include LOCKING ROW FOR ACCESS as the first line of every REPLACE VIEW … AS body. This tells Teradata to use access locks, preventing read queries from blocking concurrent writers. Omitting it causes read-lock contention on busy feature views.
  • When features will feed hyper-segmented ML (one model per partition, e.g. per region): include the partition column (e.g. region) in the view SQL and in feature_names. This registers it as a feature so build_dataset() includes it naturally — no manual JOIN to the source table needed at training time. Use partitioning= separately if you want to physically co-locate rows by that column for efficient partition elimination.
  • partitioning= may only use entity columns (and FEATURE_ID). RANGE_N refuses a DECIMAL column (Error 3732: cast it to INTEGER in the view) and DATE bounds on a TIMESTAMP column (Error 5714: use TIMESTAMP '...' bounds). Since 0.3.4.3 upload_features checks the definition on a volatile table before registering anything, and returns None with the reason when it is refused.
  • Ratios: CAST to FLOAT before dividing. A DECIMAL division keeps the operands' scale, so SUM(dec)/COUNT(*) or an AUC computed in DECIMAL is silently rounded (seen: every KPI rounded to 2 digits). CAST(x AS FLOAT) / y.
  • Document the source tables before the first upload_features (set_object_documentation or the business-dictionary loaders, see fs-document): the process documentation is written at registration from what the LLM knows then, and is not regenerated later without force_update=True.
  • A model scoring process (BYOM / ONNX) is an ordinary process: a view calling ONNXPredict (or PMMLPredict, H2OPredict...) from the database where BYOM is installed (often mldb or TD_MLDB: check with HELP DATABASE), reading the model from the model table. Several models over time: store the training date with the model (save_byom(..., additional_columns=...)) and pick, in a sub-query, the latest one trained at or before the TimeManager's business date, so each roll-out date scores with the model available then. Feed the model COALESCEd inputs (NULLs from lags). Since 0.3.5.15 the documentation reads the model table's metadata (fs-document).
  • Script / tdstone2 not available? SELECT * FROM SCRIPT(...) failing with Error 9854 means the Script table operator is not licensed on that system: use the BYOM path above (train in the client, export ONNX, score in the database) and check parity between the in-database scores and the client model on a sample.
  • Monitoring KPIs as a process: a view computing AUC / lift per business date (scores joined to the realised labels) registered with a constant entity (e.g. 'propensity' AS use_case, entity_id=['use_case']) and the same TimeManager: the KPI history is then a feature history, with lineage and freshness.
  • The primary index and partitioning are recorded with the process; passing other values when registering the same view again only logs a warning. Change them with tdfs4ds.plan_relocation(process_id=..., primary_index=..., partitioning=...).