`upload_features` — register and ingest¶

tdfs4ds.upload_features is the single entry point for going from a Teradata view to a registered, persisted, time-travel ready feature set.

Signature¶

tdfs4ds.upload_features(
    df,
    entity_id,
    feature_names,
    metadata = {},
    primary_index = None,
    partitioning = '',
    filtermanager = None,
    entity_null_substitute = {},
    force_compute = True,
    force_varchar_length = None,
    dataset_view_name = None,
    optimize_query = None,
    update_view = True,
    document = None,
    max_candidates = None,
    cascade_redocumentation = None,
    vector_indexing = None,
    sql_query = None,
    target_view_name = None,
    art_sql = None,
    art_layers = None,
    grouped = False,
    return_process_id = False,
    run_now = True,
    on_failure = None,
    raise_on_failure = None,
    check_upstream = False,
    strict_upstream = False,
    explain = None,
    time_manager = None,
    close_absent_entities = False,
    storage_tag = None,
    track_entity_refresh = False,
    dataset_time_manager = False,
    valid_time_column = None,
    soft_clone = None,
)

The arguments can be passed by position in this order; pass them by name.

Argument	Type	Notes
`df`	`teradataml.DataFrame` or `'schema.view'` `str`	A DataFrame wrapping a permanent view, or a fully-qualified view name (schema mandatory). Use `crystallize_view` first if a DataFrame wraps a temporary view.
`sql_query`	`str`	SQL-first path: a `SELECT` defining the source. Requires `target_view_name`. Takes precedence over `df`.
`target_view_name`	`str`	`'schema.view'` where the view for `sql_query` is created (schema mandatory; must not already exist).
`entity_id`	`list[str]`	The key column(s), e.g. `['CUSTOMER_ID']`. Types are read from the source with their exact length and character set. A single string also works; a dict of types only overrides an inferred type.
`feature_names`	`list[str]`	Subset of `df.columns` excluding the entity columns.
`metadata`	`dict`	Free-form metadata stored on the process catalog row.
`filtermanager`	`FilterManager`	Drives segmented ingestion. See FilterManager.
`dataset_view_name`	`str`	If set, builds a denormalised dataset view in the same call. Convention: `vw_feat_<x>` → `ds_<x>`.
`document`	`bool` (default `None` = `TDFS4DS_UPLOAD_DOCUMENT`, `True` unless configured)	When an instruct model is configured, document the process automatically. Silently skipped otherwise.
`explain`	`bool` (default `None` = `TDFS4DS_UPLOAD_EXPLAIN`, `True` unless configured)	Part of `document`: also run the LLM analysis of the view's EXPLAIN plan. `False` documents without it and keeps any stored analysis.
`optimize_query`	`bool` (default `None` = `TDFS4DS_UPLOAD_OPTIMIZE_QUERY`, `False` unless configured)	Run the query optimizer on the registered process view.

The AI steps (document, explain, optimize_query, vector_indexing, cascade_redocumentation) take their default from the configuration, so a .env can turn the agentic side on or off without editing any call. TDFS4DS_AGENTIC_FEATURES=false turns them all off. An argument passed explicitly always wins. See Environment variables.

Argument	Type	Notes
`update_view`	`bool`	When `True` and an optimization improves the score, `REPLACE VIEW` is issued so subsequent runs use the rewritten SQL.
`art_sql`	`list[str]`	UAF ART view (opt-in). A list of `EXECUTE FUNCTION ... INTO VOLATILE ART(<name>)` statements materialised per partition before ingestion so the feature view can read a secondary ART layer. Registers the process as `PROCESS_TYPE='uaf art view'`. Requires `art_layers`. See UAF ART-layer features.
`art_layers`	`list[str]`	Required with `art_sql`: the declared `LAYER(...)` names the feature view reads (e.g. `['ARTFITMETADATA']`). Used for documentation and lineage.
`return_process_id`	`bool` (default `False`)	Return the registered process's `PROCESS_ID` (a UUID `str`) instead of the dataset, so you can hold it for a later `run(process_id)` / `roll_out([process_id], …)` without looking it up in the process catalog. Mirrors `run(return_dataset=…)` — the flag toggles what is returned, it is not a tuple. The default return is already `None` unless `tdfs4ds.BUILD_DATASET_AT_UPLOAD` is `True`, so in the common case nothing is given up.
`time_manager`	`TimeManager` (default `None`)	The TimeManager the view reads, recorded at registration so `run(pid)` computes at its business date. A manager the view does not read follows `TIME_MANAGER_CHECK` (warning, or refusal under `'error'`). An explicit manager is paired as before 0.3.6.0, even the data domain's clock. `None` (0.3.6.0): the view's lineage is walked and the manager it reads is recorded; the data domain's clock (`set_domain_time_manager`) gives the process its own soft clone. `TDFS4DS_TIME_MANAGER_DETECT=false` turns detection off. See One clock per data domain.
`soft_clone`	`bool` (default `None`)	0.3.6.0. `None`: a soft clone `<view>_TM` for a process whose view reads its data domain's clock (found by detection). `True`: a clone of any manager. `False`: never.
`primary_index` / `partitioning`	`list[str]` / `str`	Primary index and partitioning of the storage tables (default: the entity columns, no partitioning). Recorded with the process.
`entity_null_substitute`	`dict`	Per-column replacement of NULL entity keys (`COALESCE`); an uncovered NULL key refuses the batch.
`force_compute`	`bool` (default `True`)	Recompute even when the follow-up records the run COMPLETED at this valid time.
`on_failure` / `raise_on_failure`	`'continue'` / `'stop'` / `'raise'`, or `bool`	What a failed FilterManager segment does; `'raise'` attempts every segment then raises `FeatureIngestionError`.
`check_upstream` / `strict_upstream`	`bool`	Refuse the run when an upstream process FAILED (also RUNNING) at this valid time.
`grouped`	`bool` (default `False`)	Strategy of the dataset built with `dataset_view_name` (see `build_dataset`).
`storage_tag`	`str`	Store this process's features in a table of their own (`__<TAG>`); see Relocating features.
`close_absent_entities`	`bool` (default `False`)	A process option, recorded in `FS_PROCESS_OPTIONS` and applied by every later `run()` / `roll_out()`. When `True`, each run ends the validity of the stored entities the process no longer produces, from the run's valid time on. See Entities that disappear.
`track_entity_refresh`	`bool` (default `False`)	A process option, recorded in `FS_PROCESS_SETTINGS` and applied by every later `run()` / `roll_out()`. The ingestion writes nothing when a value is unchanged, so a feature row only tells when a value last changed; with the option on, each batch also records in an admin table (`FS_R_*`, never a feature, in no catalog) that the process confirmed each entity. Read with `tdfs4ds.entity_refresh(pid)` / `tdfs4ds.entity_refresh_summary(pid)`; see Entity refresh. One extra `SELECT DISTINCT` and MERGE per batch; a failed write is a warning and the run stays `COMPLETED`. `set_process_options(pid, track_entity_refresh=True)`, `run(pid, track_entity_refresh=...)` and `roll_out(..., track_entity_refresh=...)` change it.
`dataset_time_manager`	`bool` (default `False`)	A process setting (`FS_PROCESS_SETTINGS`) applied by every later `run()` / `roll_out()`. When `True` and the process has a TimeManager (`time_manager=`, else the recorded one), the dataset of `dataset_view_name` is built with `build_dataset(time_manager=...)`, grouped or not: it shows the manager's business date and `tm.update(i)` moves it without a re-run. Without a TimeManager: a warning and the usual current dataset. A dataset that already exists at `CURRENT VALIDTIME` is not rebuilt silently; the warning names `tdfs4ds.rebuild_dataset(name, time_manager=..., replace=True)`. `set_process_options(pid, dataset_time_manager=True)` and `run(pid, dataset_time_manager=...)` change it.
`run_now`	`bool` (default `True`)	`False` registers the process without executing it — catalog rows written and view created, but `run()` is not called and no feature values are computed. For a lazy "declare now, compute later" flow; pair with `return_process_id=True` and drive it with `run(process_id)` / `roll_out(...)`. The source is validated with `EXPLAIN` before anything is registered (see below). Caveat: `register_entity`/`register_features` and `FS_T_` creation happen inside* `run()`, so until the first run the feature catalog has no entry for these features and their tables do not exist. Dataset documentation is skipped too.

Validation when `run_now=False`¶

With run_now=True the run itself proves the source executes. When the run is skipped nothing else does, so the source is EXPLAINed first — before any catalog row is written, so an invalid source never leaves a registered process behind. On failure nothing is registered: a prescriptive message is logged and None is returned, no traceback.

For a UAF ART view only the first art_sql statement is EXPLAINed, and the ART is not materialised. art_sql order is authoritative — a later statement may read an ART an earlier one creates — so without materialising, statements 2..n would fail for a reason unrelated to their own validity. The feature view is skipped for the same reason (it reads the ART → Error 3807).

EXPLAIN is not a guarantee

It proves the statement compiles and its objects are reachable. It does not prove the process will ingest: run() additionally resolves types (Gettdtypes / validate_feature_types) and rejects NULL entity keys not covered by entity_null_substitute, and the real ingestion statement is an unpivot CTAS, not SELECT *. A lazily-declared process can EXPLAIN clean and still fail on its first run.

Return value¶

`return_process_id`	Returns
`False` (default)	The dataset view — `None` unless `tdfs4ds.BUILD_DATASET_AT_UPLOAD` is `True`.
`True`	The `PROCESS_ID` string of the process this call registered.

Either way a recoverable input error (missing schema, target view already exists, SQL that will not compile) returns None after logging a prescriptive message — no traceback. A None return means "check the log".

process_id = tdfs4ds.upload_features(
    df, entity_id=['CUSTOMER_ID'],
    feature_names=['nb_transactions', 'total_amount'],
    return_process_id=True,
)
tdfs4ds.run(process_id)          # re-run later without a catalog lookup

Three ways to supply the source¶

upload_features accepts the feature source in three forms. The classic DataFrame path is unchanged; the other two are for pure-SQL users who don't build teradataml DataFrames.

1. A teradataml DataFrame (classic)¶

df = tdml.DataFrame(tdml.in_schema('my_db', 'FEAT_ENG_CUSTOMER'))
tdfs4ds.upload_features(df, entity_id=['CUSTOMER_ID'],
                        feature_names=['nb_transactions', 'total_amount'])

2. A view name (`'schema.view'`)¶

Already have a view? Pass its fully-qualified name. The schema is mandatory — a bare view name is rejected.

tdfs4ds.upload_features(
    'my_db.FEAT_ENG_CUSTOMER',                       # schema.view — schema required
    entity_id     = ['CUSTOMER_ID'],
    feature_names = ['nb_transactions', 'total_amount'],
)

3. A SQL query + a target view (SQL-first)¶

Give a SELECT and the schema.view to create. tdfs4ds validates the SQL with EXPLAIN, creates the view (it must not already exist), confirms it is executable, and uses it as the source. The created view is permanent — it becomes the process view re-run by run(process_id), so pick a stable name.

tdfs4ds.upload_features(
    sql_query = """
        SELECT CUSTOMER_ID,
               COUNT(*)    AS nb_transactions,
               SUM(AMOUNT) AS total_amount
        FROM source_db.TRANSACTIONS
        GROUP BY CUSTOMER_ID
    """,
    target_view_name = 'my_db.FEAT_ENG_CUSTOMER',     # schema.view to CREATE (schema required)
    entity_id        = ['CUSTOMER_ID'],
    feature_names    = ['nb_transactions', 'total_amount'],
)

Errors are logged, not raised

Recoverable input problems — a view name without a schema, a target_view_name that already exists, or SQL/a view that fails to compile or isn't executable — do not raise a Python traceback. Instead a clear message with a fix is written to the log and upload_features returns None. If a call returns None, read the log. For example, if the target view already exists you'll be told to DROP VIEW my_db.FEAT_ENG_CUSTOMER; first (or choose a different target_view_name) — the existing view is never silently overwritten.

UAF ART-layer features (`art_sql` / `art_layers`)¶

For a UAF (Unbounded Array Framework) feature whose value lives in a secondary ART layer — ARIMA fit-metrics (R_SQUARE/MAPE/AIC in ARTFITMETADATA), residual tests over ARTFITRESIDUALS, TD_PACF — pass art_sql (the INTO VOLATILE ART statements that materialise the ART) and art_layers (the layer names your view reads). tdfs4ds materialises the ART per partition, in-session, before ingestion, so the view can read the layer. See the dedicated guide: UAF ART-layer features.

Entities that disappear (`close_absent_entities`)¶

A run writes the entities its view returns and leaves every other stored entity as it was. So when a customer leaves the source, its last values stay valid (UNTIL_CHANGED) and the current datasets keep showing it.

Declare the process with close_absent_entities=True and every run also ends the validity of the entities the process no longer produces, at the run's valid time:

pid = tdfs4ds.upload_features(
    'my_db.FEAT_ENG_CUSTOMER',
    entity_id     = ['CUSTOMER_ID'],
    feature_names = ['nb_transactions', 'total_amount'],
    time_manager  = tm,
    close_absent_entities = True,
    return_process_id = True,
)
tdfs4ds.roll_out([pid], tm)        # nothing else to pass: the option belongs to the process

What it does, exactly:

The same rule as the values. A run states the truth from its valid time t onward: the entities it returns hold their values from t, the ones it no longer returns have no value from t. t is the TimeManager date under roll_out, FEATURE_STORE_TIME when set, else the run's real-time instant.
History is kept. The period before t is not touched; datasets read as_of an earlier date still return the entity.
An entity that comes back gets a new period; its absence stays a gap (a historical dataset shows two periods).
Idempotent. Re-running the same date changes nothing; replaying a history in date order gives the same result. As for the values, re-running an earlier date on its own rewrites everything from that date on: replay a history from its first date (roll_out(..., time_id_start=1, force_compute=True)).
Never on an empty run. A run whose view returned no row closes nothing (an empty or failed upstream must not end every entity) and records COMPLETED,0 rows.
A failed close fails the run: the batch is recorded FAILED with the database error and on_failure applies.
Segmented runs close only their own segment. With a FilterManager, every filter column must be an entity column, so that a segment can tell which stored entities are its own; otherwise the registration is refused (nothing registered, None returned).

Change the option of an existing process without re-registering it, or override it for one call:

tdfs4ds.set_process_options(pid, close_absent_entities=False)
tdfs4ds.run(pid, close_absent_entities=True)            # this call only
tdfs4ds.roll_out([pid], tm, close_absent_entities=False)

Re-registering a view with the default close_absent_entities=False switches the option off. get_process_info(pid)['CLOSE_ABSENT_ENTITIES'] shows it.

History from a column of the source (`valid_time_column`)¶

If the source holds one row per entity and valid time, register the process with the column that carries the time. One run writes the whole history.

tdfs4ds.upload_features(
    '<DATABASE>.<HISTORY_VIEW>',
    entity_id=['<ENTITY_COLUMN>'],
    feature_names=['<FEATURE_1>'],
    valid_time_column='<EVENT_TS>',
)

Meaning. The column is a DATE or TIMESTAMP of the source, neither an entity column nor a feature. A row's value is valid from that time until the entity's next row (the last row: 9999-01-01). The key (entity, column) must be unique; a NULL or duplicated valid time refuses the batch and the error names the column.
Rules. Equal consecutive values are merged. Each entity's history is replaced from its earliest valid time in the load onward and kept before it. A re-run is a no-op, an incremental load rewrites only its own tail, a backdated correction needs a full load. An incremental load can leave two adjacent periods with the same value at its start; a later full load merges them, and history datasets merge them too.
Valid time of the run. FEATURE_STORE_TIME and the TimeManager date do not set the values' valid time (a TimeManager can still bound the rows the view returns). The follow-up row is unchanged.
Options. close_absent_entities: an entity absent from a load has no value from the load's earliest valid time. track_entity_refresh records the load's latest valid time.
Types. DATE becomes midnight; TIMESTAMP(6) is truncated to the second; without a time zone the session time zone applies.
Refused. A UAF ART view; an entity column, a feature, a missing column or a non-date type at registration (returns None, nothing registered); a store without FS_PROCESS_SETTINGS. If that table exists but cannot be read, run() raises.
Change it. tdfs4ds.set_process_options(pid, valid_time_column='COL'), or valid_time_column=False to switch it off. get_process_info(pid)['VALID_TIME_COLUMN'] shows it; migration carries it over.
Compatibility. Opt-in. A client older than 0.3.5.0 running such a process ignores the setting.
Before 0.3.5.0. Load a TimeManager from the distinct dates and use roll_out, one run per date, replayed from the first date. Still valid when each date must be recomputed with the view's logic as of that date.

Types of features¶

force_varchar_length is a process setting, recorded in FS_PROCESS_SETTINGS (FORCE_VARCHAR_LENGTH). Not given (None, the default): the value recorded for the view's process, else 1024, so a re-registration never resets it. 0 or 'none': exact lengths. A new text feature takes the stored type of a registered text feature of the process that holds it; otherwise the new text features share one VARCHAR(ceil(longest / N) * N) per character set (longest over the new features; LATIN and UNICODE apart): at most two text tables per process. run() and roll_out() take force_varchar_length=None, meaning "as recorded" (1024 when nothing is recorded, i.e. a process registered before 0.3.5.1). Change it with tdfs4ds.set_process_options(pid, force_varchar_length=N) (or 'none'); get_process_info(pid)['FORCE_VARCHAR_LENGTH'] shows it. Migration carries it over.
A registered feature keeps its type. When the source type changes, it is compared to the stored one by whether the values fit: a shorter VARCHAR of the same character set or into UNICODE, an integer into a wider integer, FLOAT or a large enough DECIMAL, a smaller DECIMAL, a lower TIMESTAMP precision. A fitting type changes nothing.
A type that does not fit is refused. upload_features registers nothing and returns None; the log gives the alternatives with code. (1) CAST the column in this view to the stored type. (2) Widen the stored type for every process computing the feature: tdfs4ds.plan_type_change(types={...}, entity=[...], data_domain=...), then write_relocation_script, a dry run, --apply (see Changing the type of a feature). (3) For incompatible types (text and number, ...), give the column another feature name. run() / roll_out() log the same text as a warning and write with the stored type.
Truncation warning. When a registered VARCHAR feature's source is longer than its stored type, each batch reads the longest staged value and warns (it fits, or N values are too long: truncated in Teradata mode, refused in ANSI mode). The batch writes and stays COMPLETED.
Changing valid_time_column is never refused: no table definition is involved.

Full example¶

tdml.execute_sql("""
    REPLACE VIEW my_db.FEAT_ENG_CUSTOMER AS
    SELECT
        CUSTOMER_ID,
        COUNT(*)        AS nb_transactions,
        SUM(AMOUNT)     AS total_amount,
        AVG(AMOUNT)     AS avg_amount
    FROM source_db.TRANSACTIONS
    GROUP BY CUSTOMER_ID
""")

df = tdml.DataFrame(tdml.in_schema('my_db', 'FEAT_ENG_CUSTOMER'))

tdfs4ds.upload_features(
    df,
    entity_id     = ['CUSTOMER_ID'],
    feature_names = ['nb_transactions', 'total_amount', 'avg_amount'],
    metadata      = {'project': 'customer_analytics'},
    dataset_view_name = 'ds_customer_spend',
    document      = True,
)

What happens internally¶

Step	Module
Type detection on entity columns	`utils.info.Gettdtypes`
Entity & feature registration	`feature_store.entity_management`, `feature_store.feature_store_management`
Process registration	`process_store.process_registration_management`
Optional `simplify_query` + `optimize_query`	`agent.query_optimizer`
Optional `document_process_incremental`	`genai.documentation`
Volatile staging table + `MERGE` (or `UPDATE`+`INSERT`)	`feature_store.feature_data_processing`
Optional `build_dataset` + `document_dataset_incremental`	`dataset.builder`, `genai.documentation`
`COLLECT STATS` on every target feature table	`feature_store.feature_data_processing.apply_collect_stats`

A failing batch always re-raises after recording FAILED in the follow-up table. With a FilterManager, each filter is wrapped in try/except so a single failed partition does not abort the whole loop.

Rules¶

Always pass a permanent view. Temporary views vanish with the teradataml context — your registered process will be unable to re-run. Use tdfs4ds.utils.lineage.crystallize_view if your DataFrame chain produced a temporary view.
Never construct selected_features manually for the dataset. When dataset_view_name is set, tdfs4ds resolves feature versions automatically.
upload_features calls run() internally — this is intentional, not a bug. It exercises the exact same code path used by roll_out and re-execution, catching corner cases at registration time.
Entity-key types are preserved exactly. When entity_id is a list or string, the SQL type of each key is inferred from the source view with its full length and character set — a CHAR(14) key is created as CHAR(14) in the feature table, never truncated to CHAR(1). Pass a dict only when you want to override the inferred type (it is then used verbatim — include the length).
NULL entity keys are forbidden. A NULL in any entity / primary-index key column raises a ValueError during ingestion, because a NULL key produces a cartesian-product MERGE (severe skew and storage blow-up). Either filter the NULL rows out of the source view, or pass entity_null_substitute={'COL': <value>} to replace them via COALESCE.

Re-execution¶

tdfs4ds.run(process_id=process_id)

run re-applies the registered view at the current FEATURE_STORE_TIME. For historical replay, advance TimeManager and re-run — see TimeManager & roll_out.

upload_features — register and ingest¶