11. Logs¶

Every ingestion or replay leaves a row in the follow-up table (FS_FOLLOW_UP). It is the canonical record of what ran, when, with which filter, and how it ended. Read it whenever you need to answer "did that upload actually finish?" or "which runs failed last week?".

Schema¶

CREATE MULTISET TABLE {SCHEMA}.FS_FOLLOW_UP, FALLBACK, ... (
    RUN_ID         VARCHAR(255) NOT NULL,
    PROCESS_TYPE   VARCHAR(255) NOT NULL,
    PROCESS_ID     VARCHAR(255),
    START_DATETIME TIMESTAMP(6) WITH TIME ZONE NOT NULL,
    END_DATETIME   TIMESTAMP(6) WITH TIME ZONE,
    VALIDTIME_DATE TIMESTAMP(6) WITH TIME ZONE,
    APPLIED_FILTER JSON(2000),
    STATUS         VARCHAR(2048)
)
PRIMARY INDEX (RUN_ID, PROCESS_ID);

Column	Meaning
`RUN_ID`	UUID generated per `upload_features` / `run` call
`PROCESS_TYPE`	`'RUN'`, `'UPLOAD'`, etc. — what triggered the row
`PROCESS_ID`	The registered process (NULL for some bookkeeping rows)
`START_DATETIME`	When the run opened (always set)
`END_DATETIME`	When the run closed (NULL while still running)
`VALIDTIME_DATE`	`FEATURE_STORE_TIME` at the time of the run (NULL = current)
`APPLIED_FILTER`	JSON snapshot of the `FilterManager` row in use
`STATUS`	`'RUNNING'`, `'COMPLETED'`, or `'FAILED,…'` (with the error tail)

A view FS_V_FOLLOW_UP over the table provides LOCK ROW FOR ACCESS reads — use it for dashboards or BI tools.

Reading the report¶

tdfs4ds.follow_up_report()  # most recent runs first

The output is a teradataml DataFrame; filter with normal SQL or pandas:

recent_failures = (
    tdfs4ds.follow_up_report()
    .to_pandas()
    .query("STATUS.str.startswith('FAILED')", engine='python')
)

Status lifecycle¶

followup_open()           →  STATUS = 'RUNNING'
   │
   ├──  success           →  STATUS = 'COMPLETED'
   └──  exception         →  STATUS = 'FAILED,<message>'

A row that has been 'RUNNING' for hours is a stale row — the Python process crashed (or the session was killed) before followup_close could fire. Two recovery patterns:

Re-run. tdfs4ds.run(process_id) opens a new RUN_ID and ingests again; if the underlying MERGE is idempotent the old 'RUNNING' row is cosmetic.
Manual cleanup. Update the stale row to 'FAILED,manual cleanup' so reports stay clean. Stale rows do not block anything; they are just misleading on dashboards.

Known quirk

followup_close re-raises before the UPDATE executes when called with a FAIL* status. This means a failing ingestion path can leave the follow-up row in 'RUNNING' rather than 'FAILED,...'. Treat any old 'RUNNING' row as evidence of an unfinished or crashed run and apply one of the recovery patterns above. See the "followup_close" note in CLAUDE.md for the full background.

When to widen the audit trail¶

If STATUS truncates your error message (the column is VARCHAR(2048)), log the full traceback to a separate sink in your application — the follow-up table is intentionally narrow so it stays cheap to scan.

Learn more¶

Inspect & monitor — full follow_up_report parameter reference, plus process_catalog, feature_catalog, dataset_catalog read helpers.
Temporal model — the write lifecycle and why a 'RUNNING' row left over after a crash is the canonical signal of an incomplete ingestion.