Failure handling¶

Added in v0.3.1.210.

A feature-engineering chain is a pipeline: level 3 reads what level 2 produced. So the question is not only "did this process fail?" but "what should happen to everything that depends on it?" This page covers how tdfs4ds reports a failed ingestion, how it records one, and how to stop a failure from quietly contaminating the levels above it.

What changed, and what did not¶

Before v0.3.1.210, a process segmented by a FilterManager absorbed a failed filter: the error was logged, the loop moved to the next filter, and run() returned normally. An orchestrator chaining waves saw a success. Separately, a defect made the FAILED status impossible to write, so the follow-up table showed only RUNNING and COMPLETED and gave no way to reconstruct what had happened.

Both are fixed. Two things now happen without you opting in:

a failed filter is recorded in the follow-up table with a FAILED status carrying the first line of the Teradata error;
an ingestion that stages zero rows is recorded as COMPLETED,0 rows rather than COMPLETED.

Default control flow is unchanged. on_failure defaults to 'continue', which behaves exactly as before, and the upstream guard is off by default. Upgrading does not change how your chains run.

Choosing a failure policy¶

run(), upload_features() and roll_out() accept on_failure:

Policy	Behaviour
`'continue'` (default)	Every filter is attempted, each failure is recorded, a summary is logged. No exception — the historical behaviour.
`'stop'`	The loop stops at the first failed filter, records it, and raises.
`'raise'`	Every filter is attempted and recorded — so the trace is complete — then a single exception is raised at the end.

import tdfs4ds

# Make a partial ingestion failure visible to the caller.
tdfs4ds.run(process_id, on_failure='raise')

raise_on_failure=True is accepted as a shorthand for on_failure='raise'. If you pass both and they disagree, the explicit on_failure wins and a warning is logged.

The exception is a FeatureIngestionError and carries the detail as data, so an orchestrator never has to parse a message:

try:
    tdfs4ds.run(process_id, on_failure='raise')
except tdfs4ds.FeatureIngestionError as e:
    print(e.process_id, e.n_failed, "of", e.n_total, "filters failed")
    for f in e.failures:
        print(" filter", f["filter_id"], "->", f["error"])

A process without a FilterManager always raises

on_failure governs what happens to the run when some of its filters failed — a question that only exists when the work is segmented. An unsegmented ingestion has no next filter to move to, so it propagates its error regardless of the policy.

Stopping a failure from contaminating the levels above¶

A failed upstream does not make a downstream view fail. The view still compiles and still returns rows — empty or stale ones. The downstream process ingests them, succeeds, and records its own COMPLETED, which the skip logic then treats as final.

check_upstream=True refuses to compute when an upstream process has failed at the current valid time:

tdfs4ds.run(process_id, check_upstream=True)

Nothing is computed and no follow-up row is written. The log names the failing upstream processes and states both ways forward: fix the upstream and re-run, or proceed deliberately with check_upstream=False.

The test is "no upstream failed", not "every upstream completed" — a weekly reference process feeding a daily one legitimately has no row for today. Pass strict_upstream=True to also refuse when an upstream is still RUNNING.

You can ask the same question without running anything:

verdict = tdfs4ds.check_upstream_health(process_id)
print(verdict["ok"], verdict["verified"])
print(verdict["explanation"])

This guard needs the lineage graph¶

It answers by reading the persisted lineage graph, so it is only as good as that graph. It reports ok and verified separately, because "no problem found" and "nothing could be checked" are different answers:

Situation	`ok`	`verified`	What you are told
Graph empty or unreadable	`True`	`False`	run `rebuild_lineage_graph(all_data_domains=True)`
Process absent from the graph	`True`	`False`	run `rebuild_lineage_graph(objects=['<view>'], check=True)`
Process known, no upstream	`True`	`True`	it reads source objects only — nothing to check
Upstream failed	`False`	`True`	the run is refused, with the failing processes named

The first two cases do not block — missing information is not evidence of a problem — but they are never silent, because reporting "nothing to check" would read as approval from a check that never ran.

Refresh lineage as part of normal operations

If you rely on check_upstream, treat rebuild_lineage_graph() as routine maintenance, alongside documentation. A process registered after the last refresh is not yet protected.

Orchestrating waves¶

The script produced by generate_execution_script takes its own wave-level wave_on_failure:

Policy	Behaviour
`'skip_descendants'` (default)	Skips only the failed process's downstream closure; unrelated branches still run.
`'continue'`	Runs every wave, reports failures at the end.
`'stop_after_wave'`	Finishes the current wave, then stops.
`'raise'`	Same, ending with a `RuntimeError`.

plan = tdfs4ds.analyze_parallel_plan()
path = tdfs4ds.write_execution_script(plan, wave_on_failure='skip_descendants')

The generated worker runs each process with on_failure='raise', since a wave policy can only react to failures it is told about.

Two settings, two names

on_failure applies to one process (continue / stop / raise); wave_on_failure applies to a plan of waves (skip_descendants / continue / stop_after_wave / raise). The value sets overlap without agreeing, so they are deliberately not the same parameter name.

The script exits non-zero when anything failed, and prints which processes were skipped and why.

Recovering from a contaminated run¶

Re-running a fixed process replays only the filters that never completed — the ones that succeeded keep their features and their COMPLETED rows.

But re-running does not repair the levels above it. Those processes recorded a genuine COMPLETED while computing on empty inputs, so they are skipped on the next pass and keep their wrong values. To correct them, either re-run them with force_compute=True, or delete their follow-up rows for that valid time.

Processes that ingested nothing are the exception: they carry COMPLETED,0 rows, which the skip test does not match, so they recompute on their own.

Reading the follow-up table¶

tdfs4ds.process_store.process_followup.follow_up_report(process_id=process_id).to_pandas()

`STATUS`	Meaning
`COMPLETED`	Succeeded and wrote rows.
`COMPLETED,0 rows`	Succeeded but staged nothing; will recompute next pass.
`FAILED,<error>`	Failed, with the first line of the Teradata error.
`RUNNING`	Either executing now, or stopped without recording an outcome.