Skip to content

← Training overview

6. FilterManager

When the source table is too large to ingest in a single pass — or is itself partitioned and you want to exploit that — FilterManager lets you step through it in segments. The feature view stays unchanged; the filter values are injected as scalar subqueries that the optimizer can eliminate.

Why segmented ingestion

  • Memory. A single MERGE over a 10-billion-row source spools forever and can fail before the catalog records the run. Smaller chunks succeed independently.
  • Parallelism. Each filter run is independent. You can fan them out to separate sessions or schedule them across the day.
  • Partition pruning. If the source is PARTITION BY RANGE_N(DATE_COL), filtering on DATE_COL skips entire partitions on every iteration — see Module 8 › Partitioning.
  • Failure isolation. A failing iteration logs FAILED,… in the follow-up table and the loop continues with the next filter — you re-run the bad one in isolation later. See Module 11 › Logs.

Three modes

Mode Use it when
Standard One categorical column (e.g. REGION)
Hybrid Date + categorical column (e.g. TX_DATE × REGION)
Clone You already have a filter table you want to attach

How to use it

from tdfs4ds.utils.filter_management import FilterManager

fm = FilterManager(
    schema_name='MY_DB',
    view_name='V_TX_FACT',
    col_names=['region'],
    time_column=None,        # set to a date column for Hybrid mode
)
fm.load_filter()             # populate the filter table
print(fm.nb_filters)         # how many iterations

tdfs4ds.upload_features(
    df=df,
    entity_id=['customer_id'],
    feature_names=['regional_amount'],
    filtermanager=fm,        # tdfs4ds iterates fm.update(filter_id=...) internally
)

Two operational rules from production experience: seed small (load a few filters first to validate the view), and order by date DESC in Hybrid mode so the most recent data ingests first.

Cloning filters

When several processes share the same partition sequence, clone the filter instead of calling load_filter again:

fm_p2 = FilterManager(table_name='FS_FILTER_REGIONS_P2', schema_name=tdfs4ds.SCHEMA)
fm_p2.clone_filter(fm_source)          # soft clone — no data copied

A soft clone shares the hidden table. A hard clone (clone_mode='hard') copies the data into a new table that this manager owns independently. See the FilterManager user guide for ownership semantics and if_exists options.

Learn more