Skip to content

Hybrid date × partition filter

Iterate one (date, region) combination at a time. The hybrid FilterManager view exposes both BUSINESS_DATE (the time column) and the partition column.

from tdfs4ds.utils.filter_management import FilterManager
from tdfs4ds.utils.lineage          import crystallize_view

source_data = tdml.DataFrame(tdml.in_schema('source_db', 'TRANSACTIONS'))

# 1. Build the hybrid manager (5 most-recent combinations for validation)
fm = FilterManager(table_name='FS_FILTER_REGION_DATE', schema_name=tdfs4ds.SCHEMA)
fm.load_filter(
    tdml.DataFrame.from_query("""
        SELECT REGION, TX_DATE
        FROM (
            SELECT DISTINCT REGION, TX_DATE
            FROM source_db.TRANSACTIONS
            QUALIFY ROW_NUMBER() OVER (ORDER BY TX_DATE DESC, REGION) <= 5
        ) t
    """),
    time_column='TX_DATE',
)
print('Combinations:', fm.nb_filters)

# 2. Feature view uses scalar subqueries on both filter columns
df_features_raw = tdml.DataFrame.from_query(f"""
    SELECT A.*
    FROM source_db.TRANSACTIONS A
    WHERE A.TX_DATE = (SELECT BUSINESS_DATE FROM {fm.schema_name}.{fm.view_name})
      AND A.REGION  = (SELECT REGION        FROM {fm.schema_name}.{fm.view_name})
""")

df_features_proc = crystallize_view(
    df_features_raw.groupby('CUSTOMER_ID').agg({'AMOUNT': ['sum', 'mean']}),
    view_name   = 'FEAT_ENG_TX_HYBRID',
    schema_name = tdfs4ds.SCHEMA,
)

# 3. Register with the FilterManager attached — iterates automatically
tdfs4ds.upload_features(
    df_features_proc,
    entity_id     = ['CUSTOMER_ID'],
    feature_names = ['sum_AMOUNT', 'mean_AMOUNT'],
    filtermanager = fm,
)

Why scalar subqueries: the filter view always exposes exactly one row, so the Teradata optimiser resolves the partition value before scanning the source table. This is what unlocks partition elimination at the storage level.