tdfs4ds.utils.filter_management¶

tdfs4ds.utils.filter_management ¶

logger `module-attribute` ¶

logger = getLogger(__name__)

FilterManager ¶

A utility for managing dynamic, versioned filter sets as database-backed views.

The FilterManager enables lightweight scenario management by storing multiple filter definitions in a hidden Teradata table and exposing a public view that dynamically switches between them by filter_id. Each row in the hidden table represents a complete filter configuration. The active configuration is controlled by updating the view definition rather than rewriting table data.

Key Features

Store multiple filter states (scenarios) indexed by filter_id
Switch filter states instantly by updating a view
Optionally include time-based slicing using a BUSINESS_DATE column
Clone filters between managers (soft or hard clone modes)
Prune obsolete filters to control table size
Retrieve current and historical filter definitions

Workflow Overview

Create a FilterManager pointing to a target view name.
Load one or more filter definitions using load_filter().
Switch active filters using update(filter_id).
Inspect the active filter via display() or view DDL.
Optionally prune or clone filters as needed.

How It Works Internally

A hidden table named <view_name>_HIDDEN stores filter definitions.
A Teradata view named <view_name> exposes only the active filter row.
Each filter automatically receives a sequential filter_id (ROW_NUMBER() ordering ensures deterministic assignment).
If time-based filtering is used via time_column, a BUSINESS_DATE column is added and projected in all operations.

Parameters:

Name	Type	Description	Default
`table_name`	`str`	Public view name to manage or create.	required
`schema_name`	`str`	Teradata schema where artifacts will be created.	required
`filter_id_name`	`str`	Name of the filter ID column. Defaults to `'filter_id'`.	`'filter_id'`
`time_column`	`str`	Optional name of a timestamp column from input DataFrames that maps to a `BUSINESS_DATE` column for time-aware filters.	`None`

Attributes:

Name	Type	Description
`schema_name`	`str`	Target schema for view and hidden table.
`table_name`	`str`	Name of hidden table storing filters (auto-suffixed with `_HIDDEN`).
`view_name`	`str`	Name of public view pointing to current filter.
`filter_id_name`	`str`	Column containing filter ID.
`nb_filters`	`int \| None`	Number of stored filters (None until initialized).
`col_names`	`list[str] \| None`	Columns projected by the view (data columns only).
`time_filtering`	`bool \| None`	True if time-based filtering enabled.

Notes

Database objects are only created when load_filter() is first called.
Safe for iterative pipeline runs—auto-detects existing artifacts.
Designed for large production tables and Teradata-native workflows.

schema_name `instance-attribute` ¶

schema_name = schema_name

table_name `instance-attribute` ¶

table_name = get_hidden_table_name(schema_name=schema_name, view_name=table_name)

view_name `instance-attribute` ¶

view_name = table_name

filter_id_name `instance-attribute` ¶

filter_id_name = filter_id_name

nb_filters `instance-attribute` ¶

nb_filters = None

col_names `instance-attribute` ¶

col_names = None

time_filtering `instance-attribute` ¶

time_filtering = None

storage `instance-attribute` ¶

storage = storage

ofs_map `instance-attribute` ¶

ofs_map = ofs_map

sql_table `property` ¶

sql_table

Backing table as a qualified SQL identifier.

sql_view `property` ¶

sql_view

Public view as a qualified SQL identifier.

init ¶

__init__(table_name, schema_name, filter_id_name='filter_id', time_column=None, storage=None, ofs_map='TD_MAP2')

Initialize the FilterManager.

If the hidden table/view already exist, metadata (column names, maximum filter id, and time filtering status) are detected and cached. If they do not exist yet, attributes are initialized but no objects are created until load_filter() is called.

Parameters:

Name	Type	Description	Default
`storage`		Where to create the hidden backing table when this manager (re)creates it. `None` (default) = system default (block storage, unchanged DDL); `'OFS'` = Object File System on Vantage Cloud Lake (`MAP = <ofs_map>, STORAGE = TD_OFSSTORAGE`). Per-manager, so block and OFS managers can coexist in one session. Only affects table creation; reading an existing table is unchanged. NOTE: for a tdstone2-Apply view, `storage='OFS'` is necessary but not sufficient on its own — the ACC also rejects scalar subqueries against this filter (Error 6881); reference it via a JOIN instead. OFS + JOIN scores; OFS + scalar subquery still 6881s (see `tdfs4ds.utils.storage`).	`None`
`ofs_map`		Map paired with `STORAGE = TD_OFSSTORAGE` when `storage='OFS'`.	`'TD_MAP2'`

load_filter ¶

load_filter(df, primary_index=None, time_column=None)

Load a new filter set into the hidden table and (re)point the public view at filter_id=1.

Each row in df is assigned a deterministic filter_id based on ROW_NUMBER() over the ordered set of its columns (plus BUSINESS_DATE when time filtering is enabled). If time_column is provided, values from that column are copied into BUSINESS_DATE and the view will include that time dimension.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	Incoming filter definitions (one row per filter).	required
`primary_index`	`list[str]`	Primary index columns for the hidden table. Defaults to ['filter_id'] when omitted.	`None`
`time_column`	`str`	Name of the time column in `df` to map into `BUSINESS_DATE`. If provided, time-based filtering is enabled.	`None`

Raises:

Type	Description
`ValueError`	If `time_column` is provided but not present in `df`.

update ¶

update(filter_id)

Repoint the public view to a different filter id.

Parameters:

Name	Type	Description	Default
`filter_id`	`int`	Target filter id to apply.	required

Raises:

Type	Description
`ValueError`	If filter artifacts do not exist yet.

display ¶

display()

Retrieve the current view contents as a teradataml.DataFrame.

Returns:

Type	Description
	teradataml.DataFrame: Rows projected by the public view (current filter).

get_all_filters ¶

get_all_filters()

Retrieve all filter rows from the hidden table.

Returns:

Type	Description
	teradataml.DataFrame: Full set of stored filters.

get_date_in_the_past ¶

get_date_in_the_past()

Return the earliest business date/time from the current view.

The method reads the first BUSINESS_DATE value from the current view and normalizes it to a %Y-%m-%d %H:%M:%S string. Requires that time filtering is enabled.

Returns:

Name	Type	Description
`str`		Earliest datetime as formatted string ('YYYY-MM-DD HH:MM:SS').

Raises:

Type	Description
`ValueError`	If time-based filtering is not enabled.

get_current_filterid ¶

get_current_filterid()

Extract the currently active filter id from the view DDL.

Returns:

Name	Type	Description
`int`		Filter id parsed from the view's definition.

Raises:

Type	Description
`ValueError`	If the filter id cannot be parsed from the DDL.

print_view_ddl ¶

print_view_ddl()

Log the view definition (DDL) for troubleshooting/traceability.

prune_filter ¶

prune_filter(filter_id=None)

Remove all filters with ids lower than filter_id and renumber remaining ones.

If filter_id is omitted, the method uses the current filter id from the view. After pruning, filter ids are normalized so the smallest remaining id becomes 1, and the public view is repointed to filter_id=1.

Parameters:

Name	Type	Description	Default
`filter_id`	`int`	Threshold id; rows with `{filter_id_name} < filter_id` are deleted.	`None`

Returns:

Name	Type	Description
`FilterManager`		Self, to allow method chaining.

clone_filter ¶

clone_filter(source_filtermanager, filter_id_to_apply=1, take_ownership=False, clone_mode='soft', if_exists='error')

Clone filter definitions from another FilterManager.

Supports: - soft clone (default): just point to source _HIDDEN table - hard clone: copy the source _HIDDEN table and own the copy

Parameters:

Name	Type	Description	Default
`source_filtermanager`	`FilterManager`	Source FilterManager to clone.	required
`filter_id_to_apply`	`int`	Filter ID to activate. Default: 1.	`1`
`take_ownership`	`bool`	Whether this manager owns the cloned table (soft mode only).	`False`
`clone_mode`	`str`	"soft" or "hard". Default: "soft".	`'soft'`
`if_exists`	`str`	Behavior if target hidden table already exists - "error" (default): raise an exception - "replace": drop and recreate - "skip": reuse existing table	`'error'`

Returns:

Type	Description
	FilterManager

Raises:

Type	Description
`ValueError`	On invalid clone_mode or missing source.

take_ownership ¶

take_ownership()

Take ownership of the currently linked hidden filter table.

This enables this FilterManager instance to manage (and potentially drop) the hidden table via _drop() or future maintenance methods.

Returns:

Name	Type	Description
`FilterManager`		self (for chaining)

get_filter_condition ¶

get_filter_condition(output_type: str = 'str')

Retrieve the currently applied filter condition from the database.

This method executes a SQL query that aggregates the filter columns from a filter manager view and returns the result either as a Python dictionary or a raw JSON string.

Parameters:

Name	Type	Description	Default
`output_type`	`str`	Format of the returned filter condition. - "dict": Returns a Python dictionary (default). - "str": Returns a raw JSON string.	`'str'`

Returns:

Type	Description
	dict \| str \| None: The current filter condition in the requested format. Returns None if no filter is present.

Raises:

Type	Description
`ValueError`	If `output_type` is not "dict" or "str".
`RuntimeError`	If the SQL execution fails.

Example

filters = self.get_filter_condition(output_type="dict") print(filters)

qualified ¶

qualified(database, object_name, force=False)

Build db.obj with each part quoted only as needed.

qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names); qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None database yields the bare object name, so unqualified callers keep working.

force=True quotes both parts unconditionally — only for the sites that have always emitted quotes and whose exact text is already persisted (dataset view DDL, returned volatile-table names).

split_qualified ¶

split_qualified(name)

Split db.obj into (database, object), both de-quoted.

Quote-aware, so a dot inside a quoted part does not split the name: split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X'). The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"', '"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the split('.') variant.

An unqualified name returns ('', name).

logger_safe ¶

logger_safe(level, message, *args, **kwargs)

Wrapper around the global logger that only emits logs when tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.

storage_clause ¶

storage_clause(storage, ofs_map=DEFAULT_OFS_MAP)

Return the CREATE TABLE options fragment for the requested storage.

The fragment is inserted immediately after the table name, e.g.::

CREATE TABLE db.tbl{storage_clause(...)} ( ... ) PRIMARY INDEX (...)

Parameters:

Name	Type	Description	Default
`storage`		storage selector (see :func:`resolve_storage`).	required
`ofs_map`		the map paired with `STORAGE = TD_OFSSTORAGE` when OFS is requested.	`DEFAULT_OFS_MAP`

Returns:

Type	Description
	`''` for the system default (unchanged DDL), or
	`',\n MAP = <ofs_map>,\n STORAGE = TD_OFSSTORAGE'` for OFS.

get_hidden_table_name ¶

get_hidden_table_name(schema_name, view_name)

tdfs4ds.utils.filter_management¶