tdfs4ds.feature_store.feature_store_management¶

tdfs4ds.feature_store.feature_store_management ¶

logger `module-attribute` ¶

logger = getLogger(__name__)

_TEMP_TABLE_RE `module-attribute` ¶

_TEMP_TABLE_RE = compile('^tdfs_tmp_[0-9a-f]{12}$')

_ENTITY_COLUMN_RE `module-attribute` ¶

_ENTITY_COLUMN_RE = compile('^[A-Za-z_][A-Za-z0-9_$#]*$')

execute_query ¶

execute_query(query, raise_error=False, fetch_results=False, quiet_errors=(), log_errors=True)

Execute a SQL query or a list of queries using the tdml module.

Parameters:

Name	Type	Description	Default
`query`	`str or list`	A single SQL query string or a list of SQL query strings.	required
`raise_error`	`bool`	If True, re-raise exceptions after printing them. Default is False.	`False`
`fetch_results`	`bool`	If True, fetch and return the results of the SQL execution. Default is False.	`False`
`quiet_errors`	`tuple`	Lowercase substrings identifying failures the caller EXPECTS and will handle itself. A matching failure is logged at DEBUG instead of ERROR; whether it is re-raised is still governed by `raise_error` alone. Default `()` — every existing call site keeps byte-identical behaviour.	`()`
`log_errors`	`bool`	False when the caller re-raises (`raise_error=True`) and logs the failure itself, so each failure is logged once.	`True`

Returns: The result of the SQL execution if a single query is passed. None if a list of queries is passed or an exception occurs.

execute_query_wrapper ¶

execute_query_wrapper(f)

Decorator to execute a query. It wraps around the function and adds exception handling.

This decorator is used to wrap around a function that generates a query and execute that query. It handles exceptions during query execution and prints the error message along with the query. The decorator works with both TeradataML's tdml and legacy tdml.get_context() execution methods, depending on the TeradataML version.

Parameters:

Name	Type	Description	Default
`f`	`function`	Function to be decorated.	required

Returns:

Name	Type	Description
`function`		Decorated function.

Example

To decorate a function that generates and executes a query:

@execute_query_wrapper def my_query_function(): return "SELECT * FROM my_table"

This decorator can be applied to functions that generate and execute queries with ease.

create_object_if_missing ¶

create_object_if_missing(create_query, label, follow_up=())

Run an idempotent CREATE and report honestly what happened.

This exists because of a structural trap: execute_query defaults to raise_error=False, so it logs the failure itself and returns normally — an except wrapped around it never sees anything. Every catalog-creation block written that way therefore logged an ERROR for a store that was simply already provisioned, then went on to log "has been created" for a table it had not created, while its own already-exists handling sat unreachable.

Parameters¶

create_query : str The CREATE statement. label : str Human-readable object name for the log lines (e.g. "TABLE DB.FS_X"). follow_up : iterable of str Statements to run once the object is known to be present, whether this call created it or found it. They must be idempotent — COMMENT ON and REPLACE VIEW are the intended uses.

Returns¶

bool True if this call created the object, False if it was already there.

Raises¶

Exception The original driver exception for any failure that is NOT "already exists" — logged at ERROR with the full message, because the half that names the fault is often the later half.

describe_db_error ¶

describe_db_error(exc) -> str

The error message without the driver's stack trace.

Keeps EVERY message line — the half that names the fault is often not the first line — and drops only the Go frames teradatasql appends and the SQLAlchemy "Background on this error" link. Falls back to the raw text when nothing would be left.

get_column_types_simple ¶

get_column_types_simple(df, columns=None)

Retrieve simplified column types for specified columns from a DataFrame.

This function simplifies the data types of the specified columns in a DataFrame, translating database-specific data types (such as INTEGER, BYTEINT, etc.) into more generalized Python data types (e.g., int, float). It assumes the DataFrame has a specific attribute '_td_column_names_and_types' that stores column names and their types.

Parameters: - df (DataFrame): The DataFrame from which to get the column types. - columns (list or str): A list of column names or a single column name whose types are to be retrieved.

Returns: dict: A dictionary where keys are column names and values are simplified Python data types.

Notes: - This function is designed to work with DataFrames that have a specific attribute '_td_column_names_and_types'. - It uses a mapping from specific database column types to simplified Python data types for simplification.

Dependencies: - DataFrame containing the '_td_column_names_and_types' attribute.

get_feature_store_table_name ¶

get_feature_store_table_name(entity_id, feature_type, data_domain=None, primary_index=None, partitioning='', table_tag=None)

Constructs names for a database table and a corresponding view within a feature store, using the provided entity attributes, feature type, optional data domain, primary index, and partitioning information.

This function generates names that are intended for organizing and accessing features in a structured manner within a feature store. The naming convention integrates various elements, including a prefix (indicating tables or views), the specified data domain (or a default one), and aspects of the entity ID, ensuring uniqueness and clarity in the schema.

entity_id (dict, list, str, int): The identifier(s) for the entity. This can be:
- a dictionary with keys representing entity attributes,
- a list of attributes, or
- a single attribute (string or integer). The function constructs the base part of the table and view names differently based on the type of entity_id.
feature_type (str): Specifies the type of feature, contributing to the naming to indicate the kind of data stored.
data_domain (str, optional): The categorization under which the table and view will be organized. If not provided, a default value from tdfs4ds.DATA_DOMAIN is used.
primary_index (list, optional): Specifies the primary index attributes. If not provided, it defaults to the attributes derived from entity_id. This influences the order and inclusion of entity attributes in the name.
partitioning (str, optional): A string containing attributes used for partitioning, separated by spaces or commas. This affects how the name indicates the data partitioning strategy.
table_tag (str, optional): A placement tag (see tdfs4ds.feature_store.placement). Without it the names are exactly the historical ones; with it __<TAG> is appended, so the features placed under that tag live in a table of their own (same value type, same primary index and partitioning).

Returns: - tuple: Contains two strings; the first is the table name, and the second is the view name. These are constructed based on the input parameters to fit within a feature store's schema.

Example: - For entity_id={'customer_id': 123}, feature_type='purchase_history', data_domain='retail', primary_index=['customer_id'], and partitioning='year, month', this function would generate names like 'FS_T_retail_CUSTOMER_ID__YEAR_MONTH__PURCHASE_HISTORY' and 'FS_V_retail_CUSTOMER_ID__YEAR_MONTH__PURCHASE_HISTORY'.

Note: - Assumes the presence of a tdfs4ds module with a DATA_DOMAIN constant for default data domain usage. - Designed for flexibility in entity ID specification, supporting simple IDs, attribute lists, or attribute key dictionaries.

logger_safe ¶

logger_safe(level, message, *args, **kwargs)

Wrapper around the global logger that only emits logs when tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.

feature_store_catalog_view_creation ¶

feature_store_catalog_view_creation()

Constructs a SQL query to replace a view in the feature store catalog.

The function generates a SQL query that replaces a view in the feature store catalog with the content of the feature catalog table. The query uses the REPLACE VIEW statement and locks rows for access during execution.

Returns:

Name	Type	Description
`str`		A SQL query string that replaces the view with the current state
		of the feature catalog table.

feature_store_catalog_creation ¶

feature_store_catalog_creation(if_exists='fail', comment='this table is a feature catalog')

Creates or replaces a feature store catalog table in a Teradata database. The catalog table stores metadata about features such as their names, associated tables and databases, validity periods, and more.

if_exists (str, optional): Specifies the action to be taken if the catalog table already exists. Options are 'fail' (default) and 'replace'. 'fail' leaves an existing table untouched; 'replace' DROPS it and creates a new, empty one.

                     The default was 'replace' until 0.3.1.211, but that branch could never
                     run: execute_query swallowed the "already exists" failure, so the except
                     guarding it was dead code. Fixing the log noise made the failure visible,
                     which would have turned every setup() on a provisioned store into a
                     catalog wipe — setup() calls this with no argument. 'fail' therefore
                     preserves the observable behaviour exactly.

comment (str, optional): A comment to be associated with the catalog table, describing its purpose or other information. The default comment is 'this table is a feature catalog'.

Returns: str: The name of the created or replaced catalog table.

Note: - The function uses default values for the schema and table name from the tdfs4ds module. - It creates the catalog table with a primary index on 'FEATURE_ID' and a secondary index on 'FEATURE_NAME'. - The function also handles the creation of the table's comment. - In case of an existing table and 'if_exists' set to 'replace', it will drop the existing table before creating the new one. - The function assumes access to the database and appropriate permissions to create or modify tables.

Example Usage:

catalog_table_name = feature_store_catalog_creation() print(f"Feature catalog table {catalog_table_name} created successfully.")

feature_view_ddl ¶

feature_view_ddl(sorted_entity_id, table_name, view_name)

The REPLACE VIEW of the public FS_V_* view over one FS_T_* storage table.

Shared by :func:feature_store_table_creation and relocate_features so a relocated table gets exactly the view a new one would.

feature_store_table_creation ¶

feature_store_table_creation(entity_id, feature_type, if_exists='fail', primary_index=None, partitioning='', table_tag=None)

Creates a table and a corresponding view for feature storage in a Teradata database schema, based on specified entity ID and feature type.

table_tag (optional, see tdfs4ds.feature_store.placement) creates the table the features placed under that tag live in: the historical name with __<TAG> appended. Without it nothing changes.

register_features ¶

register_features(entity_id, feature_names_types, primary_index=None, partitioning='')

Orchestrates the registration or update of feature definitions in a Teradata database's feature catalog. This function selects between two methods for feature registration: update and insert, or merge, based on a configuration setting. It serves as an entry point for feature registration workflows, abstracting away the specifics of the underlying database operations.

entity_id (dict): A dictionary specifying the entity's identifiers, where keys represent attribute names. This parameter is crucial for defining the scope and granularity of feature data.
feature_names_types (dict): A dictionary mapping feature names to their properties, including data types and unique identifiers. Each value is a dictionary with keys 'type' and 'id' indicating the feature's data type and a unique identifier, respectively.
primary_index (list, optional): Specifies the primary index column(s) for the feature data. This parameter influences the organization and performance of database operations. If not specified, defaults are used based on the structure of entity_id.
partitioning (str, optional): A string that describes the partitioning strategy by listing column names used for partitioning. This parameter can affect data storage and retrieval performance.

None: The function does not return a value but triggers feature registration or update processes within the database.

Note: - The function relies on the tdfs4ds.REGISTER_FEATURE configuration to determine the registration strategy. The two supported strategies are 'UPDATE_INSERT' and 'MERGE', with the latter being the default if no specific configuration is provided. - It delegates the actual database operations to either _register_features_update_insert or _register_features_merge, depending on the selected strategy. These functions handle the specifics of the SQL operations required to register or update feature definitions in the database.

Example Usage:

entity_id = {'customer_id': 'INTEGER'} feature_names_types = {'age': {'type': 'BIGINT', 'id': 1}, 'gender': {'type': 'VARCHAR', 'id': 2}} register_features(entity_id, feature_names_types, primary_index=['customer_id'], partitioning='monthly')

This example demonstrates initiating the feature registration process for an entity with customer_id as the primary index and features such as age and gender, considering a partitioning strategy.

_placement_tags ¶

_placement_tags(entity_id)

{FEATURE_NAME upper: TABLE_TAG} of the entity in the current data domain.

Empty (and no statement at all beyond one cached existence check) in a store that does not use placements.

_tagged_names ¶

_tagged_names(entity_id, feature_type, feature_name, primary_index, partitioning, tags)

(table, view) of one feature, following its placement tag when it has one.

_warn_if_placement_reverted ¶

_warn_if_placement_reverted(entity_id, df, tags)

Warn when the catalog no longer points a placed feature at its tagged table.

A client older than 0.3.3.30 knows no placement: its register_features names the untagged table and writes the catalog row back to it. This client puts the tag back (the MERGE that follows does it); the warning says it happened, because the values the older client wrote in the meantime are in the old table (relocate_features on the same plan moves them). Fail-open: a read error is only a debug line.

_register_features_merge ¶

_register_features_merge(entity_id, feature_names_types, primary_index=None, partitioning='')

Register or update feature definitions in the feature catalog, with temporal support.

This function builds (or refreshes) entries in the Teradata feature catalog from a mapping of feature names to their metadata, computes the target feature store table and view names, stages the metadata to a temporary table, and executes a MERGE into the catalog (with optional VALIDTIME support based on tdfs4ds.FEATURE_STORE_TIME).

Parameters¶

entity_id : dict[str, Any] Mapping of entity-key column names to types. Only the keys (column names) are required here; values are not used by this function. feature_names_types : dict[str, dict] Dict of feature name -> {"type": , "id": } describing each feature’s storage type and identifier in the catalog. primary_index : list[str] | None, optional Primary index column(s) to use when deriving the feature store table/view names. If None, defaults are inferred by get_feature_store_table_name. partitioning : str, optional Partitioning expression or comma-separated column list used by get_feature_store_table_name.

Returns¶

pd.DataFrame A dataframe of the features that were (up)registered, including: FEATURE_NAME, FEATURE_TYPE, FEATURE_ID, FEATURE_TABLE, FEATURE_VIEW, ENTITY_NAME, FEATURE_DATABASE, DATA_DOMAIN.

Notes¶

When tdfs4ds.FEATURE_STORE_TIME is None, uses CURRENT VALIDTIME (non-explicit start/end). Otherwise uses VALIDTIME PERIOD ('<FEATURE_STORE_TIME>', '<END_PERIOD>') and adds the valid-time start/end when inserting.
Respects tdfs4ds.DISPLAY_LOGS via logger_safe.

_register_features_update_insert ¶

_register_features_update_insert(entity_id, feature_names_types, primary_index=None, partitioning='')

Registers or updates feature definitions in a Teradata database's feature catalog, associating entity identifiers with feature names, types, and other metadata. This function prepares and executes SQL operations to insert new feature definitions or update existing ones, considering partitioning strategies and primary index configurations.

entity_id (dict): Specifies the entity's identifiers with keys representing attribute names. This dictionary is crucial for defining the scope and granularity of feature data.
feature_names_types (dict): Maps feature names to their properties, including data types and unique identifiers. Each value is a dictionary with keys 'type' and 'id' indicating the feature's data type and a unique identifier, respectively.
primary_index (list, optional): Identifies the primary index column(s) for the feature data. This influences the organization and performance of database operations. If not specified, defaults are used based on the entity_id structure.
partitioning (str, optional): Describes the partitioning strategy through a string listing column names used for partitioning. This can impact data storage and retrieval performance.

pd.DataFrame: Contains details of the registered features, including names, types, IDs, and references to the respective feature store table and view names, alongside metadata about the entity and database schema.

Note: - The function dynamically constructs SQL queries for inserting new features or updating existing ones in the feature catalog, adapting to the provided partitioning and primary index settings. - Assumes the existence of a Teradata feature catalog table in the specified schema and that the database connection is correctly configured. - Utilizes the tdfs4ds module for database schema configurations and valid-time temporal table considerations.

Example Usage:

entity_id = {'customer_id': 'INTEGER'} feature_names_types = {'age': {'type': 'BIGINT', 'id': 1}, 'gender': {'type': 'VARCHAR_LATIN', 'id': 2}} registered_features = register_features(entity_id, feature_names_types) print(registered_features)

This example demonstrates registering features for an entity with attributes customer_id, age, and gender, where age and gender features have specified types and unique IDs.

GetTheLargestFeatureID ¶

GetTheLargestFeatureID()

Retrieves the highest feature ID currently in use from the feature catalog table of a Teradata database.

This function queries the feature catalog table to find the maximum value of the FEATURE_ID column. It is useful for identifying the next available feature ID in scenarios where a new feature needs to be added to the catalog.

Parameters: - schema (str): The schema name where the feature catalog table is located. - table_name (str, optional): The name of the feature catalog table. Defaults to 'FS_FEATURE_CATALOG'.

Returns: int: The maximum feature ID found in the feature catalog table. If the table is empty, returns 0.

Note: - The function assumes that the feature catalog table exists and is accessible in the specified schema. - If the feature catalog table has no entries, the function returns 0, indicating no feature IDs are currently in use. - The function is designed for use with Teradata databases and assumes appropriate database access.

Example Usage:

max_feature_id = GetTheLargestFeatureID() print(f"The largest feature ID is: {max_feature_id}")

GetAlreadyExistingFeatureNames ¶

GetAlreadyExistingFeatureNames(feature_name, entity_id)

Retrieves a list of already existing feature names from the feature catalog table in a Teradata database. The function checks against a specific feature name and entity ID to determine if similar features already exist in the catalog.

Parameters: - feature_name (str/list): The name(s) of the feature(s) to check. Can be a single feature name or a list of names. - entity_id (dict): A dictionary representing the entity ID, where keys are used to identify the entity.

Returns: list: A list of feature names that already exist in the feature catalog table and match the given feature name and entity ID.

Note: - The function creates a temporary table in the database to facilitate the comparison. - It assumes that the feature catalog table exists and is accessible in the specified schema. - The function is designed for use with Teradata databases and assumes appropriate database access.

cleanup_temp_tables ¶

cleanup_temp_tables(dry_run: bool = True, database: str = None, min_age_hours: float = 1.0) -> dict

List, or drop, the tdfs_tmp_<12 hex> tables left by feature registration.

Until this fix, every feature registration (GetAlreadyExistingFeatureNames) left one such table in the store's database: its cleanup called a teradataml function that does not exist and swallowed the error.

Only names matching exactly tdfs_tmp_ + 12 hexadecimal characters are considered, and only tables older than min_age_hours (a registration running now uses one for a few seconds).

Parameters:

Name	Type	Description	Default
`dry_run`	`bool`	`True` (default) lists them; `False` drops them.	`True`
`database`	`str`	default `tdfs4ds.SCHEMA`.	`None`
`min_age_hours`	`float`	leave younger tables alone.	`1.0`

Returns:

Type	Description
`dict`	`{"database", "found": [names], "dropped": [names], "failed": {name: error}}`.

Gettdtypes ¶

Gettdtypes(tddf, features_columns, entity_id)

Retrieves the data types of columns in the provided DataFrame and assigns new feature IDs for columns that are not already registered in the feature catalog table of a Teradata database. This function is useful for preparing a DataFrame for feature ingestion by ensuring each feature has a unique ID and a recognized data type.

Parameters: - tddf (tdml.DataFrame): The input DataFrame containing the features. - features_columns (list): A list of column names in the DataFrame to be considered as features. - entity_id (dict): A dictionary representing the entity ID, where keys are used to identify the entity. - schema (str): The schema name where the feature catalog table is located. - table_name (str, optional): The name of the feature catalog table. Defaults to 'FS_FEATURE_CATALOG'.

dict: A dictionary where keys are column names from the input DataFrame and values are dictionaries containing the type and ID of the feature. The type is determined based on the column's data type in the DataFrame, and the ID is either retrieved from the feature catalog (for existing features) or newly assigned.

Note: - The function checks each column in the DataFrame against the feature catalog table to determine if it already exists. - New feature IDs are sequentially assigned starting from the highest existing feature ID in the catalog. - Currently managed data types include integers (mapped to 'BIGINT') and floats (mapped to 'FLOAT'). Other data types are mapped to 'VARCHAR'. - The function assumes that the feature catalog table exists and is accessible in the specified schema.

Example Usage:

input_df = tdml.DataFrame(...) feature_columns = ['age', 'income', 'gender'] entity_id_dict = {'customer_id': 'INTEGER'} feature_types = Gettdtypes(input_df, feature_columns, entity_id_dict) print(feature_types)

tdstone2_Gettdtypes ¶

tdstone2_Gettdtypes(existing_model, entity_id, display_logs=False)

Generates a dictionary mapping feature names to their data types and unique feature IDs for a given tdstone2 model. The function filters out features that already exist in the feature catalog and includes new features with 'BIGINT' or 'FLOAT' data types.

Parameters:

Name	Type	Description	Default
`existing_model`	`object`	The tdstone2 model object containing necessary schema and scoring information.	required
`entity_id`	`dict`	A dictionary representing the entity ID, where keys are used to identify the entity.	required
`display_logs`	`bool`	Flag to indicate whether to display logs. Defaults to False.	`False`

Returns:

Name	Type	Description
`dict`		A dictionary with feature names as keys. Each key's value is a dictionary containing 'type' (either 'BIGINT' or 'FLOAT') and 'id' (unique feature ID).

Raises:

Type	Description
`ValueError`	If the data types encountered in the model are neither integer ('int') nor float ('float').

Note

This function is specifically designed for tdstone2 model objects.
It assumes that the feature catalog already exists and is accessible.
Unique IDs for new features are automatically generated, starting from the largest existing feature ID in the catalog.
Existing features in the catalog are not included in the output dictionary.

Example Usage

model = entity_id_dict = {'customer_id': 'INTEGER'} feature_types = tdstone2_Gettdtypes(model, entity_id_dict)

feature_types might look like {'feature1': {'type': 'BIGINT', 'id': 101}, 'feature2': {'type': 'FLOAT', 'id': 102}}¶

_entity_columns ¶

_entity_columns(entity_id) -> list

The entity's column names, sorted: from a list, a tuple, a dict (its keys), a comma string, or a list / dict printed as text ("['CustomerID']", seen in the field: pasted raw into the SQL, its quotes produced Error 3706 "between a string ... and the word 'USTOMER'"). A type after a name ('CustomerID BIGINT') is dropped. Raises ValueError when what is left is not a column name.

delete_feature ¶

delete_feature(feature_name, entity_id, data_domain=None)

Delete the values of a specific feature for given entities from the feature table within a specified data domain.

This function constructs and executes two SQL queries against a Teradata database to remove a feature specified by its name and entity identifiers. The first query retrieves the table name where the feature resides, based on the feature name, entity, and data domain. The second query deletes the feature values from the identified table.

Parameters: - feature_name (str): The name of the feature to be removed. - entity_id (str or list of str): Entity identifier(s). If a string is provided, it will be converted to a single-element list. The list is always sorted alphabetically before use. - data_domain (str, optional): The data domain where the feature is located. If not specified, the function uses the default data domain defined in tdfs4ds.DATA_DOMAIN.

Behavior: - The function checks if the DEBUG_MODE flag in the tdfs4ds module is set to True. If so, it prints the generated SQL queries and the resolved table name for debugging. - If the feature table cannot be resolved, the function returns without executing a delete query.

Returns: - None

Note: - The function assumes the presence of a module tdfs4ds with predefined constants such as DATA_DOMAIN, SCHEMA, FEATURE_CATALOG_NAME_VIEW, and a flag DEBUG_MODE. - It also assumes a tdml module or object with an execute_sql method capable of executing SQL queries against a Teradata database and fetching the results.

Exceptions related to SQL execution or connection issues may be raised but are not explicitly handled, except for printing the error message.

_remove_feature ¶

_remove_feature(feature_name, entity_id, data_domain=None, process_id=None)

The removal itself; see :func:remove_feature.

remove_feature ¶

remove_feature(feature_name, entity_id, data_domain=None, process_id=None, force=False)

Remove a feature from the feature catalog, with its documentation and index entries.

A feature is the triplet (data domain, entity, name) and has ONE catalog entry, whatever the number of processes computing it (each process writes its values under its own FEATURE_VERSION).

Parameters: - feature_name (str): The name of the feature to be removed. - entity_id (str or list of str): Entity column name(s); sorted before use. - data_domain (str, optional): The data domain of the feature. Defaults to tdfs4ds.DATA_DOMAIN. - process_id (str, optional, 0.3.3.23): remove the feature as produced by this process. When another process also computes the feature, its catalog entry is kept and only this process's part goes: its documentation of the feature (process feature documentation, the business-dictionary column of its view, the provenance and vector-index entries). When no other process computes it, the feature is removed entirely, as without process_id. A process that is no longer registered is accepted: the decision rests on the other producers. None (default) removes the feature whatever process computes it. - force (bool, 0.3.3.27): remove it even when datasets read it. By default the removal is refused with tdfs4ds.dependencies.DependentObjectsError, listing the datasets (and the processes reading them) in the order they must be removed. Nothing depends on it when another process still computes it and process_id scopes the removal to one process.

Each removal, and each refusal, is recorded in FS_REMOVAL_LOG (0.3.3.27).

Stored values are never deleted: they stay in their FS_T table (the storage scan of the consistency report lists the values of a removed feature as orphans).

None without process_id. With process_id, a report dict: {"mode": "process" | "full", "others": [other producing process ids], "listed": whether process_id was found among the producers}; None when the producers could not be read (nothing is removed then).

tdfs4ds.feature_store.feature_store_management¶