tdfs4ds.utils.info¶
module-attribute
¶
_COMPLETE_TYPES = frozenset({'BYTEINT', 'SMALLINT', 'INTEGER', 'INT', 'BIGINT', 'FLOAT', 'REAL', 'DOUBLE PRECISION', 'DATE'})
Retrieve the column types for specified columns from a TeradataML DataFrame.
This function retrieves the data types of specified columns in a TeradataML DataFrame. It is tailored to work with
DataFrames that have specific attributes like _td_column_names_and_types and _td_column_names_and_sqlalchemy_types,
which are not standard in typical pandas DataFrames.
Parameters: - df (DataFrame): The TeradataML DataFrame from which to get the column types. - columns (list or str): A list of column names or a single column name whose types are to be retrieved.
Returns: dict: A dictionary where keys are column names and values are their types, including length and character set for character columns (CHAR / VARCHAR).
Notes:
- This function is designed to work with TeradataML DataFrames, which may have extended column type information.
- For character columns (CHAR and VARCHAR), it retrieves the exact SQL type — including length and character set — by delegating to get_feature_types_sql_format. This is critical for entity / primary-index columns: a CHAR(14) key must not be truncated to CHAR(1) when the feature store table is created from this dict.
Dependencies: - TeradataML DataFrame containing attributes '_td_column_names_and_types' and '_td_column_names_and_sqlalchemy_types'.
Retrieve simplified column types for specified columns from a DataFrame.
This function simplifies the data types of the specified columns in a DataFrame, translating database-specific data types (such as INTEGER, BYTEINT, etc.) into more generalized Python data types (e.g., int, float). It assumes the DataFrame has a specific attribute '_td_column_names_and_types' that stores column names and their types.
Parameters: - df (DataFrame): The DataFrame from which to get the column types. - columns (list or str): A list of column names or a single column name whose types are to be retrieved.
Returns: dict: A dictionary where keys are column names and values are simplified Python data types.
Notes: - This function is designed to work with DataFrames that have a specific attribute '_td_column_names_and_types'. - It uses a mapping from specific database column types to simplified Python data types for simplification.
Dependencies: - DataFrame containing the '_td_column_names_and_types' attribute.
Converts a duration in seconds to a formatted string representing days, hours, minutes, and seconds with three decimal places.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
The duration in seconds. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
A formatted string representing the duration in days, hours, minutes, and seconds with three decimal places. |
Extracts the content within the parentheses after 'PARTITION BY' in the given partitioning string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partitioning
|
str
|
The input string containing 'PARTITION BY'. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
The content within the parentheses after 'PARTITION BY', or None if no match is found. |
Generates a partitioning clause by ensuring the presence of 'FEATURE_ID' partitioning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
partitioning
|
str or list
|
The input partitioning string or list of partitioning clauses. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
A partitioning clause string with 'FEATURE_ID' partitioning included. |
Retrieve the SQL data types of specified columns from a Teradata dataframe.
This function executes a query to fetch the SQL data types of the given columns and returns them in a dictionary format.
Parameters: tddf (tdml.DataFrame): A Teradata dataframe. columns (list): List of column names to retrieve their data types.
Returns: dict: A dictionary where keys are column names and values are their corresponding SQL data types.
Example:
tdml.load_example_data("GLM", ["admissions_train"]) df = tdml.DataFrame("admissions_train") get_feature_types_sql_format(df, columns=df.columns) {'id': 'INTEGER', 'masters': 'VARCHAR(5)', 'gpa': 'FLOAT', 'stats': 'VARCHAR(30)', 'programming': 'VARCHAR(30)', 'admitted': 'INTEGER'}
df.to_sql(table_name='test_admission_train', types={'stats': tdml.VARCHAR(length=40, charset='UNICODE')}, if_exists='replace') df = tdml.DataFrame("test_admission_train") get_feature_types_sql_format(df, columns=df.columns) {'id': 'INTEGER', 'masters': 'VARCHAR(5)', 'gpa': 'FLOAT', 'stats': 'VARCHAR(40) CHARACTER SET UNICODE', 'programming': 'VARCHAR(30)', 'admitted': 'INTEGER'}
Parse the select CAST("col" AS <type>) AS col, ... from ... query text
produced by get_feature_types_sql_format into a {column: sql_type} dict.
The exact SQL type — including length and character set — is preserved for
every column (e.g. CHAR(14), VARCHAR(40) CHARACTER SET UNICODE,
DECIMAL(38,19)). Extracted as a standalone, DB-free helper so the parsing
can be unit-tested without a live Teradata connection.
Updates the length of all VARCHAR fields in the feature_types dictionary based on an increment. The new length is calculated as ceil(previous_length / new_varchar_length) * new_varchar_length, ensuring that when new_varchar_length is equal to the current length, no change occurs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_types
|
dict
|
A dictionary where keys are feature names and values are dictionaries with 'type' and 'id'. |
required |
new_varchar_length
|
int
|
The increment value for adjusting VARCHAR lengths. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
dict |
dict
|
A dictionary with updated VARCHAR lengths. |
Issues a warning if the new length is smaller than the original length.