tdfs4ds¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
module-attribute
¶
property
¶
True when the build table and its view exist (checked once, then cached).
property
¶
True when the window table and its view exist (checked once, then cached).
property
¶
True when the offsets table and its view exist (checked once, then cached).
__init__(schema_name=None, name='DATASET', existing_objects=None, schema_name_view=None, existing_view_objects=None)
Parameters¶
existing_objects : set, optional
A pre-computed set of lower-cased table/view names already present in
schema_name. When provided, the constructor skips its own
db_list_tables() call (saves one round-trip). Pass the set
returned by connect()'s DBC.TablesV query.
schema_name_view : str, optional
Database that holds the catalog VIEW objects. When None it defaults to
tdfs4ds.SCHEMA_VIEW if a dedicated view database is configured, else
to schema_name (co-located with the tables — byte-identical to the
historical single-database behaviour).
existing_view_objects : set, optional
Pre-computed set of lower-cased view names present in schema_name_view.
Only consulted when the view database differs from the table database.
Create whatever part of the catalog is missing. Idempotent.
Creates only the absent objects and leaves the existing ones alone, returning the list of names it created (empty when there was nothing to do). Creation order is preserved, so a table is always created before the view that reads it.
It used to refuse — logging an ERROR and raising — as soon as any
of the six objects existed, while _exists only reported True when
all six did. Those two predicates disagree on every partial catalog,
which left it permanently unrepairable: the constructor sees "not all
present", calls this, and this rejects "some present". Because the
constructor self-heals on every instantiation, that fired on ordinary
READ paths (dataset_catalog(), the agent's _resolve_dataset) and
on connect()/setup() — an ERROR-level log and an exception for a
condition the caller cannot act on. Self-healing is this constructor's
long-standing contract; it was simply incomplete.
Register (or update) dataset in the catalog. Returns its DATASET_ID.
Parameters¶
build : dict, optional
How the view was built — {'strategy', 'join_type', 'time'
(DatasetTime), 'source_dataset_id'} — written to the build table
when it exists. Omitted: the build table is left untouched.
dataset_id : str, optional
Force the DATASET_ID to write. Used by restore_dataset_catalog
when a dataset was registered several times and one id must be kept;
ordinary callers let it be looked up by name.
offsets : list of dict, optional
One row per column read at an offset (lags and leads), written to the
offsets table when it exists; [] closes the rows of an earlier
build. None (default) leaves the table untouched.
Build details (strategy, join type, temporal mode) — None when unavailable.
Offset columns (one row per lag / lead column) — None when unavailable.
Join it to get_dataset_features on (DATASET_ID, FEATURE_NAME): a
column without offset has no row.
History windows (one row per historical dataset) — None when unavailable.
Join it to the catalog on DATASET_ID with a LEFT OUTER JOIN: a
snapshot dataset has no row. WINDOW_START / WINDOW_END are the
bounds at query time (WINDOW_START NULL = from the earliest stored
value); the other columns are the definition.
Remove a dataset (catalog + entity + feature rows) by DATASET_ID.
Parameters¶
dataset_id : str
The DATASET_ID to remove.
purge : bool, optional
When False (default), a CURRENT VALIDTIME DELETE valid-time-closes
the currently-valid rows (temporal soft delete; history is retained).
When True, a NONSEQUENCED VALIDTIME DELETE physically removes every
temporal row for the dataset — use this to fully reset a dataset that
carries stale/closed history before re-registering it.
close_lineage : bool, optional
True (default): the dataset view's edges in FS_LINEAGE_GRAPH are
closed as well. False when the view itself stays (a duplicate
registration removed by restore_dataset_catalog).
Raises¶
ValueError
If no dataset with this dataset_id currently exists in the catalog.
instance-attribute
¶
Parameters¶
entity, features : optional
The AUTHORITATIVE entity/feature split, supplied by build_dataset
(which knows it at build time). When BOTH are provided they are used
verbatim and the fragile DDL-regex re-derivation
(_retrieve_entities_and_features) is skipped. When omitted (e.g.
an ad-hoc Dataset(view_name=...) for inspection) the regex path is
used as before — preserving backward compatibility.
entity is {col: sql_type}; features is
{NAME: {id, version, type, database, view}}.
valid_time : str, optional
The governing temporal qualifier, supplied by build_dataset which
chose it. When omitted it is read back from the view's DDL.
A utility for managing dynamic, versioned filter sets as database-backed views.
The FilterManager enables lightweight scenario management by storing multiple
filter definitions in a hidden Teradata table and exposing a public view that
dynamically switches between them by filter_id. Each row in the hidden table
represents a complete filter configuration. The active configuration is
controlled by updating the view definition rather than rewriting table data.
Key Features
- Store multiple filter states (scenarios) indexed by
filter_id - Switch filter states instantly by updating a view
- Optionally include time-based slicing using a
BUSINESS_DATEcolumn - Clone filters between managers (soft or hard clone modes)
- Prune obsolete filters to control table size
- Retrieve current and historical filter definitions
Workflow Overview
- Create a
FilterManagerpointing to a target view name. - Load one or more filter definitions using
load_filter(). - Switch active filters using
update(filter_id). - Inspect the active filter via
display()or view DDL. - Optionally prune or clone filters as needed.
How It Works Internally
- A hidden table named
<view_name>_HIDDENstores filter definitions. - A Teradata view named
<view_name>exposes only the active filter row. - Each filter automatically receives a sequential
filter_id(ROW_NUMBER()ordering ensures deterministic assignment). - If time-based filtering is used via
time_column, aBUSINESS_DATEcolumn is added and projected in all operations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table_name
|
str
|
Public view name to manage or create. |
required |
schema_name
|
str
|
Teradata schema where artifacts will be created. |
required |
filter_id_name
|
str
|
Name of the filter ID column. Defaults to |
'filter_id'
|
time_column
|
str
|
Optional name of a timestamp column from input DataFrames
that maps to a |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
schema_name |
str
|
Target schema for view and hidden table. |
table_name |
str
|
Name of hidden table storing filters (auto-suffixed with |
view_name |
str
|
Name of public view pointing to current filter. |
filter_id_name |
str
|
Column containing filter ID. |
nb_filters |
int | None
|
Number of stored filters (None until initialized). |
col_names |
list[str] | None
|
Columns projected by the view (data columns only). |
time_filtering |
bool | None
|
True if time-based filtering enabled. |
Notes
- Database objects are only created when
load_filter()is first called. - Safe for iterative pipeline runs—auto-detects existing artifacts.
- Designed for large production tables and Teradata-native workflows.
instance-attribute
¶
__init__(table_name, schema_name, filter_id_name='filter_id', time_column=None, storage=None, ofs_map='TD_MAP2')
Initialize the FilterManager.
If the hidden table/view already exist, metadata (column names, maximum
filter id, and time filtering status) are detected and cached. If they do
not exist yet, attributes are initialized but no objects are created until
load_filter() is called.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage
|
Where to create the hidden backing table when this manager
(re)creates it. |
None
|
|
ofs_map
|
Map paired with |
'TD_MAP2'
|
Load a new filter set into the hidden table and (re)point the public view at filter_id=1.
Each row in df is assigned a deterministic filter_id based on ROW_NUMBER() over the
ordered set of its columns (plus BUSINESS_DATE when time filtering is enabled). If
time_column is provided, values from that column are copied into BUSINESS_DATE and the
view will include that time dimension.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Incoming filter definitions (one row per filter). |
required |
primary_index
|
list[str]
|
Primary index columns for the hidden table. Defaults to ['filter_id'] when omitted. |
None
|
time_column
|
str
|
Name of the time column in |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Repoint the public view to a different filter id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filter_id
|
int
|
Target filter id to apply. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If filter artifacts do not exist yet. |
Retrieve the current view contents as a teradataml.DataFrame.
Returns:
| Type | Description |
|---|---|
|
teradataml.DataFrame: Rows projected by the public view (current filter). |
Retrieve all filter rows from the hidden table.
Returns:
| Type | Description |
|---|---|
|
teradataml.DataFrame: Full set of stored filters. |
Return the earliest business date/time from the current view.
The method reads the first BUSINESS_DATE value from the current view
and normalizes it to a %Y-%m-%d %H:%M:%S string. Requires that time
filtering is enabled.
Returns:
| Name | Type | Description |
|---|---|---|
str |
Earliest datetime as formatted string ('YYYY-MM-DD HH:MM:SS'). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If time-based filtering is not enabled. |
Extract the currently active filter id from the view DDL.
Returns:
| Name | Type | Description |
|---|---|---|
int |
Filter id parsed from the view's definition. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the filter id cannot be parsed from the DDL. |
Remove all filters with ids lower than filter_id and renumber remaining ones.
If filter_id is omitted, the method uses the current filter id from the view.
After pruning, filter ids are normalized so the smallest remaining id becomes 1,
and the public view is repointed to filter_id=1.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filter_id
|
int
|
Threshold id; rows with |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
FilterManager |
Self, to allow method chaining. |
clone_filter(source_filtermanager, filter_id_to_apply=1, take_ownership=False, clone_mode='soft', if_exists='error')
Clone filter definitions from another FilterManager.
Supports: - soft clone (default): just point to source _HIDDEN table - hard clone: copy the source _HIDDEN table and own the copy
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_filtermanager
|
FilterManager
|
Source FilterManager to clone. |
required |
filter_id_to_apply
|
int
|
Filter ID to activate. Default: 1. |
1
|
take_ownership
|
bool
|
Whether this manager owns the cloned table (soft mode only). |
False
|
clone_mode
|
str
|
"soft" or "hard". Default: "soft". |
'soft'
|
if_exists
|
str
|
Behavior if target hidden table already exists - "error" (default): raise an exception - "replace": drop and recreate - "skip": reuse existing table |
'error'
|
Returns:
| Type | Description |
|---|---|
|
FilterManager |
Raises:
| Type | Description |
|---|---|
ValueError
|
On invalid clone_mode or missing source. |
Take ownership of the currently linked hidden filter table.
This enables this FilterManager instance to manage (and potentially drop)
the hidden table via _drop() or future maintenance methods.
Returns:
| Name | Type | Description |
|---|---|---|
FilterManager |
self (for chaining) |
Retrieve the currently applied filter condition from the database.
This method executes a SQL query that aggregates the filter columns from a filter manager view and returns the result either as a Python dictionary or a raw JSON string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_type
|
str
|
Format of the returned filter condition. - "dict": Returns a Python dictionary (default). - "str": Returns a raw JSON string. |
'str'
|
Returns:
| Type | Description |
|---|---|
|
dict | str | None: The current filter condition in the requested format. Returns None if no filter is present. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If the SQL execution fails. |
Example
filters = self.get_filter_condition(output_type="dict") print(filters)
Bases: RuntimeError
Raised when filter-level ingestion failures are escalated by on_failure.
Carries the machine-readable detail so an orchestrator does not have to parse
the message: process_id, n_failed, n_total and failures
(a list of {filter_id, error}).
Bases: ValueError
The selection names a feature that cannot be resolved unambiguously.
missing: names not registered for this entity and data domain.
no_producer: registered names that no current process computes.
ambiguous: {feature: [{'process_id', 'process_view_name'}]} for the
features computed by several processes, with no version given.
Apply the console settings to the tdfs4ds logger.
color / propagate set tdfs4ds.LOG_COLOR / tdfs4ds.LOG_PROPAGATE
first when given. Idempotent: one console handler at most.
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
get_available_entity_id_records(entity_id, selected_features, other=None, time_column=None, filtermanager=None, filter_conditions=None)
Generate a SQL query to retrieve records for available entity IDs based on selected features.
Parameters: entity_id (list, dict, or str): The entity IDs for which records are needed. Can be a list, dictionary, or string. selected_features (dict): A dictionary of selected features with their versions. other (optional): An additional DataFrame object for more complex queries. Defaults to None. time_column (str, optional): The name of the time column to use for temporal conditions. Defaults to None.
tuple: A tuple containing: - str: A SQL query string to retrieve records for available entity IDs. - dict: A dictionary of the list of features and their locations.
write_where_clause_filter(filtermanager, filter_conditions, list_entity_id, filter_alias='FILTER', table_alias='A')
Constructs a SQL WHERE clause based on filter criteria.
This function generates a SQL WHERE clause for filtering records in a database. The filter_conditions parameter is
a dictionary containing key-value pairs, where each key corresponds to a column name in filtermanager.col_names,
and each value is an entity ID or filter value. The clause is constructed using aliases for the table and filter
provided through filter_alias and table_alias.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filtermanager
|
object
|
An object that contains a |
required |
filter_conditions
|
dict
|
A dictionary where keys represent column names that must match |
required |
list_entity_id
|
list
|
A list of entity IDs used to validate or restrict values in the filter dictionary. |
required |
filter_alias
|
str
|
The alias used for the filter table in the SQL query. Defaults to 'FILTER'. |
'FILTER'
|
table_alias
|
str
|
The alias used for the main table in the SQL query. Defaults to 'A'. |
'A'
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
A SQL WHERE clause string that joins each column and entity ID in |
Raises:
| Type | Description |
|---|---|
ValueError
|
If any key in |
Example
filtermanager.col_names = ['column1', 'column2'] filter_conditions = {'column1': 'entity_id1', 'column2': 'entity_id2'} list_entity_id = ['entity_id1', 'entity_id2'] write_where_clause_filter(filtermanager, filter_conditions, list_entity_id) 'FILTER.column1=A.entity_id1 AND FILTER.column2=A.entity_id2'
Retrieve version identifiers for one or more features belonging to a given entity.
The function queries the underlying metadata tables to find the process (i.e., feature‑version) records that match the supplied entity and feature names. It returns a mapping from each requested feature name to either:
- None – if no matching rows were found.
- A single UUID string – if exactly one matching row exists for the feature.
- A list of dictionaries – if more than one matching row is found; each
dictionary contains:
process_id– the UUID of the process that produced the version,process_view_name– the human‑readable view name associated with that process.
Parameters¶
entity_name : str | list[str] The name (or names) of the entity whose features we are querying. If a single string is supplied it is treated as a singleton list.
str | list[str]
One or more feature names to look up. Accepts a single string or an iterable of strings; if a single string is provided it is wrapped in a list internally.
str, optional
The data‑domain partition to filter on. If omitted the default
tdfs4ds.DATA_DOMAIN constant is used.
Returns¶
dict[str, str | None | list[dict]]
A dictionary keyed by feature name. Each value is either:
* None – no records were found for that feature.
* str – a single UUID string when exactly one row matched.
* list[dict] – multiple matches; each dict has keys
process_id and process_view_name.
Notes¶
- The query joins the feature catalog view with the process catalog (specifically the “feature split” view) on data domain, entity ID, and feature name.
- SQL string literals are escaped by doubling single quotes; this is a lightweight escape that suffices for the current use‑case.
- The function preserves insertion order of features in the returned dictionary (Python 3.7+ guarantees dict order).
- When
tdfs4ds.DEBUG_MODEis true, the generated SQL statement is printed to stdout – useful for troubleshooting.
Example¶
get_feature_versions('user', ['age', 'income'])
Retrieve a list of unique entity names from a specified data domain.
This function executes a database query to extract distinct entity names from a feature catalog, filtered by the provided data domain. If no domain is specified, it defaults to a predefined data domain.
domain (str, optional): The data domain to filter the entity names. Defaults to None, in which case a predefined domain is used.
Returns: DataFrame: A pandas-like DataFrame containing the unique entity names.
Retrieve a list of feature names associated with a specific entity or entities from a given data domain.
This function constructs and executes a database query to extract feature names for the specified entity or entities from a feature catalog, filtered by the provided data domain. If no domain is specified, it defaults to a predefined data domain.
entity_name (str or list): The name of the entity or a list of entity names to fetch features for. domain (str, optional): The data domain to filter the feature names. Defaults to None, where a predefined domain is used.
Returns: DataFrame: A pandas-like DataFrame containing the feature names associated with the given entity or entities.
Return a follow-up report as a tdml.DataFrame, optionally filtered by
process_id and/or a filtermanager's applied filter.
Behavior by arguments
- process_id is None and filtermanager is None: Return all rows from SCHEMA.FOLLOW_UP_NAME, sorted by START_DATETIME desc.
- process_id is not None and filtermanager is None: Return rows for the given PROCESS_ID.
- process_id is not None and filtermanager is not None:
Return rows for the given PROCESS_ID whose APPLIED_FILTER matches the
JSON_AGG of
filtermanager's columns coming from its schema/view. - process_id is None and filtermanager is not None:
Return rows whose APPLIED_FILTER matches the JSON_AGG of
filtermanager(no PROCESS_ID constraint).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filtermanager
|
Optional[object]
|
An object exposing |
None
|
process_id
|
Optional[str]
|
Optional process identifier used to filter by PROCESS_ID. |
None
|
Returns:
| Type | Description |
|---|---|
|
tdml.DataFrame: The resulting dataframe sorted by START_DATETIME (descending). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If the query fails. |
Called automatically during import tdfs4ds.
Load order (lowest → highest priority): JSON config file → .env file → OS env vars
(Re-)load configuration from external sources.
Useful for pointing at non-default config files or refreshing values mid-session. Applies sources in priority order (lowest first):
- JSON config file (at path or auto-discovered)
.envfile (at dotenv_path or auto-discovered)- OS
TDFS4DS_*environment variables (always applied)
Parameters¶
path : str or pathlib.Path, optional
Explicit path to a JSON config file. Skips auto-discovery when given.
dotenv_path : str or pathlib.Path, optional
Explicit path to a .env file. Skips auto-discovery when given.
Examples¶
Auto-discover all sources::
import tdfs4ds
tdfs4ds.load_config()
Explicit paths::
tdfs4ds.load_config(
path='/project/configs/feature_store.json',
dotenv_path='/project/.env.local',
)
JSON config format (tdfs4ds.json)::
{
"schema": "MY_DATABASE",
"data_domain": "MY_PROJECT",
"display_logs": true,
"store_feature": "MERGE",
"varchar_size": 1024,
"instruct_model_provider": "openai",
"instruct_model_model": "gpt-4o"
}
Keys are case-insensitive. instruct_model_api_key is rejected from
JSON config — use a .env file or OS env var instead.
.env file format::
TDFS4DS_SCHEMA=MY_DATABASE
TDFS4DS_SCHEMA_VIEW=MY_VIEWS_DATABASE
TDFS4DS_DATA_DOMAIN=MY_PROJECT
TDFS4DS_INSTRUCT_MODEL_API_KEY=sk-...
Only TDFS4DS_* variables are read from the .env file.
Add .env to .gitignore to keep secrets out of source control.
Capture the current configuration as a JSON config file or a .env.
Two uses, selected by source:
source='module'(default) — snapshot the running configuration (tdfs4ds.<ATTR>), including anything set in-session. Use it to reproduce a session that works.source='env'— convert theTDFS4DS_*environment variables into a file, whether or not they ever reached the module config.
Parameters¶
path : str or pathlib.Path, optional
Where to write. When None nothing is written and the text is only
returned — inspect it before committing it to disk.
fmt : {'json', 'env'}
'json' writes a tdfs4ds.json that load_config() reads;
'env' writes TDFS4DS_* lines for a .env / container
env-file. Only 'env' can carry connection and Vector Store auth
variables — those are pass-through, never module attributes, so
fmt='json' cannot express them.
include_secrets : bool, default False
Secrets (anything whose name contains PASSWORD / TOKEN / API_KEY /
ACCESS_KEY / SECRET) are written commented out with a placeholder,
so the result is a shareable template. Pass True to write the real
values — then treat the file as a secret. Ignored for fmt='json':
_apply_file rejects credentials from JSON by design, so they are
never written there.
Returns¶
str The file content, whether or not path was given.
Examples¶
tdfs4ds.export_config() # inspect # doctest: +SKIP tdfs4ds.export_config('tdfs4ds.json') # reusable config tdfs4ds.export_config('.env', fmt='env', source='env') # env -> file
Show the configuration with secret VALUES hidden but their presence confirmed.
The counterpart to :func:export_config for the question you actually ask
when a deployment misbehaves: is this variable set? export_config
cannot answer it — it redacts a secret to a commented placeholder, which
reads identically whether the variable holds a password or nothing at all.
Never returns or prints a secret value. A secret reports its length
instead, which is enough to spot a truncated paste or a stray quote and not
enough to reconstruct. "Is this a credential?" is decided by
:func:_is_secret — the one authority for it — never re-derived here.
Parameters¶
source : {'env', 'module'}
'env' (default) reads the TDFS4DS_* environment variables,
so it reports what is set even when it never reached the module config
(the load-order trap: config.py populates tdfs4ds.<ATTR> once,
at import). 'module' snapshots the running configuration.
Pass-through variables have no module attribute and are always read
from the environment, whatever source says.
required : iterable of str, optional
Names to flag when missing, with or without the TDFS4DS_ prefix.
An unset variable is normally omitted to keep the report readable;
a required one is always shown, marked NOT SET (required).
display : bool, default True
Print the report. Set False to only receive the rows.
Returns¶
list of dict
One row per reported variable: name (with prefix), attr,
status ('set' / 'hidden' / 'unset'), value (the
real value, or None when hidden), secret, required and
length.
Examples¶
tdfs4ds.describe_config() # doctest: +SKIP tdfs4ds.describe_config(required=['TDFS4DS_TD_HOST']) # doctest: +SKIP rows = tdfs4ds.describe_config(display=False) # doctest: +SKIP
Set individual TDFS4DS_* variables in an existing .env, in place.
:func:export_config and :func:tdfs4ds.generate_ai_studio_env both
rewrite the whole file, which loses its comments, its ordering and the
commented-out alternatives they deliberately put there. This changes one
line at a time, so a probed value can be recorded into a file a human
maintains.
It is line-based rather than parse-and-rewrite because :func:_parse_dotenv
is lossy for this purpose — it drops comments, ordering and quoting, so it
cannot serve as the read side of a read-modify-write.
Which line is changed, and why:
- The last uncommented occurrence.
_parse_dotenvassigns as it reads, so on a duplicate the last one is what takes effect; patching any other would leave the file saying one thing and meaning another. - Otherwise the first commented-out slot (
#TDFS4DS_X=) is uncommented in place. A generated file offers such slots precisely so that turning one on is an edit. - Otherwise appended, preceded by comment if given.
A commented slot is never uncommented while a live line exists — a
generated file carries commented alternatives next to the live choice
(two MCP_SERVER_URL lines, for instance), and enabling one of those
would silently switch the setting to the alternative.
Parameters¶
path : str or Path
An existing .env. Creating one is :func:export_config's or
:func:tdfs4ds.generate_ai_studio_env's job, not this function's, so a
missing file raises rather than being conjured half-populated.
updates : dict
{'STRUCTURED_OUTPUT_METHOD': 'function_calling'} — keys with or
without the TDFS4DS_ prefix. Values are rendered the way
:func:_coerce reads them back, so None is written as the literal
None (which loads as None), not as an empty line.
comment : str, optional
Provenance note, written as a trailing comment on the line itself.
Say where the value came from and when: a pinned setting that was right
in one session is wrong in another, and a stale pin should be visible
rather than silent.
Trailing rather than above, because the whole line is rewritten on each
call — so re-pinning replaces the note instead of stacking a new one on
every run. It is dropped for a value that has to be quoted: the parser
takes a quoted value verbatim, so a trailing comment would become part
of it. Newlines are folded to "; " for the same reason.
allow_secrets : bool, default False
Required to write a credential. A general-purpose file patcher must not
be the thing that puts a password into a file someone later commits, so
by default a name :func:_is_secret recognises is refused.
Returns¶
dict
{'updated': [...], 'added': [...], 'unchanged': [...]} of full
TDFS4DS_* names. Nothing is written when everything is unchanged.
Raises¶
ValueError path does not exist, a key is not a registered variable (it would only earn an "unknown .env variable" warning on load), or it is a credential without allow_secrets.
Crystallizes a TeradataML DataFrame into a database view.
This function takes a TeradataML DataFrame (tddf) and converts its SQL representation into a database view.
It analyzes dependencies, creates sub-views if needed, and constructs the final view with the specified
view_name and schema_name. The resulting view can optionally be returned as a TeradataML DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tddf
|
DataFrame
|
The TeradataML DataFrame to be converted into a view. |
required |
view_name
|
str
|
The name of the main view to be created or replaced in the database. |
required |
schema_name
|
str
|
The schema where the view will be created. |
required |
output_view
|
bool
|
If |
True
|
Returns:
| Type | Description |
|---|---|
|
teradataml.DataFrame: A TeradataML DataFrame representation of the created or replaced view,
if |
Raises:
| Type | Description |
|---|---|
Exception
|
If there is an error during SQL execution for creating sub-views or the main view. |
Notes
- The
tddfparameter must be a valid TeradataML DataFrame. - The function generates a dependency graph for the SQL query representing the input DataFrame. Sub-views are created or replaced with new names based on this graph.
- Column names are replaced case-insensitively to ensure consistency in SQL queries.
- The function replaces "CREATE VIEW" with "REPLACE VIEW" to handle updates.
Example
To crystallize a TeradataML DataFrame into a view:
result_df = crystallize_view(my_teradataml_df, 'my_view', 'my_schema')
Converts a duration in seconds to a formatted string representing days, hours, minutes, and seconds with three decimal places.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
The duration in seconds. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
A formatted string representing the duration in days, hours, minutes, and seconds with three decimal places. |
The ON condition joining two relations on the entity columns.
entity_id: {column: SQL type}; entity_null_substitute: {column: value} for
the columns whose NULLs are matched through COALESCE (quoted for a VARCHAR column);
left_name / right_name: the two relations' aliases. Returns the conditions joined
with AND.
True when name cannot be written as a bare Teradata identifier.
An already-quoted name returns False — it is legal as it stands.
Return name as a Teradata identifier, quoting it only if required.
Idempotent: an already-quoted name is returned unchanged rather than being
wrapped a second time. Embedded quotes are escaped as "".
Parameters¶
name : str
The raw object/database name.
force : bool, default False
Quote even when the name does not need it. Use only at the few sites
that have always emitted unconditional quotes and whose exact output is
already baked into persisted DDL (see
feature_store/feature_query_retrieval.py) — elsewhere forcing would
change the SQL of existing stores for no reason.
Inverse of :func:quote_ident: strip a matched quote pair and unescape.
A name that is not quoted is returned stripped but otherwise untouched. This
is deliberately narrower than the .strip('"') / .replace('"','')
idioms scattered through the package: those also mangle a name whose own
content contains a quote.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Database that holds VIEW objects.
Returns SCHEMA_VIEW when a dedicated view database has been configured
(database_view= in setup()/connect(), TDFS4DS_SCHEMA_VIEW env, or JSON
schema_view), otherwise falls back to SCHEMA (the table database), so
the default single-database behaviour is unchanged. Every VIEW create/read
site resolves its target database through this helper; TABLE sites keep using
tdfs4ds.SCHEMA directly.
view_schema() as a SQL identifier (quoted only when it has to be).
Normalise a read-only store entry (str shorthand or dict) to a full
descriptor {'schema','schema_view','vector_prefix'}. Returns None for an
unusable entry (so callers can skip it).
Ordered list of VIEW schemas to read catalogs / business dictionary /
lineage from: the primary (view_schema()) first, then each attached
read-only store. Deduped, primary always first. Returns [view_schema()]
when no read-only store is attached (default => unchanged).
Ordered list of TABLE schemas (primary SCHEMA first, then read-only
stores) — used as the vector-store target_database per store.
Full store descriptors, primary synthesised first, then each read-only
store. Each dict carries schema / schema_view / vector_prefix /
is_primary. Used by the Teradata multi-store vector search.
Initialize the feature store environment by creating catalog tables and views.
Parameters¶
database : str Database that holds the physical feature-store TABLES. if_exists : str 'fail' (default) or 'replace' — drop existing catalog objects first. database_view : str, optional Dedicated database for VIEW objects (process views, dataset views, and all framework public FS_V_ views). When None (default) views are co-located with the tables in database — byte-identical to the historical behaviour. When set, every view is created in this database while tables stay in database* (the view database must be granted SELECT WITH GRANT on the table database).
connect(database=None, database_view=None, feature_catalog_name=None, process_catalog_name=None, data_distribution_name=None, filter_manager_name=None, followup_name=None, feature_catalog_name_view=None, process_catalog_name_view=None, dataset_catalog_name=None, documentation_process_business_logic=None, documentation_process_features=None, documentation_process_explain=None, documentation_business_dictionary_objects=None, documentation_business_dictionary_columns=None, documentation_business_dictionary_sections=None, databases_readonly=None, create_if_missing=False)
Connect the session to an existing feature store.
Needs an open teradataml context (tdml.create_context, tdfs4ds.open_session or
tdfs4ds.reconnect_teradata). Sets tdfs4ds.SCHEMA (and SCHEMA_VIEW), checks
which catalog objects exist (one DBC read), refreshes the catalog views, creates the
optional tables a newer version adds when they are missing (process options and
settings, placement, removal log, documentation tracking, data domain settings...;
a refused CREATE is a warning, never a failure), and configures the Vector Store
authentication when TDFS4DS_VECTOR_STORE_AUTH_* is set.
Parameters¶
database : str, optional
Database of the store's tables. Default: tdfs4ds.SCHEMA.
database_view : str, optional
A separate database for the store's views (see setup). Default: the same.
databases_readonly : list, optional
Other feature stores to read, never write (cross-store read federation): a database
name, or {'schema', 'schema_view', 'vector_prefix'}. None keeps none.
create_if_missing : bool, optional
Create the missing catalog objects (as setup would) instead of reporting them.
Default False: a store with missing catalogs is reported and not connected.
feature_catalog_name, process_catalog_name, ... : str, optional
Override the catalog object names (default: the tdfs4ds module constants).
Returns¶
bool True when the store is connected, False when its catalogs are missing.
find_feature_stores(*, complete_only: bool = False, details: bool = True, databases: Optional[Iterable[str]] = None, run=None)
List the databases that hold a feature store, as a pandas DataFrame.
Parameters¶
complete_only : bool
Keep only stores that connect() accepts as they are (every required
catalog table present).
details : bool
Also count each store's current processes and data domains (one small
query per store, through its public process-catalog view). READABLE
is False when that read is refused.
databases : iterable of str, optional
Restrict the search to these databases.
run : callable, optional
run(sql) -> rows to read through another connection (launch_app
uses a raw teradatasql one, before any teradataml session exists).
Default: the teradataml context.
Returns¶
pandas.DataFrame
One row per database holding a feature catalog and a process catalog:
DATABASE_NAME, VIEW_DATABASE (where its catalog views are, None
when not found), COMPLETE, MISSING (comma-separated tables
connect() would need), and with details READABLE,
N_PROCESSES, N_DATA_DOMAINS and ERROR (why a store is not
readable, the whole message). Sorted by name. Empty when nothing
is visible to this session.
Only objects the session may see are listed (DBC.TablesV), so a store
the user has no right on does not appear.
Re-establish the teradataml context from TDFS4DS_TD_* credentials.
Returns True on success, False on failure (or raises when raise_error=True).
A no-op-safe call: if the credentials are not configured it logs a warning and
returns False.
It refuses to destroy a working context on the way to a broken one. The
credentials are probed with the raw driver first, and a refusal leaves the
existing context untouched. This is not hypothetical tidiness: create_context
calls remove_context internally and assigns teradataml's global engine
before testing it, so a failed reconnect used to leave get_context()
returning a poisoned engine — every later statement in the process then failed
with that logon error instead of its own, and the session could not self-heal.
That is how one nominal "table already exists" during connect() ended a
notebook's session.
open_session(env_file=None, *, database=None, token=None, connect=True, refresh=True, sign_in=True, databases_readonly=None)
Load the .env, log on, and connect to the feature store. Returns a short report.
Parameters¶
env_file : str, optional
The .env to load. None (default) = the nearest one: the current folder, then
each parent folder, then ~/.tdfs4ds/.env (:func:find_env_file); the file used is
logged and returned in the report. False = load none (keep the session's settings).
database : str, optional
The store you work in (read-write); default TDFS4DS_TD_DATABASE, else
tdfs4ds.SCHEMA. When none of them gives a store, nothing is connected and a warning
says so: pass database=.
token : str, optional
A database token (%gen_token's user_token). Default: the platform's current
token when the .env logs on with a token (AI Studio), else nothing.
connect : bool
False stops after the logon (for setup() on a new store).
refresh : bool
Renew the token in the background (every 15 minutes) when the platform provides one,
so a long session keeps working; the open database session itself is not affected by
the token's expiry.
sign_in : bool
When the platform gives no token and the one in the .env has expired, run the
platform's device-flow sign-in (%gen_token_using_device_flow): a link is shown, you
sign in, and the session continues with the new token. Only in a notebook or terminal
someone is watching; False never asks.
databases_readonly : list of str (or dict descriptors), or str, optional
Other feature stores to read, read-only, next to the one you work in (read federation;
writes still go to database only). A single name is a one-item list. None (default)
passes nothing to connect, so TDFS4DS_SCHEMA_READONLY of the .env still
applies; a list replaces that set; [] detaches every read-only store. The attached
stores are returned in report['read_only'].
check_configuration(*, live=True, display=True, mcp_question=None, selftest=False, pin=False, progress=None)
Validate the tdfs4ds AI configuration and report what to fix.
Runs eight checks, in dependency order (plus an opt-in ninth, selftest):
- Config load — every check below reads
tdfs4ds.<ATTR>, whichconfig.pypopulates once duringimport tdfs4ds. A variable exported after that import (or a.envoutside the working directory) never reaches the module config, so this reports whichTDFS4DS_*variables are set in the environment but not applied — otherwise every check below blames a setting the user did make. 0b. Temporal (VALIDTIME) — with an open database session: a session-only VALIDTIME table is created, written, readCURRENTandNONSEQUENCED, and dropped. Every tdfs4ds table is temporal. 0c. Query band & DBQL — the session query band can be set (and the caller's band is restored), and DBQL can be read with logging active for this user: whatfollowup_resource_usageneeds to join process executions to their resource usage. Only a refusedSET QUERY_BANDfails; missing DBQL access or logging is reported with the statement to ask the DBA for. - Vector Store auth — resolves and applies
teradataml.set_auth_tokenfrom theTDFS4DS_VECTOR_STORE_AUTH_*variables (JWT / PAT / Basic / OAuth). Skipped when nothing is configured and the backend is not'teradata'. - Vector Store service —
CollectionManager().health()(liveness), then.list()(the first call that actually spends the credential), then the service version against :data:_VECTOR_STORE_MIN_VERSION: an older service is up and accepts the credential, and only misbehaves once something writes, so the version is the one place that is visible in advance. Only whenVECTOR_STORE_BACKEND='teradata'. - LLM transport (TLS) — credential-free, and before the model on
purpose: completes a TLS handshake with
INSTRUCT_MODEL_URLusing the configured trust, reports the certificate's identity and days to expiry, separates a hostname mismatch from an untrusted issuer from an expired certificate (three different fixes), and says whether something in front of the gateway answers instead of it. Instruct-only, becauseINSTRUCT_MODEL_VERIFY_SSL/_CA_BUNDLEare. - Instruct model — builds the client with
build_llm()and sends a hello-world prompt: can it answer at all. - JSON constraints — typed output, which a plain reply proves nothing
about. Probes both paths, because they are independent: the agent /
query-optimizer path (
STRUCTURED_OUTPUT_METHOD) and the documentation path (INSTRUCT_MODEL_PROVIDER), whose silent retry unconstrained is invisible to every other check. Only if one reports trouble does it run the nine-request wire-level sweep that identifies the serving stack; the full sweep travels under the check'sjson_sweepkey. Seepinto record the answer. - Embedding model — with the Teradata backend and
TERADATA_AI_API_TYPEset, checks only thatTeradataAI(...)instantiates (in-database embedding needs nothing more from us); otherwise embeds a hello-world string and verifies the returned vector length againstEMBEDDING_MODEL_DIM. -
MCP server (when
MCP_SERVER_URLis set) — two stages: list the tools (proving reachability, auth and TLS), then drive them with a minimal agent (proving the instruct model can actually call tools, which listing them does not). A failure in the second stage isolates the problem to the model side. -
Vector Store self-test (
selftest=Trueonly) — the checks above are all read-only, and every Vector Store failure this package has hit in the field is invisible to a read-only probe. This one writes: it embeds a handful of sentences into a throwaway collection, retrieves them, changes one, refreshes, retrieves again, and cleans up — see :func:tdfs4ds.selftest_vector_store, whose full per-step report and support reproducer are returned under the check'sselftestkey.
The Teradata database connection is not opened here — it is external to the platform and owned by the caller; the two database checks are skipped (with a hint) when no session is open.
Parameters¶
live : bool, default True
Make the outbound calls (auth handshake, health probe, one LLM prompt,
one embedding). False builds the clients and resolves the
configuration but calls nothing — for CI or an offline sanity pass.
display : bool, default True
Print an aligned summary, with the fix hint under each check that did
not pass.
mcp_question : str, optional
Question sent to the MCP tool-calling probe. Defaults to asking for the
list of available databases — answerable by both the Community and
Enterprise editions. The agent's tool guards apply to a custom question
too, so a probe reaching for feature-store internals is refused exactly
as it would be in the agent.
selftest : bool, default False
Run the write-path Vector Store self-test as a ninth check. Off by
default because it is the only part of this function that creates
and drops database objects; everything else is read-only. Ignored when
live=False or the backend is not 'teradata'.
pin : bool or str, default False
Record what the JSON-constraints check discovers. False reports the
setting and changes nothing. True applies it to the session and
writes it into the first .env on config.py's search path via
:func:tdfs4ds.patch_dotenv, stamped with the endpoint and date it was
probed against — so the value is not re-derived every session, and a
stale pin is visible rather than silent. A string pins into that file
instead.
Worth knowing before using it: ``STRUCTURED_OUTPUT_METHOD`` is
classified as ordinary config by the AgentOps packager, so a pinned
value travels into a deployed pod's ``config.yaml`` — right when the
pod shares this gateway, wrong when it does not.
Returns¶
dict
{"ok": bool, "version": str, "live": bool, "checks": [...]} where
each check is {"name", "label", "status", "detail", "hint", "error"}
and status is 'ok' / 'failed' / 'skipped'. ok is
True when no check failed (a skipped check does not fail the run).
progress(event, data) (optional) is called with "start" and
{"name", "label", "index", "total"} before each check, then with
"result" and the check's row once it is done, so a UI can show each row
as it arrives instead of waiting for the slowest check. A failing hook is
logged and ignored: the report never depends on it.
Notes¶
Never raises, and never returns or prints a secret — only variable names, URLs, model names and presence flags.
Examples¶
import tdfs4ds report = tdfs4ds.check_configuration() # doctest: +SKIP report["ok"] # doctest: +SKIP True tdfs4ds.check_configuration(live=False) # doctest: +SKIP
Exercise the Vector Store end to end on throwaway objects.
This writes to the database, unlike every other tdfs4ds diagnostic: it creates a small table and a vector-store collection, then removes both. Names are unique per run, and nothing pre-existing is touched.
Parameters¶
cleanup : bool, default True
Destroy the collection and drop the table at the end. False keeps
both for inspection — the report then names them, and it is on you to
remove them.
display : bool, default True
Print the per-step report.
repro : bool, default True
On failure, build the standalone reproducer into report["repro"].
Returns¶
dict
{"ok", "version", "collection", "table", "steps": [...], "repro",
"leftovers": [...]}. ok is True when no step failed. Each step is
{"name", "label", "status", "detail", "error", "elapsed_s"} with
status ok/failed/skipped.
Report which AI Studio platform variables are visible.
Never raises and never returns a credential value -- a token is reported as present with its decoded username, not as its own text.
Returns¶
dict
{'on_ai_studio': bool, 'found': [name, ...], 'missing': [name, ...],
'username': str or None}. on_ai_studio is True when any platform
variable is present; username is the database user decoded from the
token's claim.
generate_ai_studio_env(path=None, *, auth=None, token=None, mcp='gateway', mcp_token=None, llm=None, prompt=True, include_secrets=False, check=True, pin=False, force=False, **overrides)
Generate a tdfs4ds .env from AI Studio's platform environment.
Derives everything the platform exposes, asks for the rest, checks that the result actually works, and only then writes the file.
The order matters. An earlier version wrote first and offered to check
afterwards, filling anything unanswered with <set-this>; that produces
a file which looks finished, loads that literal string as a real value, and
fails much later somewhere that names neither the variable nor this
function. So now a blank answer is re-asked with the reason it is
needed, the configuration is validated before the write, and a file with
gaps or failing checks is written only if you say so.
Parameters¶
path : str, optional
Where to write. None writes nothing and only returns the preview.
tdfs4ds's .env discovery is first-match-wins (./.env, then
~/.tdfs4ds/.env), so a file written anywhere else has to be passed
to load_config(dotenv_path=...) explicitly.
auth : {'password', 'jwt', None}
None (default) asks, on AI Studio and with a terminal, laying
out the trade-off -- it is the one choice here that cannot be derived
and that changes how every component connects. Falls back to
'password' when there is nobody to ask.
``'password'`` writes a durable configuration: one database
password, asked once and reused for the Vector Store.
``'jwt'`` writes no password at all -- the platform token serves all
three -- but **the token expires about two hours after it was issued**,
so the file stops working and has to be regenerated. The unused variant
is emitted commented out either way.
token : str, optional
The token everything under auth='jwt' authenticates with — the
database logon (TD_LOGDATA), the Vector Store, and the MCP gateway.
**AI Studio hands out two tokens that look alike and are not
interchangeable.** The session's ``OAUTH_ACCESS_TOKEN`` authenticates
against the platform's own services; ``%gen_token``'s ``user_token`` is
minted for a **Vantage logon**, which is what the notebooks already
pass to ``create_context(logmech='JWT', logdata='token=...')``. Where
the platform token is not a database credential — confirmed on at least
one site — every component that opens a database session with it fails
with the database's own ``Error 8017``, and the gateway does exactly
that on the caller's behalf::
%gen_token
tdfs4ds.generate_ai_studio_env('.env', token=user_token)
Defaults to ``OAUTH_ACCESS_TOKEN``, which is right wherever the
platform token *is* database-capable. Under ``auth='jwt'`` the logon is
attempted before the file is written, so a token the database refuses
is reported instead of being discovered later as an unexplained 8017.
Ignored under ``auth='password'`` except for the MCP bearer.
mcp_token : str, optional The bearer for the Enterprise gateway alone, overriding token — use it when the gateway and the database want different tokens. There are two different tokens in AI Studio and only one is a database credential.
The gateway validates the bearer and then opens a Teradata session with
it -- which is why a failure there surfaces as the *database's*
``Error 8017`` rather than an HTTP 401. The session's
``OAUTH_ACCESS_TOKEN`` passes the first step and has been observed
failing the second, so where that happens supply the token minted for a
Vantage logon instead -- the one ``%gen_token`` leaves in
``user_token``, which the notebooks already use for
``create_context(logmech='JWT', logdata='token=...')``::
%gen_token
tdfs4ds.generate_ai_studio_env('.env', mcp_token=user_token)
Falls back to ``MCP_TOKEN`` from the environment, then to *token*, then
to :func:`generate_mcp_token`'s device flow.
mcp : {'gateway', 'incluster', None}
'gateway' (default) is https://{JH_FQDN}/one-td/mcp with
edition=enterprise, the verified combination. 'incluster' uses
TD_MCP_BASE_URL, which appears to be a Community-edition server (it
embeds its own credentials, so tdfs4ds sends none). None omits MCP.
llm : {'incluster', 'ingress', None}
Which route to the LiteLLM gateway. 'incluster' is the platform's
own TD_LITELLM_BASE_URL -- the shortest path, and the one an
AgentOps pod takes, but http://, so the API key and every prompt
travel in clear. 'ingress' is https://{JH_FQDN}/one-td/litellm,
the same one-td pattern as the MCP gateway, which adds TLS at the
cost of a hop. Whichever is not chosen is written commented out.
``None`` (default) **asks**, like *auth*: which route the traffic takes
is not derivable from the environment -- the platform publishes only
the plaintext one -- and defaulting to it silently is how a file ends
up sending an API key in clear without anyone choosing that. Without a
terminal it resolves to ``'incluster'``, the historical default.
On an ``http://`` URL ``INSTRUCT_MODEL_VERIFY_SSL`` is omitted entirely:
there is no TLS to verify, and setting it only builds a custom HTTP
client and warns on every call.
prompt : bool, default True Ask for values that cannot be derived, re-asking once or twice when the answer is blank. Without a terminal this is ignored and the unanswered values are reported as gaps instead, so the function stays usable from a script -- it just will not write a file it knows to be incomplete unless force says to.
Two answers are treated specially. The **instruct model** is chosen
from the list the endpoint itself serves (once the URL and key are
known), defaulting to a Ministral 14B if one is offered. The **API
key** is written as ``None`` when you have none, which loads as
``None`` -- correct for a gateway that does not require one, and honest
rather than a placeholder pretending to be a key.
include_secrets : bool, default False
The returned text renders secrets as set (hidden, N chars); the
file written to path always carries the real values. Pass True to
get the real text back -- then treat it as a secret.
check : bool, default True
Run describe_config() and check_configuration() before
writing, against the values staged into the session, and report what
fails. Failing checks block the write unless you confirm (or force).
Set False to skip, which is also what makes this function usable
offline.
One caveat: checking spends the credentials. ``set_auth_token`` is
called for real, so the Vector Store session is configured from here on
whether or not the file ends up written.
pin : bool, default False
After a successful write, record what the JSON-constraints check
discovered -- typically STRUCTURED_OUTPUT_METHOD -- into the file
just written, stamped with the endpoint and date. Requires check.
force : bool, default False
Write even with unanswered values or failing checks, without asking.
This is how a non-interactive script gets a file at all, and how the
tests do.
**overrides
Supply a value directly instead of deriving or asking for it, keyed by
the tdfs4ds variable name with or without the prefix -- e.g.
instruct_model_model='...' or TDFS4DS_TD_PASSWORD='...'.
Returns¶
str The rendered file, with secrets hidden unless include_secrets. It is returned whether or not the file was written, so a refused write still shows you exactly what it would have contained.
Notes¶
A successful write ends with load_config(dotenv_path=path), so the
running session is configured by the time this returns. Configuration is
otherwise read once, during import tdfs4ds -- a file written afterwards
would not reach the module, which is the load-order trap this package
documents everywhere. Doing it here means there is no step left to forget.
Examples¶
print(tdfs4ds.generate_ai_studio_env('.env')) # doctest: +SKIP tdfs4ds.generate_ai_studio_env('.env', auth='jwt') # doctest: +SKIP tdfs4ds.generate_ai_studio_env('.env', check=False, force=True) # doctest: +SKIP
generate_mcp_token(*, idp_url=None, client_id=None, site_id=None, scope='openid profile email', timeout=300.0, open_browser=True)
Obtain an Enterprise MCP bearer token by OAuth 2.0 device flow.
The Enterprise MCP gateway on AI Studio does not accept the database username and password. It wants a bearer token from the platform's identity provider, and that token is obtained by the Device Authorization Grant: this prints a URL and a short code, you approve in a browser, and it polls until the token is issued.
That is why mcp='gateway' writes MCP_SERVER_HEADERS rather than
MCP_USERNAME/MCP_PASSWORD. Sending Basic credentials there gets the
database's own rejection -- Error 8017, The UserId, Password or Account
is invalid -- which reads like a wrong password and is really the wrong
kind of credential.
Parameters¶
idp_url : str, optional
Identity provider. Defaults to the production IdP; see
:data:MCP_IDP_URLS for the staging and dev ones.
client_id : str, optional
Defaults to onetd-<SITE_ID>.
site_id : str, optional
Defaults to the first label of JH_FQDN, upper-cased.
scope : str
OAuth scopes requested.
timeout : float
Seconds to keep polling before giving up.
open_browser : bool
Try to open the verification URL. The URL is printed either way.
Returns¶
str or None
The access token, or None if it could not be obtained. Never
raises: this is one input among several, and a failure here should
leave the caller free to carry on and paste a token by hand.
Notes¶
The token is not OAUTH_ACCESS_TOKEN. That one is the notebook
session's own token, issued for a different audience; this one is issued to
the onetd-<SITE> client for the MCP gateway.
The site id behind fqdn (default: JH_FQDN), or None.
tdicamworldpp69.private.preprod.cloud.teradata.com -> TDICAMWORLDPP69.
configure_interactively(path='.env', *, sections=SECTIONS, check=True, write_secrets='ask', prompt=True)
Configure tdfs4ds by answering questions; update path in place.
Existing TDFS4DS_* lines are the defaults and are edited in place, so
comments, ordering and unrelated variables survive. A missing file is
created. Only values that changed are written.
Parameters¶
path : str or Path, default '.env'
The file to update.
sections : iterable of str
Any of 'connection', 'instruct', 'embedding',
'vector_store'. Default: all, in that order.
check : bool, default True
Run the live checks (model call, vector-store service). With False
only the questions are asked and model lists are not fetched.
write_secrets : {'ask', 'never', 'always'}, default 'ask'
Whether API keys, passwords and tokens may be written to the file.
'ask' asks once, with the default answer no.
prompt : bool, default True
False (or no terminal) asks nothing and writes nothing.
Returns¶
dict
{'path', 'updated', 'added', 'unchanged', 'skipped', 'checks'}
-- variable names and statuses only, never a value.
Ask the instruct endpoint which JSON-constraint syntaxes it accepts.
Sends nine chat completions (see :func:_variants) and classifies each
as enforced / accepted / rejected / unreachable, then infers
the serving stack from the pattern and names the tdfs4ds setting to change.
Parameters¶
model : str, optional
Overrides tdfs4ds.INSTRUCT_MODEL_MODEL for this probe.
timeout : float, default 30.0
Per-request timeout, in seconds.
display : bool, default True
Print the ASCII report. The return value carries the same information.
Returns¶
dict
{"ok", "status", "endpoint", "model", "variants", "verdicts",
"engine", "recommend", "notes", "detail"}. status is 'ok' /
'failed' / 'skipped'; recommend maps a TDFS4DS_*
variable name (without the prefix) to the value to set, and is empty
when nothing needs changing.
Notes¶
Never raises, and never returns or prints the API key.
rebuild_lineage_graph(objects=None, all_data_domains: bool = True, check: bool = True, prune: bool = True, extra_views=None, require_exists: bool = True) -> int
Rebuild the persisted lineage graph — fully, or scoped to a list of objects.
Use this to initialise FS_LINEAGE_GRAPH on a store that already has
registered processes/datasets, to refresh a graph whose edges have drifted,
or to scoped-refresh just the objects you edited. The incremental write hook
(:func:record_process_lineage, called from upload_features) keeps lineage
current going forward; this function recomputes on demand.
The table is valid-time temporal: every write is diff-based, so re-running
on an unchanged store touches zero rows (no history churn). There is no blind
delete+insert — :func:record_process_lineage leaves unchanged edges alone,
inserts new ones, and closes (CURRENT VALIDTIME DELETE) ones that disappeared.
Full refresh (objects=None):
- Ensure the table/view exist (idempotent).
- Seed from every registered process view and every dataset (all data
domains by default), re-deriving each subgraph via
:func:
record_process_lineage(diff-based — unchanged edges are untouched). - When
prune=True(default), edges whose parent object is no longer a seed (e.g. a since-dropped process) are closed withCURRENT VALIDTIME DELETE. This is a targeted prune of stale parents, not a wipe — it does not re-open the edges that are still valid.
Scoped refresh (objects=[...]): no prune, no catalog-wide seed. The names
are validated by :func:resolve_objects (raises on any unmatched name — case /
quoting / db.name safe), then record_process_lineage refreshes each object's own
outgoing edges (diff-based). Use this when a (possibly non-process) view was edited.
When check=True, :func:check_downstream_processes runs afterwards.
Parameters¶
objects : str | Iterable[str] | None
When None (default), full refresh. Otherwise refresh only these objects.
all_data_domains : bool
Full-refresh only: all DATA_DOMAINs (default) or the active one.
check : bool
Scoped refresh only: run the downstream EXPLAIN compile check afterwards.
prune : bool
Full-refresh only: close edges of parents that are no longer registered
(default True). Set False to never delete (purely additive refresh).
extra_views : str | Iterable[str] | None
Full-refresh only: additional standalone view names to seed into the
graph alongside every registered process view + dataset. Use this to
capture a non-process / non-dataset view (e.g. a hand-built or
crystallized reporting view) that nothing registered references — it
would otherwise never enter FS_LINEAGE_GRAPH. Each name is validated
via :func:resolve_objects (case / quoting / db.name safe; raises on
any unmatched name). There is no automatic schema scan — only the
names you pass.
require_exists : bool
Scoped refresh only. True (default): an unknown name raises before
anything is written. False: unknown names are logged and skipped, and
the others are refreshed (what a migration wants).
Returns¶
int Number of edge changes written (0 when nothing changed).
Deterministic compile check for the process views downstream of changed objects.
When a non-process view changes its result columns, the process views that
read it can silently break. This runs EXPLAIN SELECT * FROM <process_view>
on every registered process downstream of the given object(s) — EXPLAIN only
verifies the query compiles (all columns resolve), it does not execute it.
Same compile-validity idiom as the pre-check in upload_features.
Parameters¶
objects : str | Iterable[str]
Changed object name(s); resolved via :func:resolve_objects.
Returns¶
dict
{"checked": int, "ok": [view, ...],
"broken": [{"view", "process_id", "error"}, ...]}.
Broken processes are also logged as a warning.
Normalise and verify a user-supplied list of object names so case / "
quoting / db.name variance never causes a silent mismatch.
Each input ("db"."NAME", db.name, NAME, any case) is parsed with
the same _parse_node_name + _norm helpers record_process_lineage uses to
write the table, so a resolved name compares equal to stored rows. An
object is considered to exist if it is a node in the persisted lineage graph,
or (fallback, for a freshly-created view not yet in the graph) present in
DBC.TablesV.
Parameters¶
objects : str | Iterable[str]
One name or a list of names.
require_exists : bool
When True (default), raise ValueError naming every unmatched input.
When False, unmatched inputs are logged and dropped.
graph : networkx.DiGraph, optional
Pre-loaded lineage graph (avoids a reload when the caller already has one).
Returns¶
(resolved, unmatched)
resolved is a list of canonical (DB_UPPER, OBJECT_UPPER) tuples
(default DB filled from tdfs4ds.SCHEMA when the input was unqualified);
unmatched is the list of original input strings that resolved to nothing.
Retrieve and display all data domains available in the feature store.
Queries both the process catalog and the feature catalog for distinct DATA_DOMAIN values, then prints them with the currently active domain marked with an asterisk.
Parameters: - verbose (bool): If True, prints the list of data domains with the current one marked.
- list or None: When verbose=False, returns the list of data domain strings. When verbose=True, prints the list and returns None.
Set the active data domain for feature store operations.
All subsequent feature queries, registrations, and interactions with the feature store are scoped to this domain.
Parameters: - data_domain (str): The name of the data domain to set as active.
Raises: - ValueError: If the data domain does not exist in the feature store.
Create a new data domain in the feature store.
A data domain serves as a logical grouping for features, isolating them by project, business unit, or data type. The domain is activated locally; features uploaded afterward will be tagged with this domain.
Parameters: - data_domain (str): The name of the new data domain to be created.
Retrieve a Teradata DataFrame of features registered in the feature store at the current (or time-travelled) valid time.
Parameters¶
all_data_domains : bool, optional If False (default), only features belonging to the current DATA_DOMAIN are returned. Set to True to return features from all data domains.
Retrieve a Teradata DataFrame of processes registered in the process store at the current (or time-travelled) valid time.
Parameters¶
all_data_domains : bool, optional If False (default), only processes belonging to the current DATA_DOMAIN are returned. Set to True to return processes from all data domains.
Retrieve a Teradata DataFrame of all datasets registered in the dataset catalog.
Cross-store federation (v0.3.1.148): when read-only stores are attached, browses
a UNION ALL of every store's dataset-catalog view (with a FEATURE_STORE
provenance column); otherwise byte-identical. Federating here — at the read-only
accessor — keeps DatasetCatalog's own write/registration logic primary-only.
Return the entity definition (columns and types) for the given dataset ID. If dataset_id is None, returns entity info for all registered datasets. Federated across read-only stores when attached (dataset IDs are globally unique UUIDs, so the dataset_id filter selects the owning store's rows).
Return the feature list for the given dataset ID. If dataset_id is None, returns feature info for all registered datasets. Federated across read-only stores when attached (see get_dataset_entity).
Return how each dataset was built: BUILD_STRATEGY (MULTIJOIN / GROUPED),
JOIN_TYPE, TIME_MODE (CURRENT / AS_OF / TIME_MANAGER), the
AS_OF_TIMESTAMP or the TimeManager's location, and SOURCE_DATASET_ID
for a dataset rebuilt from another one.
Primary store only. Returns None when the build table does not exist
(a store whose catalog was created before 0.3.1.216 and never reconnected by
a user allowed to create it). A dataset registered before that version has
no row until it is rebuilt or restore_dataset_catalog(dry_run=False) runs.
Return the window of each historical dataset (build_historical_dataset),
one row per dataset, from FS_V_<catalog>_WINDOW:
WINDOW_START/WINDOW_END: the bounds when queried (a sliding window moves);WINDOW_STARTNULL means from the earliest stored value;START_KIND(FIXED/END_MINUS_DURATION/EARLIEST) andEND_KIND(FIXED/START_PLUS_DURATION/NOW);- the definition:
START_TS,END_TS,DURATION_ISO,DELAY_ISO(plus months/seconds),SLIDINGand the readableWINDOW_TEXT.
A snapshot dataset has no row, so join it to the catalog with a LEFT OUTER
JOIN on DATASET_ID. Primary store only. Returns None when the
window table does not exist (needs one connect() by a user allowed to
create it).
Return the columns of each dataset that read a feature at an offset (lags and
leads, build_dataset(feature_offsets=...)), one row per column, from
FS_V_<catalog>_OFFSETS:
FEATURE_NAME(the dataset column, e.g.AMOUNT_LAG_1D),BASE_FEATURE_NAME,FEATURE_ID,FEATURE_VERSION(the process);OFFSET_DIRECTION(LAG/LEAD/NONEfor '0'),OFFSET_MONTHS/OFFSET_SECONDS(signed),OFFSET_ISO(-P1D),OFFSET_TEXT("1 day before");OFFSET_STRATEGY(PIVOT/PER_OFFSET) andLEAD_HORIZON(NOW: a lead is NULL until its instant is reached).
A column without offset has no row: join it to get_dataset_features with a
LEFT OUTER JOIN on (DATASET_ID, FEATURE_NAME). Primary store only. Returns
None when the table does not exist (needs one connect() by a user
allowed to create it).
Retrieves the process ID associated with a given view name in a database.
This function is designed to obtain the process ID for a specified view within a database.
Initially, it formats the provided view_name to conform to the database's naming conventions,
including handling scenarios where the view name is prefixed with a database name. After formatting
the view name, the function queries a list of processes using an assumedly predefined function
list_processes. It then filters this list to find the process corresponding to the formatted view name
and returns the associated process ID. The function assumes specific column names ('VIEW_NAME' and
'PROCESS_ID') in the data structure returned by list_processes and that these columns are in a
specific format (e.g., uppercase for view names).
view_name (str): The name of the view for which the process ID is to be retrieved. This can be either just the view name or a combination of database name and view name, formatted as 'database.view'.
int: The process ID associated with the given view name. This ID is retrieved from a list of processes, where it is associated with the view name that matches the input.
Raises: IndexError: If no process is found with the given view name.
remove_process(process_id, nonsequence_validtime=True, drop_view=False, refresh_index=True, force=False)
Deletes a process from the feature store's process catalog based on the given process ID, with what it leaves elsewhere.
Removed, before the catalog rows (each step non-fatal: a failure is logged as a warning and does not prevent the process from being removed):
- its documentation (business logic, features, EXPLAIN analysis, business dictionary entries of its view) and the provenance of that documentation;
- its entries in the vector index (Chroma: deleted by id; Teradata Vector Store: the collections are refreshed, since their embeddings are keyed by row position);
- its stored SQL versions (
FS_PROCESS_SQL) and its FilterManager / TimeManager catalog rows; - the lineage edges of its view, when the view no longer exists (
drop_view=Truedrops it first). A view kept after its process is removed is still a real object, so its edges stay.
The values the process stored in the FS_T_* tables are kept (their history).
Args:
process_id (str): The unique identifier of the process to be removed.
nonsequence_validtime (bool): When True (default), uses NONSEQUENCED VALIDTIME DELETE to
remove all temporal versions from the catalog.
drop_view (bool): Also drop the process view (default False).
refresh_index (bool): On the Teradata Vector Store backend, refresh the collections after the
documentation rows are deleted (default True). False leaves it to the next
build_vector_index().
force (bool): Remove it even when datasets read its values or other process views read its
view (0.3.3.27). By default the removal is refused with
tdfs4ds.dependencies.DependentObjectsError, which lists them in the order
they must be removed; tdfs4ds.plan_removal([process_id]) and
tdfs4ds.write_removal_script remove them all, in that order.
Each removal, and each refusal, is recorded in FS_REMOVAL_LOG (0.3.3.27,
tdfs4ds.get_removal_log()), with the view name so the process can still be named
once its catalog rows are gone.
Returns: None.
set_process_options(process_id: str, close_absent_entities: Optional[bool] = None, track_entity_refresh: Optional[bool] = None, dataset_time_manager: Optional[bool] = None, valid_time_column=None, force_varchar_length=None) -> dict
Change the execution options of an already-registered process.
close_absent_entities=True: every later run() / roll_out() of the
process ends the validity of the entities it no longer produces, at the run's
valid time. False switches it off. None leaves it as it is.
track_entity_refresh=True: every later run records, in an admin table
(FS_R_*, see tdfs4ds.entity_refresh), when each entity was last confirmed
by the process. False switches it off (the recorded history is kept).
dataset_time_manager=True: the dataset view a later run() builds follows the process's
TimeManager (build_dataset(time_manager=...)); False switches it off.
valid_time_column='<COLUMN>' (0.3.5.0): every later run writes each row of the process view
at its own valid time, read from that DATE / TIMESTAMP column (see upload_features).
False or '' switches it off. The next run replaces each entity's history from its
earliest valid time in the load onward.
force_varchar_length=N (0.3.5.1): the VARCHAR rounding later runs use for features they
register ('none' = exact lengths). A registered feature keeps its stored type: a run whose
source no longer fits it logs a warning naming tdfs4ds.plan_type_change.
Returns {"process_id", "close_absent_entities", "track_entity_refresh"} (the options
now in force). Raises ValueError for an unknown process or a FilterManager whose
columns are not entity columns, and RuntimeError when an options table cannot be used.
Removes all database entities (tables and views) associated with a given entity name within a specific data domain, and then removes the entity's metadata from the feature catalog.
This function first constructs a SQL query to select distinct table and view names associated with the given entity name and data domain from the feature catalog. It then iterates over the list of tables and views, attempting to drop each one. Progress and status of these operations are tracked using a progress bar. If an error occurs during the drop operation, the error message is displayed in the progress bar description. Finally, the function constructs and executes another SQL query to delete the entity's metadata from the feature catalog.
Parameters:
- entity (str): The name of the entity to be removed.
- data_domain (str, optional): The data domain where the entity is located. If not provided,
the default data domain from tdfs4ds.DATA_DOMAIN is used.
The function utilizes the tdfs4ds.DEBUG_MODE flag to control the printing of SQL queries for debugging purposes. If
DEBUG_MODE is set to True, the queries for selecting and deleting entities are printed to the console.
Dependencies:
- Requires the tdfs4ds module for configuration constants like DATA_DOMAIN, SCHEMA, and FEATURE_CATALOG_NAME,
as well as the DEBUG_MODE flag.
- Requires the tdml module or object with an execute_sql method for database operations.
- Uses the tqdm module to display a progress bar for the operations.
Note: - This function assumes that the necessary permissions are in place for dropping tables and views, as well as deleting metadata entries from the feature catalog. - Exception handling within the dropping operations is limited to displaying the first line of any encountered error messages in the progress bar description. Further error handling or cleanup may be necessary depending on the specific requirements and database configuration.
- None. The primary effect of this function is to alter the state of the database by removing entities and their metadata, not to return data to the caller.
Delete the values of a specific feature for given entities from the feature table within a specified data domain.
This function constructs and executes two SQL queries against a Teradata database to remove a feature specified by its name and entity identifiers. The first query retrieves the table name where the feature resides, based on the feature name, entity, and data domain. The second query deletes the feature values from the identified table.
Parameters:
- feature_name (str): The name of the feature to be removed.
- entity_id (str or list of str): Entity identifier(s). If a string is provided,
it will be converted to a single-element list. The list is always sorted
alphabetically before use.
- data_domain (str, optional): The data domain where the feature is located.
If not specified, the function uses the default data domain defined in
tdfs4ds.DATA_DOMAIN.
Behavior:
- The function checks if the DEBUG_MODE flag in the tdfs4ds module is set to True.
If so, it prints the generated SQL queries and the resolved table name for debugging.
- If the feature table cannot be resolved, the function returns without executing
a delete query.
Returns: - None
Note:
- The function assumes the presence of a module tdfs4ds with predefined constants
such as DATA_DOMAIN, SCHEMA, FEATURE_CATALOG_NAME_VIEW, and a flag DEBUG_MODE.
- It also assumes a tdml module or object with an execute_sql method capable of
executing SQL queries against a Teradata database and fetching the results.
- Exceptions related to SQL execution or connection issues may be raised but are not explicitly handled, except for printing the error message.
Remove a feature from the feature catalog, with its documentation and index entries.
A feature is the triplet (data domain, entity, name) and has ONE catalog entry, whatever the number of processes computing it (each process writes its values under its own FEATURE_VERSION).
Parameters:
- feature_name (str): The name of the feature to be removed.
- entity_id (str or list of str): Entity column name(s); sorted before use.
- data_domain (str, optional): The data domain of the feature. Defaults to
tdfs4ds.DATA_DOMAIN.
- process_id (str, optional, 0.3.3.23): remove the feature as produced by this
process. When another process also computes the feature, its catalog entry is
kept and only this process's part goes: its documentation of the feature
(process feature documentation, the business-dictionary column of its view,
the provenance and vector-index entries). When no other process computes it,
the feature is removed entirely, as without process_id. A process that is
no longer registered is accepted: the decision rests on the other producers.
None (default) removes the feature whatever process computes it.
- force (bool, 0.3.3.27): remove it even when datasets read it. By default the
removal is refused with tdfs4ds.dependencies.DependentObjectsError, listing
the datasets (and the processes reading them) in the order they must be
removed. Nothing depends on it when another process still computes it and
process_id scopes the removal to one process.
Each removal, and each refusal, is recorded in FS_REMOVAL_LOG (0.3.3.27).
Stored values are never deleted: they stay in their FS_T table (the storage scan of the consistency report lists the values of a removed feature as orphans).
- None without
process_id. Withprocess_id, a report dict:{"mode": "process" | "full", "others": [other producing process ids], "listed": whether process_id was found among the producers};Nonewhen the producers could not be read (nothing is removed then).
drop_dataset(dataset_id: str, schema_name: str = None, purge: bool = False, close_lineage: bool = True) -> None
Remove a dataset entry (catalog + entity + feature rows) by DATASET_ID.
purge=True physically removes all temporal rows (history included);
the default soft-deletes by valid-time-closing the currently-valid rows.
close_lineage=False keeps the view's lineage edges (the view stays).
To also remove its documentation and index entries, use
:func:tdfs4ds.removal.remove_dataset.
remove_dataset(dataset: str, *, drop_view: bool = False, purge: bool = False, dry_run: bool = True, refresh_index: bool = True, force: bool = False) -> dict
Remove a dataset from the catalog, with what it leaves behind (0.3.3.19).
dataset: its DATASET_ID, database.name or name. Removes the catalog,
entity, feature, build and window rows and closes its lineage edges
(drop_dataset), then its business-dictionary rows (description,
sections, columns), their provenance and its vector-index entries. The view
itself is kept unless drop_view=True.
dry_run=True (default) changes nothing and returns what would go, with the
processes whose view reads it (dependents). Only a dataset of the primary
store can be removed. A dataset that process views read is not removed unless
force=True (0.3.3.27): tdfs4ds.dependencies.DependentObjectsError
lists them in the order they must be removed. Every step after the catalog is
non-fatal and reported; the call raises only when the dataset is unknown or
ambiguous, or when the catalog rows cannot be removed.
run(process_id=None, return_dataset=False, force_compute=False, force_varchar_length=None, dataset_view_name=None, grouped=False, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, time_manager=None, close_absent_entities=None, _time_manager_checked=False, track_entity_refresh=None, dataset_time_manager=None, *, at=None, data_domain=None, order='dependencies')
Execute a specific process from the feature store identified by the process ID.
Uses global logger for diagnostics (gated by tdfs4ds.DISPLAY_LOGS).
Several processes at one time (0.3.6.0): process_id may be a list, or pass
data_domain='X' (True = tdfs4ds.DATA_DOMAIN) for every process of a data domain
(primary store only). They run one after the other in dependency order (from the
lineage graph; order='given' keeps yours), all at ONE time: at='2024-03-31', else
the database clock read once (at='now', the default for a list), or the current
date of time_manager=. Each TimeManager view the processes read (their soft clones)
is set to that date for the computation and put back on its cursor afterwards: no
cursor moves. A process reading no TimeManager is stored at the same time, with a
warning. A failed process's downstream processes are skipped; on_failure applies
per process ('raise' raises FeatureIngestionError after the list, 'stop' stops at
the first failure). Returns {process_id: 'COMPLETED' | 'FAILED: ...' |
'SKIPPED: ...'}. run('pid', at=...) computes one process at that date and returns
what run('pid') returns.
grouped is forwarded to build_dataset whenever a dataset view is (re)built.
See upload_features for details. Default: False (classic multi-join, unchanged
behaviour). Note: not persisted in the process catalog — re-running an existing
process_id without passing grouped=True again reverts to the classic join.
Valid time of the ingested values, decided once per run:
tdfs4ds.FEATURE_STORE_TIME, when set — whatroll_outdoes at each step.time_manager=— the manager's current business date (the pairing is recorded, asroll_outrecords it).- the TimeManager recorded for the process (by
roll_outor a previoustime_manager=), at its current business date. - otherwise real time: the database time, read once, shared by every feature and partition of the run.
For 2 and 3, FEATURE_STORE_TIME is set for the duration of the run (the
follow-up row records that date) and restored afterwards. Running without a
FEATURE_STORE_TIME or a TimeManager is a fallback, not a best practice.
A time_manager= the process view does not read (checked on the lineage graph,
deep) is governed by tdfs4ds.TIME_MANAGER_CHECK: 'warn' (default) runs as
before but records no pairing; 'error' raises ValueError before computing
anything; 'off' skips the check. A real-time run of a process whose view reads a
TimeManager that is not declared logs a warning naming link_time_managers.
close_absent_entities: None (default) applies the option recorded for the
process (upload_features(..., close_absent_entities=True) or
set_process_options); True / False override it for this call only. When on,
the entities the process no longer produces have their validity ended from the
run's valid time on (the MERGE's "from t onward" rule).
track_entity_refresh: None (default) applies the option recorded for the process
(upload_features(..., track_entity_refresh=True) or set_process_options); True /
False override it for this call only. When on, the run records in an admin table
(FS_R_*, read by tdfs4ds.entity_refresh) that it confirmed each entity of a batch,
whether or not the value changed. A failed write is a warning; the run stays COMPLETED.
dataset_time_manager: None (default) applies the setting recorded for the process
(upload_features(..., dataset_time_manager=True) or set_process_options); True / False
override it for this call only. When on, the dataset view built by this run (dataset_view_name)
follows the process's TimeManager (the time_manager= argument, else the recorded one): it is
built with build_dataset(time_manager=...), so tm.update() moves it. A dataset that already
exists in another mode is left as it is, with a warning naming rebuild_dataset.
Other parameters:
process_id: a PROCESS_ID, or a list of them (0.3.6.0).return_dataset: return the dataset view built by the run (withdataset_view_nameortdfs4ds.BUILD_DATASET_AT_UPLOAD) instead of None.force_compute: recompute even when the follow-up table records the run as COMPLETED at this valid time (default False: such a run is skipped).force_varchar_length: VARCHAR rounding of new text features; None = the value recorded for the process (seeupload_features).dataset_view_name,grouped: a dataset to (re)build after the run, and its strategy.on_failure('continue' | 'stop' | 'raise') /raise_on_failure(True = 'raise'): what a failed FilterManager segment does; 'raise' attempts every segment, then raisesFeatureIngestionError.check_upstream/strict_upstream: refuse the run when an upstream process (lineage graph) FAILED at this valid time / is also RUNNING.time_manager: compute at this manager's current date (see above); a process reading a soft clone of it gets the clone set to that date for the run (0.3.6.0).at,data_domain,order(keyword, 0.3.6.0): see "Several processes at one time".
upload_features(df, entity_id, feature_names, metadata={}, primary_index=None, partitioning='', filtermanager=None, entity_null_substitute={}, force_compute=True, force_varchar_length=None, dataset_view_name=None, optimize_query=None, update_view=True, document=None, max_candidates=None, cascade_redocumentation=None, vector_indexing=None, sql_query=None, target_view_name=None, art_sql=None, art_layers=None, grouped=False, return_process_id=False, run_now=True, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, explain=None, time_manager=None, close_absent_entities=False, storage_tag=None, track_entity_refresh=False, dataset_time_manager=False, valid_time_column=None, soft_clone=None)
Upload feature data to the feature store for a specified entity.
All diagnostics go through logger_safe() which respects tdfs4ds.DISPLAY_LOGS.
The feature source can be supplied in three ways:
- A teradataml DataFrame (
df=...) — the classic path. - A fully-qualified view name (
df='SCHEMA.view') — for SQL users who already have a view. The schema is mandatory; a bare view name (no schema) is rejected. The view must already exist and be executable. - A SQL query (
sql_query='SELECT ...') together with a mandatorytarget_view_name='SCHEMA.view'. The query is validated withEXPLAIN, a view is created attarget_view_name(the schema is mandatory; the view must NOT already exist), and that view — which is permanent and becomes the process view re-executed byrun(process_id)— is used as the feature source. Whensql_queryis given it takes precedence overdf.
Recoverable input problems (missing schema, target view already exists, SQL
or view fails to compile / is not executable) do not raise a traceback:
a clear message with a fix is logged via logger_safe('error', …) and the
function returns None. Check the log when a call returns None.
Parameters¶
df : teradataml DataFrame or str
The feature source: a DataFrame over a permanent view, or 'SCHEMA.view'.
Pass None with sql_query.
feature_names : list of str
The columns of the source to register as features.
metadata : dict, optional
Free-form description stored with the process (METADATA JSON). Default: {}.
primary_index : list of str, optional
Primary index of the storage tables. Default: the entity columns.
partitioning : str, optional
Partitioning clause of the storage tables. Default: none.
filtermanager : FilterManager, optional
Segments the process runs one at a time (see the fs-filter skill).
force_compute : bool, optional
Recompute even when the follow-up table records the run as COMPLETED at this
valid time. Default: True.
dataset_view_name : str, optional
'SCHEMA.view' of a dataset built from the features after the run.
on_failure : {'continue', 'stop', 'raise'}, optional
What a failed FilterManager segment does: 'continue' (default) records it
and goes on, 'stop' stops at the first one, 'raise' attempts every
segment then raises FeatureIngestionError.
raise_on_failure : bool, optional
Shorthand: True = on_failure='raise'. An explicit on_failure wins.
check_upstream : bool, optional
Refuse the run when an upstream process (from the lineage graph) FAILED at this
valid time. Default: False.
strict_upstream : bool, optional
With check_upstream, also refuse when an upstream is RUNNING.
sql_query : str, optional
A SQL SELECT defining the feature source. Requires target_view_name.
The view is created from this query (after an EXPLAIN check) and kept.
target_view_name : str, optional
'SCHEMA.view' — where to create the view for sql_query (mandatory
when sql_query is given; schema required; must not already exist).
art_sql : list of str, optional
Opt-in UAF ART view mode. A list of solitary SQL statements that
materialise a Teradata UAF operator's Analytical Result Table (ART) using
EXECUTE FUNCTION ... INTO VOLATILE ART(<name>). When provided, the
process is registered with PROCESS_TYPE='uaf art view' and, on every
run(), these statements are executed in list order, per FilterManager
partition, immediately before the ingestion CTAS (same session, so the
session-scoped volatile ART is visible to the CTAS). This unlocks features
whose value lives in a secondary ART layer (e.g. ARIMA fit-metrics
R_SQUARE/MAPE/AIC in ARTFITMETADATA, residual tests over
ARTFITRESIDUALS, TD_PACF) — which a single ingestion statement
cannot reach. df/sql_query+target_view_name still define the
plain feature view (the one reading the layer via
TD_EXTRACT_RESULTS(ART_SPEC(..., LAYER(...)))); art_sql is stored in
the process METADATA and never turned into a view (a view cannot contain
INTO VOLATILE ART). Order matters — a later statement may read an ART
an earlier one created. For this mode optimize_query is force-disabled
and the view EXPLAIN in documentation is skipped (the ART does not exist at
optimize/document time). Requires art_layers. Default: None (classic
'denormalized view' path, unchanged).
art_layers : list of str, optional
Required when art_sql is given: the declared LAYER(...) name(s) the
feature view reads (e.g. ['ARTFITMETADATA']). Used for documentation and
lineage only — it is not validated against the view DDL (the view is
authoritative for what is computed). Default: None.
entity_id : str or list (recommended), or dict (advanced override)
Entity / primary-index key(s). Pass a string for a single-column key or a
list for a composite key, e.g. 'CUSTOMER_ID' or
['CUSTOMER_ID', 'EVENT_DATE'] — this is the recommended form. You do not
need to specify SQL data types: they are inferred from the source DataFrame
with their exact length and character set preserved (a CHAR(14) key stays
CHAR(14) in the created feature table, not truncated to CHAR(1)). Only pass a
dict (e.g. {'CUSTOMER_ID': 'BIGINT', 'EVENT_DATE': 'DATE'}) in the rare
case where you need to override an inferred type; the supplied type strings are
then used verbatim.
entity_null_substitute : dict, optional
Per-column replacement values for NULLs in entity keys (e.g.
{'CLI_CD': '-'}), applied via COALESCE. NULL values in an entity key column
that is NOT covered here will raise a ValueError during ingestion: a NULL key
produces a cartesian-product MERGE (severe skew / storage blow-up). Filter the
NULL rows upstream or provide a substitute. Default: {}.
optimize_query : bool, optional
If True and an instruct model is configured, run the query optimizer on the
registered process after ingestion. The full report is saved to a file named
tdfs4ds_optim_{timestamp}_{view}_{process_id}.md in the current working
directory. Only the score summary is emitted to the log.
Default: None = tdfs4ds.UPLOAD_OPTIMIZE_QUERY (False unless configured).
The AI steps (``optimize_query``, ``document``, ``explain``,
``cascade_redocumentation``, ``vector_indexing``) all default to None,
which reads ``tdfs4ds.UPLOAD_<NAME>`` (``TDFS4DS_UPLOAD_<NAME>`` in a .env).
``tdfs4ds.AGENTIC_FEATURES = False`` (``TDFS4DS_AGENTIC_FEATURES=false``)
turns all of them off at once. An explicit argument always wins.
update_view : bool, optional
If True and query optimization finds an improvement, replace the process view
DDL with the optimized SQL. Default: True.
document : bool, optional
If True and an instruct model is configured, run incremental documentation on
the registered process (and after any view update). If dataset_view_name is
also provided, document_dataset_incremental is called after run() so
that the dataset-level documentation can propagate the already-generated feature
descriptions bottom-up.
Default: None = tdfs4ds.UPLOAD_DOCUMENT (True unless configured).
explain : bool, optional
Part of document: also run the LLM analysis of the view's EXPLAIN plan
(scores, warnings, recommendations). False documents the process without it
and keeps any stored analysis. Ignored when document is False.
Default: None = tdfs4ds.UPLOAD_EXPLAIN (True unless configured).
max_candidates : int, optional
Maximum number of SQL rewrite candidates to generate during query optimization.
Overrides tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATES for this call only.
Default: None (uses the global value).
cascade_redocumentation : bool, optional
If True (and document=True with an instruct model configured), re-document
every downstream process/dataset that depends on this process — but only when
this process's documentation actually changed. The descendant set is read from
the persisted lineage graph (FS_LINEAGE_GRAPH). This propagates a changed
parent meaning down to everything that consumes it, at the cost of one extra LLM
pass per affected descendant; hence it is opt-in.
Default: None = tdfs4ds.UPLOAD_CASCADE_REDOCUMENTATION (False unless configured).
vector_indexing : bool, optional
If True (and document=True with an instruct model configured), rebuild the
vector index after documentation is written so the consumer agent's search
results reflect the new/updated business descriptions immediately. Uses an
additive upsert (force_rebuild=False) — fast but stale entries from
deleted/renamed objects persist until a full rebuild_vector_indexing() is
run. Non-fatal: a failure logs a warning without aborting the ingestion.
Default: None = tdfs4ds.UPLOAD_VECTOR_INDEXING (False unless configured).
grouped : bool, optional
Forwarded to build_dataset whenever a dataset view is built (i.e. when
dataset_view_name is set or tdfs4ds.BUILD_DATASET_AT_UPLOAD is True).
If False (default), uses one sub-query per feature joined together — the
classic multi-join approach; every feature sub-query can be targeted by a
PI-equality filter, keeping single-entity lookups single-AMP.
If True, groups features sharing the same storage table and process
version into one sub-query with MAX(CASE WHEN …) pivoting, reducing JOIN
count from N_features to N_distinct_(table, version) pairs — recommended for
wide datasets with many features from the same source table. See
build_dataset for the full explanation. Default: False (unchanged
behaviour). Not persisted in the process catalog — a later run(process_id)
without grouped=True reverts to the classic join for any dataset rebuild.
return_process_id : bool, optional
If True, return the PROCESS_ID (a UUID string) of the process registered by
this call instead of the dataset — so the caller can hold on to it for a later
run(process_id) / roll_out([process_id], ...) without having to look it
up in the process catalog. Mirrors run(return_dataset=...): the flag toggles
what is returned, it does not wrap it in a tuple. Note the default return
(dataset) is already None unless tdfs4ds.BUILD_DATASET_AT_UPLOAD is
True, so in the common case nothing is given up by setting this. The recoverable
input-error paths still return None regardless (check the log). Default:
False (unchanged behaviour — the dataset is returned).
run_now : bool, optional
If False, register the process without executing it — the process catalog row,
the data-distribution row and any filtermanager row are written and the view is
created, but run() is not called, so no feature values are computed. Use it
for a lazy "declare now, compute later" flow: the returned PROCESS_ID (pair with
return_process_id=True) can be executed at any later point via run(process_id)
or roll_out([process_id], time_manager), which drives every vintage including the
first. A dataset_view_name given with it is recorded as a process setting and built
by each later run() that passes none (so by roll_out, at each date; 0.3.5.14).
Caveat: register_entity / register_features and the FS_T_*
table creation happen inside run(), so until the first run the feature
catalog has no entry for these features and their storage tables do not exist —
get_feature_versions and friends return nothing for them. That is the intended
lazy state, not an error. Dataset documentation is also skipped (the dataset view
does not exist yet).
**Validation.** Because the run is what normally proves the source executes,
``run_now=False`` first validates it with an ``EXPLAIN`` — **before** any catalog
row is written, so an invalid source never leaves a registered process behind. On
failure nothing is registered: a prescriptive message is logged and ``None`` is
returned (no traceback), like the other recoverable paths. For a **UAF ART view**
only the **first** ``art_sql`` statement is EXPLAINed and the ART is **not**
materialised — later statements may read an ART an earlier one creates, so without
materialisation they would fail for a reason unrelated to their own validity; the
feature view is skipped for the same reason (it reads the ART → Error 3807).
**What EXPLAIN does and does not prove.** It proves the statement compiles and its
objects are reachable. It does **not** prove the process will ingest: ``run()`` also
performs type resolution (``Gettdtypes`` / ``validate_feature_types``) and rejects
NULL entity keys not covered by ``entity_null_substitute``, and the real ingestion
statement is an unpivot CTAS rather than ``SELECT *``. So a lazily-declared process
can EXPLAIN clean and still fail on its first run. Default: True (unchanged
behaviour — register **and** run).
storage_tag : str, optional
Store this process's features in a table of their own (the table of their entity and type with
__<TAG> appended, same primary index and partitioning), recorded in FS_FEATURE_PLACEMENT
before the first run. Two processes of one entity under different tags write different tables, so
a parallel run does not serialise them. A feature already registered in another table, or shared
with another process, is refused: relocate_features moves features with their history.
Default None: nothing changes.
track_entity_refresh : bool, optional
Process option (0.3.3.30), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out() of the process. The ingestion MERGE writes nothing when a value is
unchanged, so a feature row only tells when a value last changed. When True, each batch also
records, in an admin table of its own (FS_R_*, never a feature, exposed by no catalog or
view), that the process confirmed each entity it produced, with a valid-time history. Read it
with tdfs4ds.entity_refresh(process_id) and tdfs4ds.entity_refresh_summary(process_id)
(entities by last refresh, and those the latest run did not confirm). Cost: one
SELECT DISTINCT over the staged batch and one MERGE per batch. A failed write is a warning;
the ingestion stays COMPLETED. A batch that staged 0 rows records nothing. Default False: the
ingestion statements are unchanged.
dataset_time_manager : bool, optional
Setting (Part D), recorded in FS_PROCESS_SETTINGS and applied by every later run() /
roll_out(). When True and the process has a TimeManager (time_manager= here, else the
recorded one), the dataset of dataset_view_name is built with build_dataset(time_manager=...)
(grouped or not): it shows the business date of the manager, so tm.update(i) moves it without a
re-run. Without a TimeManager: a warning and the usual current dataset. A dataset that already
exists at CURRENT VALIDTIME (or in another mode) is not rebuilt silently: a warning names
tdfs4ds.rebuild_dataset(<name>, time_manager=..., replace=True). Default False: unchanged.
force_varchar_length : int, optional
Process setting (0.3.5.1), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out(). The VARCHAR features being registered take one type per character
set: the stored type of a registered text feature of the process that holds them, else
VARCHAR(ceil(longest / N) * N) over the new features. None (default): the value
recorded for this view's process, else 1024 (a recorded value is never reset by the default).
0 or 'none': exact lengths. A registered feature keeps its stored type; a source type
it does not hold refuses the registration with the alternatives (cast, or
tdfs4ds.plan_type_change).
valid_time_column : str, optional
Process setting (0.3.5.0), recorded in FS_PROCESS_SETTINGS and applied by every later
run() / roll_out(). A DATE or TIMESTAMP column of the source (neither an entity column
nor a feature) giving each row its own valid time: the row's values are valid from it until
the entity's next row (or 9999-01-01), and one run writes a whole history. The source may
then hold several rows per entity, one per valid time. Equal consecutive values are merged;
each entity's stored history is replaced from its earliest valid time in the load onward and
kept before it, so a re-run is a no-op and an incremental load rewrites only from its own
dates. FEATURE_STORE_TIME and the TimeManager date no longer set the values' valid time
(a TimeManager can still bound the rows the view returns). With close_absent_entities, an
entity absent from a load has no value from the load's earliest valid time on; with
track_entity_refresh, the load's latest valid time is recorded. Not for a UAF ART view.
A tdfs4ds older than 0.3.5.0 running the process ignores the setting. Default None: unchanged.
time_manager : TimeManager, optional
The TimeManager the process view reads. The pairing is recorded at registration
(FS_TIME_MANAGER, as roll_out records it), so run(process_id) later
needs no argument and computes at the manager's business date; with
run_now=True the first run already does. The view is checked first: a
manager it does not read is a warning under tdfs4ds.TIME_MANAGER_CHECK='warn'
(default; the pairing is not recorded) and refused under 'error' (nothing
registered, None returned). An explicit manager is paired as before 0.3.6.0,
even when it is the clock of the data domain (no soft clone).
None (default, 0.3.6.0): the view's lineage is walked and the manager it reads
is recorded: the clock of the data domain (set_domain_time_manager), which
then gets its own soft clone (see soft_clone), or the one manager it reads
(several: a warning, nothing recorded). tdfs4ds.TIME_MANAGER_DETECT = False
restores the pre-0.3.6.0 behaviour (nothing recorded until a roll_out).
soft_clone : bool, optional
(0.3.6.0) None (default): a process whose view reads the clock of its data
domain, found by detection, gets its own soft clone <view>_TM (same dates, own
cursor) and its view is rewritten to read it. True: a soft clone of any
manager, passed or found. False: never.
close_absent_entities : bool, optional
Process option (0.3.3.22), recorded in FS_PROCESS_OPTIONS and applied by
every later run() / roll_out() of the process. When True, each run
ends the validity of the stored entities the process no longer produces, from
the run's valid time on (the TimeManager date under roll_out) — the same
"from t onward" rule as the MERGE for the entities still produced. A re-run at
the same date is a no-op, and an entity that comes back gets a new period (the
absence stays a gap). Replay a history in date order, from its first date. A
batch that staged 0 rows closes nothing; a failed close records the batch FAILED. With a
filtermanager, every filter column must be an entity column (a segment
closes only its own entities), else nothing is registered and None is
returned. False (default): registration unchanged, and an option recorded
by an earlier registration of the same view is switched off.
tdfs4ds.set_process_options changes it without re-registering.
Returns¶
teradataml DataFrame or str or None
By default the dataset view (None unless tdfs4ds.BUILD_DATASET_AT_UPLOAD
is True); the PROCESS_ID string when return_process_id=True; None on
a recoverable input error, with a prescriptive message logged.
_upload_features(df, entity_id, feature_names, feature_versions=_FEATURE_VERSION_DEFAULT, primary_index=None, partitioning='', filtermanager=None, entity_null_substitute={}, process_id=None, force_compute=False, force_varchar_length=None, dataset_view_name=None, grouped=False, art_sql=None, on_failure=None, raise_on_failure=None, valid_start=None, close_absent=False, track_refresh=False)
Uploads a set of features into the Feature Store for a given entity.
This function registers an entity and its associated features in the feature catalog if they are not already defined, prepares the data for ingestion, and stores it in the feature store. It also supports incremental feature computation and conditional execution depending on prior runs.
Parameters¶
df : pandas.DataFrame
Input dataframe containing entity keys and feature columns to upload.
entity_id : str, list, or dict
Identifier(s) for the entity. List/str inputs have their SQL types inferred
from df with exact length + character set preserved (e.g. CHAR(14) keys are
not truncated); a dict is used verbatim.
feature_names : list of str
List of feature column names to upload from df.
feature_versions : dict or str, optional
Feature version(s). Default is _FEATURE_VERSION_DEFAULT ('dev.0.0').
primary_index : str or list, optional
Primary index for Teradata feature tables.
partitioning : str, optional
Partitioning clause. Default is ''.
filtermanager : FilterManager, optional
If provided, features are built iteratively per filter step.
entity_null_substitute : dict, optional
Replacement values for nulls in entity keys (applied via COALESCE). A NULL in
an entity key column not listed here raises a ValueError during ingestion
(NULL keys cause a cartesian-product MERGE).
process_id : str, optional
Process execution identifier for follow-up logging.
force_compute : bool, optional
If True, forces recomputation even if already completed. Default is False.
force_varchar_length : int, optional
If provided, all VARCHAR feature columns are resized to this length.
dataset_view_name : str, optional
Name of the output dataset view to create after ingestion.
grouped : bool, optional
Forwarded to build_dataset whenever a dataset view is built. See
upload_features for details. Default: False (classic multi-join).
art_sql : list of str, optional
UAF ART view mode. Forwarded to every _compute_features_batch call so the
volatile ART(s) are materialised per partition before ingestion. None for
the classic path. See upload_features / _materialise_art.
close_absent : bool, optional
Process option close_absent_entities: every batch (each FilterManager
segment) ends the validity of the stored entities it no longer produces.
track_refresh : bool, optional
Process option track_entity_refresh: every batch records, in the process's admin
FS_R_* table, that it confirmed each of its entities (see entity_refresh).
Returns¶
pandas.DataFrame or None If BUILD_DATASET_AT_UPLOAD is enabled, returns the built dataset. Otherwise None.
_compute_features_batch(df, entity_id, feature_names, selected_features, primary_index, partitioning, entity_null_substitute, process_id, filtermanager, dataset_view_name, dataset_already_created, grouped=False, art_sql=None, valid_start=None, close_absent=False, track_refresh=False)
Execute one feature ingestion batch: followup_open → [materialise ART] → prepare → store → followup_close (COMPLETED) → optional dataset build.
Returns (features_infos, dataset_or_None) on success.
On failure: logs the error, attempts to record FAILED in the followup table, then re-raises.
Callers that want to continue after a failure (e.g. the filtermanager loop) should wrap
this call in their own try/except and use continue rather than re-raising.
Does NOT call apply_collect_stats — the caller is responsible for that.
grouped is forwarded to build_dataset — see upload_features for details.
art_sql (UAF ART view mode only): a list of INTO VOLATILE ART statements
materialised before ingestion, in this same session — so the session-scoped
volatile ART is visible to the ingestion CTAS. Runs after filtermanager.update()
(done by the caller), so the series-input view's scalar subquery already points at
the current partition. None/empty for the classic path (a no-op).
valid_start: the run's pinned valid start for a real-time run (no
FEATURE_STORE_TIME), so every feature of the run shares one start. None
keeps the per-statement CURRENT VALIDTIME (see run).
close_absent (process option close_absent_entities, 0.3.3.22): after the
values are stored, end the validity of the entities of this process (and of the
active FilterManager segment) that this batch no longer produces, at the same
valid time. Skipped when the batch staged 0 rows; a failure records the batch FAILED.
track_refresh (process option track_entity_refresh, 0.3.3.30): after the values (and
the close of absent entities) are stored, record in the process's admin FS_R_* table
that this batch confirmed each of its entities. Skipped when the batch staged 0 / an
unknown number of rows. Fail-open: a failed write is a warning, the batch stays COMPLETED.
Upload features from a tdstone2 model's predictions to the feature store.
Handles the full workflow: extract feature names/types, register in the feature catalog, prepare for ingestion, store in the feature store, and build a dataset view.
Parameters: - model: The tdstone2 model object whose predictions contain features to upload.
Precondition — the model must expose one of two output accessors. This function
dispatches on whether the model has a score member: if it does, it reads
model.get_model_predictions(); if it does not, it reads
model.get_computed_features(). A model class exposing neither is not
supported by this bridge and will fail with an AttributeError. The entity is
likewise taken purely from the model's mapper (tdstone2_entity_id reads
mapper_scoring.id_row / mapper.id_partition / mapper.id_row) — tdfs4ds
injects no origin/as-of column, so a contract keyed on an origin date only gets one
if the model's own mapper already carries it. For a model that does not fit this
shape, register a permanent view over its output table with upload_features
instead — that path has no such requirement.
Returns: - DataFrame: A DataFrame representing the dataset view created in the feature store.
roll_out(process_list=None, time_manager=None, time_id_start=1, time_id_end=None, force_compute=False, force_display_logs=False, on_failure=None, raise_on_failure=None, check_upstream=False, strict_upstream=False, close_absent_entities=None, track_entity_refresh=None, dataset_time_manager=None, rebuild_history=False, *, data_domain=None, order='dependencies')
Execute a series of processes for each date in a time manager, managing time, computation settings, and logging.
Iterates over a range of time steps, updating the TimeManager with each step, and executes all processes in process_list for that time step.
Parameters:
- process_list (list): List of process IDs to execute for each time step.
- time_manager (TimeManager, optional): the manager whose dates are rolled out.
None (0.3.5.15) = the TimeManager attached to the processes; they must share one
table of dates (one manager, or a manager and its soft clones), else ValueError.
- time_id_start (int, optional): Starting time step ID. Default is 1.
- time_id_end (int, optional): Ending time step ID. If None, runs until last step.
- force_compute (bool, optional): Force recomputation even if results exist. Default False.
- force_display_logs (bool, optional): Force log display during rollout. Default False.
Each process is first checked against the lineage graph, before any step runs:
- its view reads the manager: fine;
- its view reads a soft clone of the manager (same table of dates, own cursor): at each step the clone's cursor is moved to the same step before the processes run, so the values stay consistent (0.3.5.15); its own pairing is kept;
- its view reads another manager (other dates):
ValueError, also underTIME_MANAGER_CHECK='warn'(0.3.5.15; it used to warn and write values stamped with this manager's dates but computed at the other one's cursor); -
its view reads no manager: a warning under
'warn'(default; the pairing is not recorded), aValueErrorunder'error'.'off'skips the check (and the alignment). -
close_absent_entities (bool, optional):
None(default) applies each process's recorded option;True/Falseoverride it for this roll-out. With it on, each date ends the validity of the entities a process no longer produces at that date; a later date where they come back opens a new period. Replay a history in date order (from its first date). - track_entity_refresh (bool, optional):
None(default) applies each process's recorded option;True/Falseoverride it for this roll-out. With it on, each date records which entities each process confirmed (tdfs4ds.entity_refresh). - rebuild_history (bool, optional): before step 1,
roll_outreads where each process's stored values start (0.3.5.16). A run writes from its date onward but never before the first value already stored, so when they start after the first date rolled out (a first run in real time, or at a later date), the roll-out would add nothing: it is refused withValueErrorand nothing runs.Truedeletes the stored values of those processes from the first date rolled out onward (logged inFS_REMOVAL_LOG) and replays every date withforce_compute=True; with nothing to delete it only forces the recomputation. A process whose valid time comes from a source column is not checked; a start that cannot be read is a warning and the roll-out goes on. - on_failure / raise_on_failure: what a failed FilterManager segment does in each run
('continue' default, 'stop', 'raise' = attempt every segment then raise
FeatureIngestionError;raise_on_failure=Trueis the shorthand). A raised failure stops the roll-out at that date. - check_upstream / strict_upstream: refuse a process when an upstream process (lineage graph) FAILED at that date / is also RUNNING.
- dataset_time_manager (bool, optional):
None(default) applies each process's recorded setting;True/Falseoverride it: the dataset a run builds follows the process's TimeManager. - data_domain (str or True, keyword, 0.3.6.0): roll out every process of a data domain
(
True=tdfs4ds.DATA_DOMAIN; primary store only), driven by the domain clock (set_domain_time_manager) whentime_manageris not given.process_list=Nonemeans the processes oftdfs4ds.DATA_DOMAIN. Passing both a list anddata_domainis aValueError. - order (keyword, 0.3.6.0):
'dependencies'(default) runs the processes of each date in dependency order, from the lineage graph; the sort is stable, so a list already in a valid order is unchanged.'given'keeps the list's order (the behaviour before 0.3.6.0).
Assess whether process_id's upstream processes are healthy.
Parameters¶
process_id : str
The process about to run.
feature_store_time : str or None
Valid time to assess. None means the current-validtime rows.
all_domains : bool
Read every data domain when building the process DAG (default).
strict : bool
Also treat an upstream stuck at RUNNING as blocking. Off by default:
a RUNNING upstream is legitimate during parallel execution.
Returns¶
dict
process_id, ok, verified, blocking, suspect,
n_upstream, lineage_status, view_name, feature_store_time,
explanation, error.
ok and verified are different questions and both matter: ok=False
means "a problem was found", while verified=False means "no problem was
found because nothing could be checked". Only ok=False blocks a run, but a
caller that treats verified=False as an all-clear is trusting a guard that
never ran.
Never raises: any failure yields ok=True with error set, because a
guard that cannot read its own inputs must not be what stops a run.
build_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, feature_store_time=False, join_type='INNER', grouped=False, document=False, vector_indexing=False, record_lineage=True, as_of=None, time_manager=None, feature_offsets=None, offset_strategy=None)
Generate a SQL query and create a Teradata view to retrieve records for the specified entity IDs based on selected features.
Parameters:¶
entity_id : list, dict, or str The entity IDs for which records are needed. Can be a list, dictionary, or string. - If list: contains multiple entity IDs. - If dict: keys are the entity IDs. - If str: a single entity ID.
list, str or dict
The feature names alone (['amount', 'nb_tx']) when each feature is
computed by one process: its version is found in the catalog. A version is
needed only for a feature several processes compute; write it in a dict,
with None for the others ({'amount': '<UUID>', 'nb_tx': None}).
Such a feature named without a version raises FeatureSelectionError
(a ValueError) naming its processes and the dict to write; so does a
name not registered for this entity. All features are resolved with one
catalog query. In the dict, version is:
- None: the feature's only version,
- {"alias": <column name>}: the only version under a name of its own,
- a process UUID string (the column takes the feature's name),
- {"process_id": <UUID>, "alias": <column name>}: one version under a name of its own, or
- a list of {"process_id": <UUID>, "process_view_name": <str>, "alias": <str, optional>}:
several versions side by side, one column each, named <feature>_<process view>
unless an alias is given.
An alias is a plain identifier (letters, digits, _) and may not repeat another column;
offsets (feature_offsets) are suffixed to it (spend_LAG_1D).
str
The name of the view to be created in the database.
str, optional
The name of the database schema where the view will be created.
Defaults to the global tdfs4ds.SCHEMA if not provided.
bool, optional
If True, the function will return the SQL query string instead of executing it. Defaults to False.
bool, optional
If True, the query will use a specific VALIDTIME based on tdfs4ds.FEATURE_STORE_TIME.
If False, the query will use CURRENT VALIDTIME. Defaults to False.
str, datetime, date or pandas.Timestamp, optional
Freeze the dataset at this point in time (VALIDTIME AS OF TIMESTAMP).
Any ISO date or timestamp is accepted — '2024-03-31',
'2024-03-31 23:59:59', '2024-03-31T23:59:59+02:00' — and
normalised to the literal Teradata needs (a bare date becomes midnight;
Teradata rejects a date-only TIMESTAMP literal). Without an offset the
value is read in the session time zone, like FEATURE_STORE_TIME.
TimeManager or str, optional
Make the dataset follow a TimeManager: every feature is read as of
the manager's current BUSINESS_DATE, so time_manager.update(i)
moves the dataset without rebuilding it. A 'database.view' string
naming the manager's public view is accepted too.
as_of, time_manager and feature_store_time=True are mutually
exclusive. With none of them the dataset reads CURRENT VALIDTIME (the
default, unchanged). The chosen mode is recorded in the dataset catalog:
DATASET_VALIDTIME holds the qualifier and the build table
(FS_V_<catalog>_BUILD) holds the mode, timestamp or TimeManager.
str, optional
Specifies the type of join to use between features: 'INNER' (default) or 'FULL OUTER'.
bool, optional
If False (default), uses one sub-query per feature joined together — the classic multi-join approach. Every individual feature sub-query can be targeted by a PI-equality filter pushed from the outer WHERE, making single-entity lookups single-AMP. If True, groups features that share the same storage table and process version into a single sub-query with MAX(CASE WHEN …) pivoting, reducing the number of JOINs from N_features to N_distinct_(table, version) pairs. A HAVING clause enforces non-NULL for every feature in the group when join_type='INNER', preserving the same missing-feature semantics as the classic approach. PI push-down is preserved for INNER JOIN (outer SELECT uses A1.entity); for FULL OUTER the COALESCE in the outer SELECT prevents push-down, same as the classic approach.
bool, optional
If True, generate and persist business-dictionary documentation for the
newly built dataset via document_dataset_incremental (LLM-powered).
Requires tdfs4ds.INSTRUCT_MODEL_PROVIDER and tdfs4ds.INSTRUCT_MODEL_MODEL
to be configured; silently skipped (with an info log) otherwise. Failures
are non-fatal — the dataset view is always created/returned regardless.
Defaults to False (opt-in). Mirrors upload_features. Leave False when
the dataset is built via upload_features/run() — that path already
documents it, so enabling it here would double-document.
bool, optional
If True (and document=True and an instruct model is configured), refresh
the consumer-agent vector index via build_vector_index(force_rebuild=False)
after documentation so the new dataset is searchable immediately. Non-fatal.
Defaults to False (opt-in).
bool, optional
Persist the dataset view's lineage subgraph (dataset -> process views ->
sources) into FS_LINEAGE_GRAPH via record_process_lineage. No LLM is
involved and the write is diff-based (an unchanged graph writes nothing).
Non-fatal. Defaults to True since 0.3.1.220: with the former opt-in
default no dataset reached the lineage graph unless
rebuild_lineage_graph() was run afterwards. Pass False to skip it.
dict, optional
Read features at other instants than the dataset's own (lags and leads),
e.g. {'amount': ['0', '-1 day', '-2 days'], 'churn': ['+1 month']}:
'-1 day'is the value one day BEFORE the dataset's instant (a lag, for autoregressive inputs),'+1 month'one month AFTER (a lead, e.g. a forecast target),'0'the instant itself.- A listed feature gets one column per offset (
amount,amount_LAG_1D,amount_LAG_2D,churn_LEAD_1MO); it keeps its plain column only when'0'is listed. - The instant is the dataset's: now,
as_of, or the TimeManager's business date (time_manager.update()moves every offset column). - A lead is NULL while its instant is later than the database's current
time: what is valid after now is not known yet. Under
CURRENTa lead is therefore always NULL (refused withjoin_type='INNER'). - Units: year, month, week, day, hour, minute, second; months are
calendar months (
ADD_MONTHS). No fraction of a second: valid times are stored to the second.
Without it the SQL is exactly what it was. See the user guide, "Lags and leads".
{'PIVOT', 'PER_OFFSET'}, optional
How the offset columns are read. 'PIVOT' (default) reads each
storage table (and process version) once for all its offset columns:
one aggregation step, local to the AMPs, partition elimination on the
FEATURE_IDs. 'PER_OFFSET' gives each offset column its own sub-query
and join; measured on Vantage 20 it parses far slower (32 columns: 31 s
against 0.35 s) and is kept for comparison.
Returns:¶
str or tdml.DataFrame
- If return_query is True, returns the generated SQL query as a string.
- Otherwise, returns a Teradata DataFrame containing the results.
Raises:¶
ValueError
If an invalid join_type is provided.
build_dataset_opt(entity_id, selected_features, view_name=None, schema_name=None, comment='dataset', no_temporal=False, time_manager=None, query_only=False, entity_null_substitute={}, other=None, time_column=None, filtermanager=None, filter_conditions=None)
Experimental: a dataset view built by augment_source_with_features (one
LEFT JOIN per feature onto the entity list), named view_name in schema_name
(default the view database). Only entity_id, selected_features, view_name
and schema_name are used; the other arguments are accepted for the signature of
build_dataset and ignored. Prefer build_dataset.
augment_source_with_features(source_schema, source_name, entity_id, feature_selection, output_view_schema, output_view_name, join_type='LEFT JOIN')
Augment a source table with selected features by creating a view that joins the source data with feature tables based on specified entity identifiers and feature selections.
The join type can be specified as either 'LEFT JOIN' or 'INNER JOIN': - 'LEFT JOIN': Retains all rows from the source table, with missing values for unmatched features. - 'INNER JOIN': Returns only rows where all specified features have matching, non-missing values.
Parameters: source_schema (str): The schema of the source table to augment. source_name (str): The name of the source table to augment. entity_id (str or list of str): The entity ID(s) for joining the source data with feature tables. feature_selection (list): A list of features to select and join with the source data. output_view_schema (str): The schema in which to create the output view. output_view_name (str): The name of the output view to create with augmented features. join_type (str, optional): 'LEFT JOIN' or 'INNER JOIN'. Defaults to 'LEFT JOIN'.
Returns: tdml.DataFrame: A DataFrame object that references the newly created view.
rebuild_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, grouped=None, join_type=None, as_of=None, time_manager=None, current=False, replace=False, comment=None, return_query=False, start=None, end=None, duration=None, delay=None, history=False)
Rebuild an existing dataset with another strategy or at another point in time.
The source view's own DDL is the specification: the same storage tables,
feature ids, process versions and column names are kept. Nothing is
re-resolved against the feature catalog, so the result does not depend on
the session's DATA_DOMAIN nor on which versions are registered today.
Parameters¶
view_name, schema_name :
The source dataset (view_name may be 'database.view').
target_view_name, target_schema_name :
Where to write the result. Default: the source itself, which requires
replace=True.
grouped : bool, optional
True → one pivoted sub-query per (storage table, version);
False → one sub-query per feature. None keeps the source's.
join_type : {'INNER', 'FULL OUTER'}, optional
None keeps the source's.
as_of, time_manager, current :
New temporal mode — a literal point in time, a TimeManager, or
current=True for CURRENT VALIDTIME. None of them keeps the
source's mode. Given on a historical dataset, they turn it back into a
snapshot.
start, end, duration, delay, history :
Rebuild as a HISTORICAL dataset over that window (see
build_historical_dataset); history=True alone means the full
history. On a historical source they replace its window.
replace : bool
Allow overwriting a view that already exists. Without it the call
refuses (REPLACE VIEW would silently overwrite it).
return_query : bool
Return the DDL instead of executing it.
Returns the new dataset as a teradataml DataFrame (or the DDL), None on
a recoverable error (logged with the fix).
build_grouped_dataset(view_name, schema_name=None, target_view_name=None, target_schema_name=None, replace=False, comment=None, return_query=False)
Build the grouped version of an existing (non-grouped) dataset.
The grouped strategy reads each storage table once per process version —
features sharing a table are pivoted with MAX(CASE WHEN …) — instead of
once per feature, so a wide dataset needs far fewer joins. Same rows, same
columns, same temporal mode as the source.
target_view_name defaults to <view_name>_GROUPED (a new dataset, the
source is left untouched). Pass the source's own name with replace=True
to convert it in place.
build_historical_dataset(entity_id, selected_features, view_name, schema_name=None, comment=None, return_query=False, join_type='INNER', start=None, end=None, duration=None, delay=None, record_lineage=True, document=False, vector_indexing=False, process_version=False, feature_offsets=None)
Create a HISTORICAL dataset view: one row per entity per period of constant values.
Where build_dataset returns a snapshot (each entity's feature values at
one valid instant), this returns their history over a window::
<entity columns>, ValidPeriod, ValidStart, ValidEnd, <feature columns>
Each row says: from ValidStart (included) to ValidEnd (excluded),
this entity had exactly these feature values. Consecutive rows of an
entity always differ in at least one value (NORMALIZE ON MEETS OR
OVERLAPS merges equal neighbours, should the stored history hold any). ValidPeriod is the same interval as a Teradata
PERIOD(TIMESTAMP(0) WITH TIME ZONE), for CONTAINS / OVERLAPS.
The result is a plain view: nothing is materialised, new ingestions show up immediately, and a relative window moves by itself.
Parameters¶
entity_id, selected_features, view_name, schema_name, comment, return_query :
As in build_dataset (feature names alone when each has one version).
join_type : {'INNER', 'FULL OUTER'}
'INNER' keeps a period only where EVERY feature has a value — an
entity's history starts when its last feature appears, and an entity
whose features never overlap is absent. 'FULL OUTER' keeps every
period where AT LEAST ONE feature has a value, with NULLs elsewhere.
Either way, at any instant t of the window the history equals
build_dataset(..., as_of=t, join_type=...).
Example — feature A from January (10, then 20 from July), feature B
only from April::
FULL OUTER: [Jan, Apr) A=10 B=NULL | [Apr, Jul) 10 x | [Jul, ...) 20 x
INNER: [Apr, Jul) 10 x | [Jul, ...) 20 x
start, end : str, date, datetime or pandas.Timestamp, optional
Fixed window bounds (ISO strings accepted, a bare date is midnight).
The window is half-open [start, end): end='2024-04-01' covers
all of March. Without end the window ends at now.
duration : str, tuple or timedelta, optional
Window length: '3 months', '90 days', '12 hours',
'1 year 2 months', ISO 8601 'P3M', (3, 'months') or a
timedelta. Combined with start → [start, start + duration);
with end → [end - duration, end); alone → [now - duration, now).
Months use ADD_MONTHS (31 May minus 3 months is 29 Feb).
delay : same forms as duration, optional
Shift now back: duration='12 months', delay='1 day' is
[now - 1 day - 12 months, now - 1 day). Not combinable with
start/end.
``now`` is the current valid time when the view is QUERIED, so any
window without a fixed end slides with time. No argument at all means
the full history up to now. A period still open in the store therefore
ends at the window's end, never at 9999-12-31.
record_lineage, document, vector_indexing :
As in build_dataset.
process_version : bool, default False
Also create <view_name>_PV: the same history with a
<feature>_PROCESS_VERSION column per feature (the version of the
process SQL that wrote each period, NULL before versioning). The dataset
view itself is never changed. Whatever this flag, a history that spans
several process versions is reported in the log
(tdfs4ds.history_versions).
feature_offsets : dict, optional
Lags and leads, as in build_dataset: each instant of the history
reads amount_LAG_1D one day earlier, churn_LEAD_1MO one month
later (NULL until that date is reached). A history with offsets is
computed with elementary intervals for both join types (a shifted period
cannot go through the native sequenced join). Not combinable with
process_version=True yet.
Returns¶
tdml.DataFrame on the new view, the DDL when return_query=True, or
None when no feature could be resolved.
Notes¶
INNER uses Teradata's native SEQUENCED VALIDTIME join (each storage
table read once, rowkey merge joins, the window folded into the join).
FULL OUTER cuts each entity's timeline at every change point and attaches
each feature on containment; it reads each storage table more than once.
See tdfs4ds.dataset.history.
The DatasetSpec of an existing dataset view, read from its own DDL.
view_name may be 'database.view'. Returns None (with an error
logged) when the view is missing or is not a tdfs4ds dataset view.
restore_dataset_catalog(databases=None, discover=True, dry_run=True, drop_orphans=False, display=True, datasets=None)
List every dataset and restore the dataset catalog from the views themselves.
Parameters¶
datasets : list of str, optional
Restrict the check (and any repair) to these datasets, given as
'view' or 'database.view'. Default: every dataset found.
databases : list of str, optional
Databases to scan for dataset views. Default: the view database
(tdfs4ds.view_schema()), the table database (tdfs4ds.SCHEMA) and
every database the catalog already references.
discover : bool
Also look for dataset views the catalog does not know (DBC.TablesV
scan) and offer to register them. False checks catalogued datasets
only.
dry_run : bool
True (default) changes nothing and reports what would be done.
False applies it.
drop_orphans : bool
When applying, also close the catalog rows of datasets whose view no
longer exists (CURRENT VALIDTIME DELETE — history kept). Off by
default: a missing view may be a permission problem, not a deletion.
Returns¶
pandas.DataFrame
One row per dataset: STATUS is one of OK, REPAIR (catalog
disagrees with the view), REGISTER (view not catalogued),
DUPLICATE (registered several times), ORPHAN (view gone) or
UNPARSEABLE (catalogued view that is not a build_dataset view — left
alone); ISSUES says what disagrees, ACTION what was (or would
be) done. STRATEGY/JOIN_TYPE/TIME_MODE describe the view.
check_storage_statistics(domains=None, processes=None, tables=None, add_missing=False, refresh=False, display=True)
Review the statistics of the feature storage tables; optionally complete and refresh them.
Parameters¶
domains : list of str, optional
Restrict to the storage tables of these data domains.
processes : list of str, optional
Restrict to the tables these processes (PROCESS_IDs) write.
tables : list of str, optional
Restrict to these tables ('database.FS_T_...').
add_missing : bool
Define the missing statistics (collecting them immediately), with the
options ingestion uses. Later merges then keep them current.
refresh : bool
Re-collect every statistic the table carries (COLLECT STATS ON),
the call ingestion makes after each merge. Multi-column statistics keep
their THRESHOLD 15 PERCENT: Teradata skips them until the table has
changed by 15%, which is why their collection date can be older.
display : bool
Print the report.
With both flags off (the default) nothing is executed: the call only reads the data dictionary.
Returns¶
pandas.DataFrame
One row per table: STATUS is OK, MISSING (some expected
statistics are not defined), NO_STATS, or TABLE_MISSING (the
feature catalog references a table that no longer exists — never acted on); MISSING / EXTRA
list column sets; OLDEST_COLLECTION / NEWEST_COLLECTION bound
the collection dates; ACTION says what was done.
followup_resource_usage(process_id=None, run_id=None, days=30, flush_minutes=DEFAULT_FLUSH_MINUTES, only_logged=False)
Resources used by each process execution: follow-up rows joined to DBQL.
Needs SELECT on DBC.QryLogV and DBQL logging for the user that ran
the processes (tdfs4ds.check_configuration() checks both).
DBQL is not immediate. Logged queries sit in a cache and are written to
the DBQL tables at the flush interval (DBQLFlushRate, 10 minutes by
default), so an execution that just ended has no DBQL row yet. Every
follow-up row is returned with DBQL_STATUS — LOGGED, PARTIAL
(totals may still grow), PENDING (nothing written yet: ask again
later) or NOT LOGGED — so a missing or incomplete measure is never
silent. Set
flush_minutes to the system's flush rate; a DBA can also force the
write with FLUSH QUERY LOGGING WITH ALL;.
Returns a pandas DataFrame.
followup_resource_usage_sql(process_id=None, run_id=None, days=None, flush_minutes=DEFAULT_FLUSH_MINUTES, only_logged=False, process_ids=None) -> str
The SELECT joining FS_V_FOLLOW_UP to DBC.QryLogV per execution.
One row per follow-up execution: all its follow-up columns, then the resource usage of its queries, summed over them:
N_QUERIES,N_ERRORS;- CPU —
AMP_CPU_SEC(total),MAX_AMP_CPU_SEC(busiest AMP of the heaviest query),IMPACT_CPU_SEC(busiest AMP × active AMPs: what the system effectively spent),CPU_SKEW_PCT=1 − AMP_CPU / IMPACT_CPU(0 = even),PARSER_CPU_SEC; - I/O —
TOTAL_IO(logical),PHYSICAL_IO,IO_SKEW_PCT(same rule onMaxAmpIO); - spool —
MAX_SPOOL_BYTES(peak of any one query); FIRST_QUERY_START,LAST_RESPONSE(DBQL times, system clock);
and DBQL_STATUS:
LOGGED— its DBQL rows were found, and it ended long enough ago for all of them to be written;PARTIAL— rows were found, but the execution is still running or ended less thanflush_minutesago: the totals may still grow;PENDING— nothing found yet, and its last activity (end, or start when it never ended) is less thanflush_minutesold: DBQL writes from a cache, flushed every 10 minutes by default (DBQLFlushRate);NOT LOGGED— older than that and still absent: logging was off for the user, the rule excluded these queries, or the run was not tagged. A row leftRUNNINGby a crash lands here once its start is old enough, since its first query (the follow-up insert) would have been flushed.
days bounds both the follow-up rows and the DBQL scan;
only_logged=True keeps the LOGGED rows only; process_ids (a
list) restricts to several processes.
Create the join as a view in the view database (no days bound).
A view over DBC.QryLogV works only when its database holds SELECT
... WITH GRANT OPTION on it — usually a DBA grant. Returns the view name,
or raises with the database message.
Copy the executions whose DBQL rows are complete into the history tables.
One INSERT ... SELECT per table, run in the database (no row comes to
the client); only executions not stored yet. days bounds the DBQL scan
(default: from two days before the last stored execution). dry_run=True
returns the SQL without running it. Never raises: returns {"ok",
"executions", "statements", "error", "sql"}.
Executions in the window (as_of - days, as_of] with their resource use.
Returns (frame, info): one row per execution (stored history first, then
the live follow-up <-> DBQL join for executions not stored yet), with
ELAPSED_SEC, PROCESS_VERSION and DBQL_STATUS (STORED /
LOGGED / PARTIAL / PENDING / NOT LOGGED / UNAVAILABLE).
info: {"state", "error", "hint", "history", "live"}. Never raises on
a DBQL failure: the follow-up alone is returned with the reason.
The statements of one execution: (frame, info).
DBQL first (it has the SQL text, from DBC.QryLogSQLV when readable),
else the stored copy (no SQL text). Never raises.
Per process (and per process version when by_version): executions, runs, failures, how often it runs, last run, run duration and consumption.
frame: :func:read_execution_resources' rows. A run groups the segments
of one execution (:func:~tdfs4ds.process_store.process_followup.execution_run_ids),
so frequency and per-run figures do not count FilterManager segments as
runs. Returns {pid: stats} or {pid: {version|None: stats}}. Pure.
Normalise value to the body of a Teradata TIMESTAMP '...' literal.
Returns 'YYYY-MM-DD HH:MI:SS[.ffffff][+HH:MM]'. A time part is always
present: Teradata rejects TIMESTAMP '2024-03-31' (Error 3706, verified
live), so a bare date becomes midnight. Fractional seconds and a time-zone
offset are kept when given (both verified accepted in VALIDTIME AS OF);
without an offset the literal is read in the session time zone, exactly as
tdfs4ds.FEATURE_STORE_TIME is when features are ingested.
Accepted inputs: ISO strings ('2024-03-31', '2024-03-31 23:59',
'2024-03-31T23:59:59.5Z', '2024-03-31 23:59:59+0200'),
datetime.datetime, datetime.date, pandas.Timestamp and
numpy.datetime64. Anything else — including ambiguous forms such as
'03/31/2024' — raises ValueError naming the accepted formats, rather
than being guessed.
retrieve_batch(dataset, list_entity_keys, entity_cols=None, *, columns=None, method='auto', volatile_threshold=DEFAULT_VOLATILE_THRESHOLD, conn=None, dedup=True, pushdown='auto')
Retrieve rows of dataset for a list of entity-key tuples.
Parameters¶
dataset : str
Fully-qualified DB.dataset_view to read from.
list_entity_keys : list[tuple]
Entity-key tuples in the order of entity_cols (each tuple one row's key).
entity_cols : list[str] | None
Grain/entity key columns. When None they (and their SQL types) are
auto-derived from the dataset catalog (needs an active teradataml context).
columns : list[str] | None
Columns to project; None selects all.
method : {'auto','in_list','volatile_join'}
'in_list' builds a WHERE predicate (single-col IN / composite
OR-of-ANDs); 'volatile_join' loads the keys into a volatile
table co-located on the feature-store primary index and joins; 'auto'
(default) uses the volatile join when the unique-key count reaches
volatile_threshold OR when the in_list predicate would exceed the
parser limit, else the in_list predicate.
volatile_threshold : int
Key count at/above which 'auto' switches to the volatile join.
conn : teradatasql connection | None
Optional raw connection (benchmark reuse). None uses the teradataml
context.
dedup : bool
Drop duplicate key tuples first (required for the volatile join; also
shrinks the in_list predicate).
pushdown : {'auto', True, False}
'auto' (default) regenerates the dataset's query from its DDL with
the keys inside every storage-table read, so each FS_T_* table is
read by primary index and only in the partitions the keys fall in -
filtering the view from outside reaches neither, and under FULL OUTER
nothing can be pushed through the view at all. Falls back to the view
for a historical dataset or a view the parser does not understand.
False keeps the previous SQL exactly. Same rows either way.
Returns¶
list[tuple] The fetched rows.
Resolve the REAL primary index / partitioning of a dataset's feature tables.
Returns a dict::
{"primary_index": [...], "partitioning": "...",
"source": "DBC.IndicesV" | "fallback_entity_cols",
"feature_tables": ["db.FS_T_…", …]}
The volatile key table should use primary_index so the join co-locates
with the feature storage tables. Fail-open: returns entity_cols when the
real distribution can't be read (restricted DBC, view-only, no match).
Return the dataset grain as [(column_name, sql_type), …].
Reads ENTITY (name) + ENTITY_TYPE (type) from the dataset catalog's
entity table. Requires an active teradataml context. Returns [] when the
dataset can't be resolved.
run_benchmark(dataset, entity_cols=None, *, sample_size=10000, skip=(), batch_sizes=(1, 5, 10, 50, 100, 500), large_sizes=(10000, 100000, 1000000), strategy_sizes=(500, 5000, 50000), concurrency=(1, 2, 4), time_travel_dates=(), max_in_list_keys=None, max_parse_ms=DEFAULT_MAX_PARSE_MS, batch_pushdown=True, conn_kwargs=None, progress=None)
Run the benchmark against dataset and return the report dict.
Parameters¶
dataset : str
Fully-qualified DB.dataset_view to benchmark.
entity_cols : list[str] | None
Grain/entity key columns. When None, auto-derived from the tdfs4ds
dataset catalog (needs an active teradataml context).
skip : set[str]
Any of {'parse','explain','warm','cold','batch','large_batch',
'batch_strategies','concurrent','time_travel','server_timing'} to skip.
max_parse_ms : float | None
The parse phase runs FIRST, before anything executes the view: it
times EXPLAIN SELECT * FROM <dataset> (parse + optimise, no
execution). When its median exceeds max_parse_ms the run stops
there — every request on such a view pays that cost before reading a
row, so the latency phases would mostly measure parsing and could take
very long. The report then explains why and what to try (the grouped
strategy). Default 5000 ms; None or 0 never stops.
strategy_sizes : tuple[int]
Batch sizes at which the batch_strategies phase compares the
in_list predicate vs the volatile_join retrieval side-by-side.
max_in_list_keys : int | None
Maximum number of keys put into an in_list predicate. When None
it is inferred from the entity-column count as
max(50, 500 // n_entity_cols) — wide composite keys produce large
per-key disjuncts whose parse time dominates the measurement, so the
cap keeps in_list batches cheap. It clamps batch_sizes and the
in_list arm of batch_strategies ONLY; the volatile_join arm
and large_batch keep their full sizes.
batch_pushdown : bool
True (default): the batch phases run what retrieve_batch runs -
the dataset's query regenerated with the keys inside every storage-table
read, so each FS_T_* table is read by primary index and only in the
partitions the keys fall in. False: the previous
SELECT * FROM <view> WHERE <keys> form, to measure the difference.
conn_kwargs : dict | None
Raw teradatasql kwargs; defaults to the TDFS4DS_TD_* environment.
progress : callable(str) | None
Receives one human-readable line per test phase (for a UI bar).
Returns the {"metadata":…, "results":…} report dict.
The HTML report of a benchmark (run_benchmark's result): a self-contained page with
inline SVG charts and CSS, no external JavaScript. Every section tolerates a phase that
is missing or failed. Use write_benchmark_report to save it to a file.
Render report to HTML and write it to disk; return the file path.
When out_path is None, writes under {tdfs4ds.BENCHMARK_REPORT_DIR}/ with
a timestamped, dataset-derived filename (directory created on demand).
Compute a parallel-execution plan for feature-engineering processes.
Parameters¶
seeds : list[str] | None
Mixed list of process (id or view name) / dataset / feature names.
The plan is the upstream closure of the seeds' producing processes
(everything needed to (re)build them). None = the whole store.
all_domains : bool
Read every data domain (default). False restricts to the active
tdfs4ds.DATA_DOMAIN.
time_managers : ignored placeholder
Reserved for supplying explicit TimeManagers to the content-equivalence
check when the catalog has no TIME_* association yet.
Returns¶
dict
seeds, resolved_seeds, unresolved_seeds, closure,
processes (per-pid metadata incl. filter/time manager + wave index
+ lineage_unknown), waves (ordered list of parallel batches),
wave_groups (per-wave grouping by (filter, time) manager),
serialize_together (write-conflict groups), unify_candidates
(equivalent-but-distinct managers), edges (process dependency
edges within the closure), cycle (None or a cycle), plus counts.
generate_execution_script(plan: Dict[str, Any], *, out_path: Optional[str] = None, mode: str = 'process_pool', force_compute: bool = False, max_workers: Optional[int] = None, wave_on_failure: str = 'skip_descendants') -> str
Turn an :func:analyze_parallel_plan result into a runnable .py script.
Parameters¶
plan : dict
The plan returned by :func:tdfs4ds.analyze_parallel_plan.
out_path : str | None
If given, also write the script there (see also
:func:write_execution_script).
mode : {'process_pool', 'sequential'}
'process_pool' (default) parallelises each wave across processes;
'sequential' runs everything in order (debugging).
force_compute : bool
Passed to run in the generated worker.
Processes of a wave that share a TimeManager form one time group, run in
lockstep: the script moves the manager's cursor one date at a time, the group's
processes compute that date in parallel with run(pid, time_manager=tm) (which
reads the cursor and never moves it), and the next date starts when they are all
done. A per-process roll_out would move one shared cursor from several
workers at once. Give processes their own cursor with
:func:tdfs4ds.split_time_manager when they must run on different dates.
max_workers : int | None
Cap on parallel workers per wave. None (default) falls back to the
tdfs4ds.SCHEDULING_MAX_WORKERS global; when that is also None the
script runs unbounded (one worker per parallel unit). An explicit int here
overrides the global for this call.
wave_on_failure : {'skip_descendants', 'continue', 'stop_after_wave', 'raise'}
What the generated script does when a process fails. Named apart from the
per-process on_failure on run/roll_out deliberately: the two
take different value sets, and 'raise' is legal in both while meaning
something different, so one name for both would make an invalid value read
as a documentation error.
'skip_descendants' (default) skips only the failed process's
downstream closure and lets independent branches finish - a failure
poisons its own subtree, not the plan. 'raise' reproduces the
historical abort-everything behaviour.
Note the generated worker always calls ``run``/``roll_out`` with
``on_failure='raise'``: a wave policy can only react to failures it is
told about, and the per-filter loop does not report them otherwise.
Returns¶
str The script source.
Generate the script and write it to disk, returning the path.
out_path defaults to {tdfs4ds.SCHEDULING_SCRIPT_DIR}/tdfs4ds_execution_<ts>.py.
Extra keyword args are forwarded to :func:generate_execution_script.
Repoint process_ids onto a single canonical FilterManager.
Parameters¶
process_ids : list[str]
Processes to unify (typically one unify_candidates['filter'] group's
processes from :func:analyze_parallel_plan).
canonical : dict | None
{'database','view','table'} of the manager to keep. When None the
first process that already has a filter manager is chosen.
dry_run : bool
When True (default) returns the SQL that would run without executing.
Repoint process_ids onto a single canonical TimeManager (see unify_filter_managers):
each process view is rewritten to read canonical (default: the first process's manager)
and its FS_TIME_MANAGER row updated. dry_run=True (default) returns the statements
without executing them.
split_time_manager(process_ids, table_name: str, schema_name: Optional[str] = None, clone_mode: str = 'soft', dry_run: bool = True) -> Dict[str, Any]
Give process_ids their own TimeManager cursor (0.3.1.240) - the inverse of
:func:unify_time_managers.
The processes must share one TimeManager today. A clone named table_name is
made with TimeManager.clone_timer: clone_mode='soft' (default) keeps the
same dates (the clone reads the source's hidden table) with its own cursor;
'hard' copies the dates into a table of its own. Each process view is then
rewritten onto the clone and the pairing is recorded in FS_TIME_MANAGER. The
processes can then run at another business date than the ones left on the
source manager, for example in parallel.
A process that reads the manager only through an intermediate view cannot be
rewritten here (its own view does not name the manager); it is reported as
not_direct and left alone.
dry_run=True (default) creates nothing and returns the statements. schema_name:
the database of the clone (default the source manager's). With a domain clock
(0.3.6.0), upload_features makes the clones by itself and
tdfs4ds.align_domain_time_managers adopts processes registered before it.
time_manager_dependency(process_id: str, time_manager=None, *, graph=None, proc_df=None, registry_df=None, view_name: Optional[str] = None) -> Dict[str, Any]
What a process reads and what it declares.
Returns {"process_id", "view", "status" (LINKED/NOT_LINKED/UNKNOWN),
"reads": [db.view, ...], "declared": db.view or None, "passed": db.view or None,
"verdict" (OK/UNRELATED/MISMATCH/UNKNOWN, only with *time_manager*)}; with
time_manager also same_calendar (soft clones of it the process reads) and
other_managers. Never raises. graph / proc_df / registry_df: an already loaded
lineage graph / process catalog / manager registry (read when omitted); view_name:
the process view when the process is not registered yet.
Declare (or correct) the TimeManager of processes from what their views read.
For each process (all registered ones by default) the lineage graph says which
TimeManager the view reads, deep. Returns a pandas DataFrame with one row per
process: PROCESS_ID, VIEW_NAME, DECLARED, READS, ACTION
(RECORD / CORRECT / OK / AMBIGUOUS / NONE / UNKNOWN) and
APPLIED. dry_run=True (default) writes nothing; dry_run=False records
RECORD and CORRECT rows in FS_TIME_MANAGER. A process reading two
managers is left alone (AMBIGUOUS). process_ids: the processes to check (default
all); graph / proc_df: an already loaded lineage graph / process catalog.
Backfill the registry with the managers the store already uses.
Managers come from the process catalog (TIME_* / FILTER_*). For each, the
lineage graph tells which hidden table its view reads: another manager's table
is a soft clone of that manager; a table of its own is the manager itself.
Hard clones (a copy of another manager's table) are not detectable from the
lineage alone and are left as plain managers; the Report compares contents.
Returns a pandas DataFrame of the rows (ACTION = INSERT / OK);
dry_run=False writes them.
deploy_to_agentops(dest=None, *, database=None, databases_readonly=None, database_view=None, data_domain=None, token=None, base_url=None, auth='service', mcp=False, name=None, description=None, dry_run=False, timeout=1800, require_published=True, agent_runtime='auto', verify=True, startup_timeout=300, progress=None, pod_host='auto')
Package, deploy and verify the consumer agent on AgentOps.
Parameters¶
dest : str, optional
The scaffolded agent directory holding the platform's agent_runtime/.
Defaults to tdfs4ds.AGENTOPS_PACKAGE_DIR.
database : str
Required. The feature store the deployed agent serves. Stated rather
than inherited, so the pod's store is visible at the call site.
databases_readonly : list of str, optional
Stores attached read-only. Each is validated before deploying.
Omitted = none (0.3.6.9): like database, the pod's read-only stores
are stated, never taken from the session; the session's ones left out
are named in the log. Before 0.3.6.9 they were copied unvalidated.
database_view, data_domain : str, optional
View database and default data domain for the deployed agent.
token : str, optional
JWT for the AgentOps API. In AI Studio, the user_token from
%gen_token.
auth : {'service', 'jwt'}
Credential model baked into the generated agent.
**Prefer ``'jwt'`` where the environment allows it** — and note the
reasoning is the opposite of the notebook case, where a token that
expires in two hours makes a ``.env`` stop working:
* the token arrives **per request** from the caller, so expiry
self-heals instead of accumulating;
* the pod connects **as the caller**, so database permissions and
audit reflect the real user rather than one shared account;
* no durable database password sits in ``runtime_config``, which is
rendered **in clear** in the deployment UI — verified against a live
deployment. Its guarantee is "not committed to source control", not
secrecy.
``'service'`` is nevertheless the default, because ``'jwt'`` is
**conditional on something outside tdfs4ds**: the platform's
``auth_context.token`` has to be accepted by *Vantage* as a JWT logon
(``logmech='JWT'``, ``logdata='token=...'``). A token minted for a
platform API is not automatically a database credential — this package
has watched the MCP gateway accept a platform bearer and the database
then refuse the logon it attempted with it (Error 8017). A pod cannot
mint a token of its own, so where the platform's token is not a valid
Vantage JWT, ``'jwt'`` is simply not deployable and ``'service'`` with
a real account is the way in.
Test it before deploying, from the notebook, with the token the pod
will be handed::
tdml.create_context(host=..., logmech='JWT',
logdata='token=' + os.environ['OAUTH_ACCESS_TOKEN'])
A session there means ``auth='jwt'`` is viable; a refusal means it is
not, whatever else is configured.
mcp : bool, default False
Enable MCP tools (Enterprise edition only). Off is the recommended first
deployment: it removes Enterprise auth and the 132-tool context budget
from the run, so a failure is unambiguously about the agent.
dry_run : bool, default False
Validate, preflight and build the zip, then stop and return the payload
without touching the API.
require_published : bool, default True
Refuse when the running version is not on PyPI. Set False when a private
index carries it.
verify : bool, default True
Run the smoke test once the deployment is active. False returns as
soon as it is deployed, leaving verification to a later
:func:~tdfs4ds.agentops.smoke.check_deployed_agent call on
report["agent_url"] — useful when the pod is slow to warm, or when
the caller holding a token is not the one running the deploy.
startup_timeout : int, default 300
Seconds to wait for the pod's tdfs4ds startup block to appear in the log
before drawing conclusions from it. active is the platform's verdict
on the pod, which it reaches before tdfs4ds has finished importing, so
reading the log immediately finds nothing on a healthy deployment. The
wait ends early as soon as the block appears or an error signature
does. Set 0 to read once and move on.
progress : callable, optional
Called with a status line per step. None (default) shows a progress
bar over the stages instead; either way every stage is logged, a failed
one at ERROR with its cause and fix.
pod_host : str, 'auto' or None, default 'auto'
The database host the pod connects to. 'auto': on AI Studio, where
a pod cannot resolve the database name the notebook uses, the name is
resolved here (as is, then with cop1..cop3) and the pod gets the
address; elsewhere the session's TDFS4DS_TD_HOST is used unchanged.
A string is used as given; None keeps the session's value. The
session itself is never changed.
Returns¶
dict
{"ok", "verified", "stages", "store", "agent_definition_id",
"agent_id", "agent_url", "insights", "smoke", "payload"}. Stages use
the same {name, status, detail, hint} shape as every other tdfs4ds
report.
``ok`` is the **deployment's** verdict: the pod is active and has a URL.
``verified`` is the **smoke test's** — ``True``/``False``, or ``None``
when it did not run. They are separate because a check that depends on
pod warmth failing does not mean the deployment failed, and reporting it
as one flag sent a user to debug a deployment that was fine.
Raises¶
ValueError
database not given.
RuntimeError
The running version is not published and require_published is set.
Return findings for a pod log.
Parameters¶
text : str
Raw log text, as returned by GET /agents/{id}/logs.
max_excerpt : int
Per-finding excerpt cap. The excerpt is the first matching line, which
for a stack trace is the one naming the cause rather than the innermost
frame.
Returns¶
list of dict
{"name", "severity", "insight", "excerpt"}, errors first then
warnings then info, in table order within each severity. Empty when
nothing matched — which is itself a finding the caller should report as
"no known signature", never as "healthy".
check_agent_deployment(*, live=True, display=True, dest=None, auth='service', mcp=False, agent_runtime='auto', database=None, database_view=None, databases_readonly=None, data_domain=None)
Validate that this session can be packaged and deployed to AgentOps.
Runs :func:tdfs4ds.check_configuration's checks first — a configuration
broken here cannot become correct in a container — then the deployment
checks: the config split, package installability, the platform runtime
library, structured output, the vector backend, the business dictionary and
the connection environment.
Parameters¶
live : bool, default True
Make the outbound calls. False inspects configuration only.
display : bool, default True
Print the aligned summary, with a fix hint under anything not passing.
dest : str, optional
The scaffolded agent directory, so agent_runtime/ can be verified.
auth : {'service', 'jwt'}, default 'service'
Which credential model the deployment will use. 'jwt' expects the
caller's token per request, so only TDFS4DS_TD_HOST is required at
deploy time. Must match the auth= passed to
:func:generate_agentops_package.
Returns¶
dict
{"ok", "version", "live", "checks": [...]}, the same shape as
:func:tdfs4ds.check_configuration.
Notes¶
Never raises, and never returns or prints a secret — variable names only.
Examples¶
import tdfs4ds tdfs4ds.check_agent_deployment() # doctest: +SKIP tdfs4ds.check_agent_deployment(live=False) # doctest: +SKIP
generate_agentops_package(*, mcp_enabled=False, backend=None, auth='service', database=None, database_view=None, databases_readonly=None, data_domain=None, td_host=None)
Render the deployment package.
Model, embedding and Vector Store settings are read from the running session — you have proved those work by asking the agent questions, and reproducing them is the point. The feature store is stated, not inherited: it is a deployment target, and a pod that silently picks up whatever the session had is impossible to review at the call site.
Nothing is written and nothing is sent anywhere.
Parameters¶
mcp_enabled : bool, default False
Baked into the generated agent as the mcp_enabled argument to
:func:tdfs4ds.consumer_agent. Off by default; True is
Enterprise-edition only and adds per-caller MCP authentication.
backend : str, optional
Vector store backend to classify against. Defaults to the running
tdfs4ds.VECTOR_STORE_BACKEND.
auth : {'service', 'jwt'}, default 'service'
Credential model for the generated agent.
database, database_view, data_domain : str, optional
The feature store the deployed agent serves. Each falls back to the
running session when omitted; :func:tdfs4ds.deploy_to_agentops
requires database so a deployment always names its target.
databases_readonly : list of str, optional
Additional stores attached read-only (cross-store federation).
td_host : str, optional
The database host the POD connects to, when it differs from the
session's TDFS4DS_TD_HOST (a pod that cannot resolve the name gets
its address). Only runtime_config changes; the session is untouched.
Returns¶
dict
{"files": {name: text}, "runtime_config": {...}, "dropped": [...],
"missing_runtime": [...], "version": str, "auth": str, "store": {...}}.
``runtime_config`` goes in the deployment payload, **not** on disk.
``store`` reports the resolved feature store, so "which database will
this read?" is answerable before anything is deployed.
write_agentops_package(dest=None, *, mcp_enabled=False, backend=None, auth='service', database=None, database_view=None, databases_readonly=None, data_domain=None, agent_runtime='auto', make_zip=True, td_host=None)
Write the package into dest and (by default) build the deployable zip.
Parameters¶
dest : str, optional
The scaffolded agent directory — the one holding agent_runtime/.
Defaults to {tdfs4ds.AGENTOPS_PACKAGE_DIR}. Created when absent,
with a warning: a package with no agent_runtime/ registers fine and
then fails at pod start, so the gap is worth naming early.
make_zip : bool, default True
Also build <dest>.zip next to the directory — the artifact uploaded
to POST /agent_definitions.
Returns¶
dict
The :func:generate_agentops_package result plus "dest",
"zip_path" and "written" (the file names written).
deployment_payload(agent_definition_id, runtime_config, *, deployment_name=None, environment='dev', docker_image=None)
Build the POST /agents payload with the constraints already applied.
The result is sent to the API verbatim, so it carries only fields the API defines — no generation timestamp or other bookkeeping that a strict endpoint would reject.
The tutorial's payload is sized for a toy agent. Three fields differ here, and each is load-bearing rather than a preference:
resources
512Mi does not hold teradataml + langchain + pandas + networkx.
replicas: 1
Conversation memory is an in-process MemorySaver; a second replica
answers with no history.
enable_memory: False
The platform's PostgreSQL checkpointer cannot be injected into the
tdfs4ds graph, so enabling it would provision a database nothing reads.
Exercise a deployed AgentOps agent and report what works.
Sends three turns: an opening question, a follow-up on the same
session_id, and the same follow-up on a fresh one. Between them they
establish that the pod is reachable, answers, answers from the feature
store, remembers within a session and does not leak across sessions.
Parameters¶
agent_url : str
The agent_url from GET /agents/{agent_id}. A trailing
/invoke is added when absent.
token : str, optional
JWT bearer token. Defaults to TDFS4DS_TD_PASSWORD-independent
environment lookup of TDFS4DS_AGENTOPS_TOKEN; in AI Studio pass the
notebook's user_token.
live : bool, default True
`False makes no HTTP call — every check reports as skipped. For
wiring tests.
display : bool, default True
Print the aligned summary.
Returns¶
dict
{"ok", "version", "live", "checks": [...]}, the same shape as
:func:tdfs4ds.check_configuration.
Notes¶
Never raises, and never prints the token.
Examples¶
tdfs4ds.check_deployed_agent(agent_url, token=user_token) # doctest: +SKIP
remove_agent(agent, *, delete_definition=True, dry_run=False, others=False, token=None, base_url=None, timeout=300, progress=None, client=None)
Remove a deployed agent from AgentOps.
Parameters¶
agent : str
The deployment's UUID, a prefix of it (the short id the Playground shows,
e.g. '85f752e'), or its deployment name.
delete_definition : bool, default True
Also delete the registered agent definition, unless another deployment
that is not terminated still uses it (then it is kept and the report says
which).
dry_run : bool, default False
Resolve and report what would be removed; send nothing.
others : bool, default False
GET /agents lists the whole workspace, so a name can belong to another
user's agent. By default a deployment whose created_by is not the
token's user is refused; True removes it anyway (if the platform allows).
token, base_url
As for :class:AgentOpsClient (in an AI Studio notebook, the token from
%gen_token).
timeout : int, default 300
Seconds to wait for the platform to report the deployment gone.
progress : callable, optional
Called with a status line per step.
Returns¶
dict
{"ok", "dry_run", "agent_id", "name", "definition_id", "status_before",
"status_after", "definition", "steps", "error"}. Never raises for an
API refusal: the full response is in error and logged at ERROR.
The URL of a deployed agent, found by name, id or short id.
Parameters¶
agent : str
Deployment name (e.g. 'tdfs4ds-propensity-demo'), UUID, or the short
id the Playground shows.
invoke : bool, default False
False: the agent's base URL, what deploy_to_agentops reports as
agent_url and check_deployed_agent takes. True: the /invoke
route a client POSTs a question to.
token, base_url
As for :class:AgentOpsClient (in an AI Studio notebook, %gen_token).
Raises¶
LookupError No deployment, several, or the deployment is not active (no URL to call).
Attach one or more feature stores to the session in READ-ONLY mode.
Parameters¶
databases : str | dict | list
A store (or list of stores). Each is either a string ('DB_B' ⇒
schema == schema_view == 'DB_B', default vector prefix) or a dict
descriptor ({'schema':..., 'schema_view':..., 'vector_prefix':...}) for
a table/view split or a custom vector prefix.
validate : bool
When True (default), each store is probed for the catalog views + SELECT
access + column compatibility; a store that fails is logged and skipped
(the good ones still attach).
Returns¶
list[dict] The full read-only descriptor list after the attach.
Detach named read-only stores (matched on schema or view schema,
case-insensitively), or all of them when databases is None.
Return the current read-only store descriptors (normalised dicts, including
the has_bd / has_lineage capability flags discovered at attach).
plan_migration(seeds=None, source=None, *, all_domains: bool = True, target_data_domain: Optional[str] = None, data_strategy: str = 'recompute', attach: bool = True, database_map: Optional[Dict[str, Any]] = None) -> Dict[str, Any]
migrate_objects(plan: Dict[str, Any], approvals: Optional[Dict[str, str]] = None, *, dry_run: bool = False, rebuild_vector_index: bool = True, progress: Optional[Callable[[str], None]] = None) -> Dict[str, Any]
Execute an approved migration manifest against the primary store.
Parameters¶
plan : dict
The manifest from :func:tdfs4ds.plan_migration.
approvals : dict[str, str] | None
{conflict_key: 'skip'|'replace'} for every conflict. Required whenever
the plan has conflicts; a replace that is still blocking (the
downstream-safety gate) is rejected.
dry_run : bool
Validate approvals and return the plan of record without writing.
rebuild_vector_index : bool
Rebuild the Teradata vector index after BD migration (default True; skipped
unless VECTOR_STORE_BACKEND == 'teradata').
progress : callable(str) | None
Per-step progress hook (for the SSE panel).
Returns¶
dict
Execution report: pid_map (source→primary), per-object outcomes,
skipped/replaced, and any post-migration downstream compile breakages.
Render the migration manifest as an approval-ready report.
Parameters¶
plan : dict
A manifest from :func:tdfs4ds.plan_migration.
fmt : 'markdown' | 'html'
Output format (default markdown).
generate_migration_script(plan: Dict[str, Any], approvals: Optional[Dict[str, str]] = None, *, data_strategy: Optional[str] = None) -> str
Return a standalone .py (string) that re-plans and executes this migration
with the given approvals baked in. Inspect before executing.
write_migration_script(plan: Dict[str, Any], out_path: Optional[str] = None, approvals: Optional[Dict[str, str]] = None, **kw) -> str
Write the generated migration script under tdfs4ds.SCHEDULING_SCRIPT_DIR
(or out_path); return its path.
install_skills(target_dir: Optional[Union[str, Path]] = None, *, overwrite: bool = True, skills: Optional[List[str]] = None, verbose: bool = True) -> List[Path]
Copy bundled SKILL.md files to target_dir.
Parameters¶
target_dir : str or Path, optional
Destination root. Defaults to ~/.claude/skills/ (Claude Code
user-level skills directory).
overwrite : bool, default True
Replace existing SKILL.md files. Set to False to skip skills
that already exist at the destination.
skills : list[str], optional
Subset of skill names to install. Defaults to all bundled skills.
verbose : bool, default True
Print one line per installed skill plus a final summary.
Returns¶
list[pathlib.Path]
Destination paths of every SKILL.md that was written (skipped
files are not included).
Notes¶
The destination directory layout mirrors the package source — one
directory per skill, each containing a SKILL.md::
target_dir/
fs-setup/SKILL.md
fs-upload/SKILL.md
...
install_skills_global(*, overwrite: bool = True, skills: Optional[List[str]] = None, verbose: bool = True) -> List[Path]
Install bundled SKILL.md files to the global Claude Code skills directory.
Convenience wrapper around :func:install_skills that targets
~/.claude/skills/ — the default Claude Code user-level skills location
shared across all local projects.
Use this when you want to make tdfs4ds skills available to Claude Code globally (in your home directory), so they appear in the IDE extensions, CLI, and web app without repeating the installation per project.
Parameters¶
overwrite : bool, default True
Replace existing SKILL.md files. Set to False to skip skills
that have already been installed.
skills : list[str], optional
Subset of skill names to install. Defaults to all bundled skills.
Example: skills=['fs-upload', 'fs-dataset'].
verbose : bool, default True
Print progress — one line per installed skill and a final summary.
Returns¶
list[pathlib.Path]
Destination paths of every SKILL.md file that was written.
Skipped files (when overwrite=False and already present) are
not included in the return value.
Notes¶
This is equivalent to calling:
.. code-block:: python
install_skills(
target_dir=Path.home() / ".claude" / "skills",
overwrite=overwrite,
skills=skills,
verbose=verbose,
)
Examples¶
Install all bundled skills globally (one-time setup):
import tdfs4ds tdfs4ds.install_skills_global() installed fs-agent -> /Users/you/.claude/skills/fs-agent/SKILL.md installed fs-analyze -> /Users/you/.claude/skills/fs-analyze/SKILL.md ... 19 skill(s) installed to /Users/you/.claude/skills
Install only a few specific skills:
tdfs4ds.install_skills_global( ... skills=['fs-upload', 'fs-dataset', 'ca-search'], ... verbose=True ... )
Reinstall a skill (replace if already present):
tdfs4ds.install_skills_global( ... skills=['fs-upload'], ... overwrite=True ... )
See Also¶
install_skills_local : Install to a local project directory instead. install_skills : Lower-level function accepting arbitrary target paths.
install_skills_local(*, overwrite: bool = True, skills: Optional[List[str]] = None, verbose: bool = True) -> List[Path]
Install bundled SKILL.md files to the local project's Claude Code directory.
Convenience wrapper around :func:install_skills that targets
./.claude/skills/ — the project-level skills directory for Claude Code
in the current working directory.
Use this when you want to version-control tdfs4ds skills alongside your project code (e.g. in a monorepo or framework where custom agents may modify or extend the skill set). Skills installed locally override the global installation for this project.
Parameters¶
overwrite : bool, default True
Replace existing SKILL.md files. Set to False to skip skills
that have already been installed locally.
skills : list[str], optional
Subset of skill names to install. Defaults to all bundled skills.
Example: skills=['fs-upload', 'ca-explain'].
verbose : bool, default True
Print progress — one line per installed skill and a final summary.
Returns¶
list[pathlib.Path]
Destination paths of every SKILL.md file that was written.
Skipped files (when overwrite=False and already present) are
not included in the return value.
Notes¶
This is equivalent to calling:
.. code-block:: python
install_skills(
target_dir=Path.cwd() / ".claude" / "skills",
overwrite=overwrite,
skills=skills,
verbose=verbose,
)
The local skills directory is created automatically if it does not exist.
Examples¶
Install all bundled skills locally (one-time setup in a project):
import tdfs4ds import os os.chdir('/path/to/my/project') tdfs4ds.install_skills_local() installed fs-agent -> /path/to/my/project/.claude/skills/fs-agent/SKILL.md installed fs-analyze -> /path/to/my/project/.claude/skills/fs-analyze/SKILL.md ... 19 skill(s) installed to /path/to/my/project/.claude/skills
Install only consumer-agent reference skills locally:
tdfs4ds.install_skills_local( ... skills=[ ... 'ca-search', 'ca-definition', 'ca-explain', 'ca-lineage', ... 'ca-summary', 'ca-usage', 'ca-freshness', 'ca-dataset' ... ] ... )
Add new skills to an existing local installation:
tdfs4ds.install_skills_local( ... skills=['fs-new-workflow'], ... overwrite=False # skip if already installed ... )
See Also¶
install_skills_global : Install to the global Claude Code directory instead. install_skills : Lower-level function accepting arbitrary target paths.
export_skills(target_dir: Union[str, Path], *, overwrite: bool = False, skills: Optional[List[str]] = None, verbose: bool = True) -> List[Path]
Export bundled SKILL.md files to target_dir, skipping any that
already exist there.
Same behaviour as :func:install_skills but with a mandatory destination
and overwrite=False by default — use this when copying skills into a
target directory and you only want to add missing ones.
Parameters¶
target_dir : str or Path
Destination root.
overwrite : bool, default False
When False (default), skills already present at the destination
are left untouched. Pass True to replace them unconditionally.
skills, verbose
See :func:install_skills.
Returns¶
list[pathlib.Path]
Destination paths of every SKILL.md that was written (skipped
files are not included).
Examples¶
import tdfs4ds tdfs4ds.export_skills('./my_agent/skills') # skip existing tdfs4ds.export_skills('./my_agent/skills', overwrite=True) # replace all tdfs4ds.export_skills('/srv/skills', skills=['fs-setup', 'fs-upload'])
Return the absolute path to the bundled SKILL.md (or prompt .md) for name.
Searches in two locations:
tdfs4ds/skills/<name>/SKILL.md— top-level skill directory (ca-*,qo-skill-*, etc.)tdfs4ds/skills/<skill>/prompts/<name>.md— flat prompt file nested inside a skill'sprompts/subfolder (e.g.qo-skill-optimize/prompts/qo-strategies.md).
Parameters¶
name : str
Skill or prompt identifier (e.g. 'fs-upload', 'qo-strategies').
Returns¶
pathlib.Path
Path to the .md file.
Raises¶
FileNotFoundError If no bundled skill or prompt with that name exists.
Return the prompt template body (after frontmatter) for skill name.
The returned string may contain {variable} placeholders intended for
use with 
meth:str.format (pure-Python prompts) or passed directly to
ChatPromptTemplate.from_template (LangChain prompts).
Parameters¶
name : str
Skill identifier, e.g. 'qo-candidates'.
Returns¶
str Prompt body with frontmatter stripped.
Return a mapping of every bundled skill to its frontmatter metadata.
Each value contains description (one-line summary used by agent
dispatchers) and path (absolute SKILL.md path). Frontmatter is
parsed without a YAML dependency — only top-level key: value lines
inside a leading --- block are extracted.
Returns¶
dict[str, dict[str, str]]
Examples¶
import tdfs4ds tdfs4ds.skill_catalog()['fs-upload']['description'][:60] 'Generate Python code to engineer features ...'
Return metadata for all consumer-agent skills, keyed by skill name.
Consumer-agent skills are identified by the ca- name prefix convention.
They are installed to ~/.claude/skills/ alongside other skills and are
also used by the LangGraph consumer agent to build its intent classifier and
skill registry dynamically.
Returns¶
dict[str, dict[str, str]]
Skill name → frontmatter metadata (includes intent, description,
and path).
Examples¶
import tdfs4ds tdfs4ds.consumer_agent_skill_catalog().keys() dict_keys(['ca-dataset', 'ca-definition', 'ca-explain', ...])
Return metadata for all query-optimizer skills, keyed by skill name.
Query-optimizer skills are identified by the qo-skill- name prefix and
must have an intent field in their frontmatter. They are used by the
LangGraph query optimizer agent to build its intent classifier and skill
registry dynamically.
Returns¶
dict[str, dict[str, str]]
Skill name → frontmatter metadata (includes intent, description,
and path).
Examples¶
import tdfs4ds tdfs4ds.query_optimizer_skill_catalog().keys() dict_keys(['qo-skill-explain', 'qo-skill-optimize', 'qo-skill-simplify'])
Lazy-load heavy optional modules (genai, agent) on first access.
Keeps import tdfs4ds fast by deferring LangChain / LangGraph / pandas
imports until the functions that need them are first called.
Submodules¶
- agent
- agentops
- ai_studio
- app_launcher
- attached_registry
- benchmark
- catalog
- catalog_export
- cli
- config
- configure
- data_domain
- data_domain_settings
- dataset
- datasets
- dependencies
- diagnostics
- docs_location
- documentation_export
- execution
- feature_store
- feature_values
- genai
- lifecycle
- lineage
- log_output
- mcp_launcher
- migration
- plotting
- process_store
- read_federation
- removal
- removal_log
- scheduling
- serve
- session
- store_discovery
- utils
- vector_store_selftest
- workflow_assessment
str, optional
A comment to attach to the created view for documentation purposes. Defaults to None.