Skip to content

Changelog

The authoritative changelog lives at the repository root.

Changelog

All notable changes to tdfs4ds are documented in this file.

[0.3.6.0] - 2026-10-31

New: one clock per data domain, soft clones made for you, run() over a list at one time

A data domain can now declare its business calendar once, and the processes of the domain stay in step without anyone managing cursors (tdfs4ds.data_domain_settings, exported on tdfs4ds). It is an alternative to attaching a TimeManager per process, which stays fully supported and gives more control (a calendar of its own, hand-made clones).

  • set_domain_time_manager(tm, data_domain=None) declares the clock, stored in a new optional table FS_DATA_DOMAIN_SETTINGS (created at setup() / connect(), like FS_PROCESS_SETTINGS; no existing table is altered). get_domain_time_manager, detach_domain_time_manager, domain_processes, and align_domain_time_managers(dry_run=True) for processes registered before the clock.
  • upload_features finds the TimeManager by itself: it walks the source view's lineage (through intermediate views) and records the manager the view reads. When that is the domain clock, the process gets its own soft clone <process view>_TM (same dates, own cursor) and its view is rewritten to read it, with a log line. New last parameter soft_clone=None (False = keep the clock, True = clone any manager). An explicit time_manager= is paired as in 0.3.5.19, even when it is the clock (no clone: tm.update(i); run(pid) keeps following it).
  • run(pid, time_manager=clock) for a process reading a soft clone of that clock sets the clone to the clock's date for the run, as roll_out does. It used to warn (or refuse under 'error') and stamp a date the view did not compute at.
  • roll_out(data_domain='X') (or roll_out() for tdfs4ds.DATA_DOMAIN) rolls out every process of the domain, driven by its clock, every clone moved to the same date at each step.
  • run([...]), run(data_domain='X'), run(pid, at='2024-03-31') compute processes at ONE time: at=, else the database clock read once. Each TimeManager view the processes read is swapped for that literal date during the computation and put back on its cursor afterwards, in a finally. No cursor moves, so a refresh "now" after a roll-out is possible without touching the managers. Processes run in dependency order; a failure skips the processes downstream of it; the call returns {pid: status}. A process reading no TimeManager is stored at the same time, with a warning. The swap changes only the value the view selects: it still reads FROM <table> WHERE TIME_ID = n, so lineage to the table of dates holds and an interrupted run is repaired by the next one (TimeManager.restore_cursor() by hand).
  • Documentation: a soft clone takes its clock's documentation (PROPAGATED, tdfs4ds.genai.clone_docs), no LLM call per clone; a clock without documentation leaves its clones to the usual path.
  • Migration: plan_migration reads the source domains' clocks (plan["domain_clocks"]) and migrate_objects recreates each clock view over the migrated table of dates, at its cursor, and declares it for the target domain. A time manager that came with the objects is no longer copied again (a copy per clone would split the calendar).
  • Reporting: the Report's Time drivers lists each domain clock with the processes on and off it; the workflow assessment adds process_not_on_domain_clock; get_process_info adds DOMAIN_TIME_MANAGER.
  • Admin app: the feature catalog's process card has a Domain clock row (the clock, and whether the process reads it, its own soft clone, no TimeManager or another calendar); the dataset builder offers the domain clocks first in its TimeManager list; the EFS map groups FS_DATA_DOMAIN_SETTINGS with the run drivers. Additive: a domain without a clock (or a store without the table) shows exactly what it showed before. Live-verified on efs-dev.

New: the documentation ships with the package, and the app shows it

  • The MkDocs site (docs/, built by mkdocs build) was never in the wheel: a pip install had no documentation, and the app's docs server started on a missing folder. setup.py now copies it into the package as tdfs4ds/docs_site (build_py hook; MANIFEST.in grafts docs for the sdist). Wheel 2.3 MB -> 9.2 MB.
  • The admin app serves it at /documentation on its own port (works behind a path prefix and Jupyter's proxy) and shows it in a new rail section, Help › tdfs4ds guide. The separate docs port (TDFS4DS_SERVE_DOCS_PORT) is kept and now finds the packaged copy too. A source checkout keeps serving its own docs/.
  • tdfs4ds.docs_path(page='index.html'): the local path of a documentation page.

Documentation and docstrings brought up to date

  • Docstrings: connect (had none), build_benchmark_html, build_dataset_opt (says which arguments it ignores), generate_on_clause, write_relocation_script; the undocumented parameters of run, roll_out and upload_features (and the run_now validation notes, which sat under close_absent_entities); the 0.3.6.0 functions and split_time_manager, unify_time_managers, link_time_managers, time_manager_dependency.
  • upload_features guide: the signature block is generated from the function (it showed a *, wrong defaults and lacked a dozen parameters), and rows for time_manager (missing), soft_clone, primary_index, force_compute, on_failure, check_upstream, grouped, storage_tag.
  • Configuration page: 20 settings that were not listed (TIME_MANAGER_CHECK, TIME_MANAGER_DETECT, PROCESS_SQL_VERSIONING, the MCP_* and embedding batch settings, proxies...).
  • docs_src/api/index.md, a static API export last regenerated on 2026-07-13 (its run() had 4 parameters), now points to the generated API reference. Broken anchors fixed; the admin UI section of the deployment page lists today's rail.

Changed (announced, each with an off switch)

  • upload_features without time_manager= records the manager the view reads, even with no domain clock. Before, such a process stayed undeclared until its first roll_out, so run(pid) stamped real time while the view computed at the manager's date (the warning warn_if_undeclared already called that out). Several managers read: a warning, nothing recorded. tdfs4ds.TIME_MANAGER_DETECT = False (env TDFS4DS_TIME_MANAGER_DETECT) restores the old behaviour. This changes the valid time of the runs of such processes from real time to the manager's date.
  • roll_out runs the processes of each date in dependency order (from the lineage graph). The sort is stable: a list already in a valid order, or whose processes do not depend on each other, runs exactly as before. order='given' keeps the list's order.

Fixed

  • Migration of a TimeManager failed every time (recreate_manager: "tddf must be an instance of tdml.DataFrame"; a pandas frame was passed to load_time_steps). It is now a hard clone (table copied, view at the source's cursor). Seen live; it went unnoticed because the manager's view and table usually come with the migrated objects.
  • A soft clone of a manager in another database looked for its table in its own database: update(), _exists(), nb_time_steps, take_ownership and generate_timeline (which also hardcoded <view>_HIDDEN) read the wrong table, and roll_out placed it on another calendar. TimeManager.table_schema now holds the table's database (read from the view, the registry's TABLE_DATABASE_NAME otherwise), manager_index takes the registry's table database, and the generated parallel script groups processes by table of dates, so soft clones of one manager run in one lockstep group (every clone moved at each date).
  • TimeManager._exists() failed on a database with no table yet (db_list_tables returns no TableName column).

Backward compatibility

  • With no domain clock, the calls run(pid) (real time, explicit time, argument, recorded manager), roll_out (plain and with a soft clone) and the generated script make are compared with traces captured from 0.3.5.19 (tests/fixtures/domain_clock_baseline.json): identical.
  • New parameters are keyword-only and last (at, data_domain, order on run; data_domain, order on roll_out; soft_clone on upload_features). run(pid) returns what it returned; the status dict is only for a list or a domain.
  • Older clients sharing a store ignore FS_DATA_DOMAIN_SETTINGS; a soft clone is an ordinary manager view to them.

Backward compatibility, live (efs-dev): the same classic workflow (TimeManager attached at registration, a process registered without one, roll_out, tm.update + run, run(time_manager=), an explicit FEATURE_STORE_TIME) run by 0.3.5.19 and by 0.3.6.0 in two throwaway stores: catalogs, follow-up rows, stored values and view DDL identical at every step, except the announced one (the process registered without time_manager= is paired at registration instead of at its first roll_out). A 0.3.5.19 client on a store where 0.3.6.0 made a domain clock and clones connects, continues the history with its own roll_out (clones kept in step) and runs run(pid) on the recorded clone. A migration of a domain with a clock carries it (clock declared in the target, clones on its table, roll_out(data_domain=) and run(data_domain=) there).

Live-verified on Vantage 20 (efs-dev): domain clock with three processes (one reading another, one with no manager), detection and soft clones at registration, roll_out(data_domain=) history equal to the expected sums in dependency order, clones in step, run(data_domain=) at one instant with every clone back on its cursor, run([p], at=), a clone left on a literal date repaired by the next run and by restore_cursor(), a soft clone in another database (update, dates, swap and restore). Guarded by tests/test_domain_clock.py.

[0.3.5.19] - 2026-10-30

Skills: the remaining verified traps moved into the package

The repository's .claude/skills/fs-shared/gotchas.md (written for 0.3.1.141, never shipped, no longer referenced by any skill) is removed; what is still true moved into the package skills, so install_skills() delivers it:

  • fs-setup: never setup(if_exists='replace'), which drops the catalogs of the database (every user's features).
  • fs-upload: df=None is still required with sql_query=; re-running a process writes nothing for an unchanged (bit-identical) value.
  • fs-inspect: new section Querying the catalogs in SQL (object names with the real FS_FS_ prefix, ENTITY_NAME vs ENTITY_ID, FEATURE_VIEW vs VIEW_NAME, joining on FEATURE_NAMES, LIKE ... ESCAPE, TRIM(TableName), ValidStart/ValidEnd instead of BEGIN(VALIDTIME), no ORDER BY in from_query, lazy DataFrames).
  • fs-dataset: a dataset is a view, not a copy.
  • td-analytics-workflow: copy_to_sql(if_exists='append') with NaN in a text column.

Outdated points (connection before import, grouped not persisted, unqualified names rejected, DATASET_TYPE snapshot only) were dropped. No code change.

[0.3.5.18] - 2026-10-30

Fixed: the consumer agent follows up on what its previous answer listed

Seen live on AI Studio (both demos in one store, 7 datasets): after "What datasets are available?", "Is it up to date?" answered "I don't have any features or datasets in our conversation context yet", and "Give me a sample" asked a generic three-way question; with one dataset listed, the freshness answer named a feature ''. And "give me a sample" after a question on one dataset sampled other datasets.

  • A follow-up that points at nothing it names ("is it up to date?", "give me a sample", "where does it come from?", French too) right after a listing (dataset inventory, feature summary, search) refers to the listed items: one item is used directly; several are offered as a numbered question, each with the question rewritten ("Is db.ds_x up to date?"), and the reply number runs it. Also when the intent classification fails on the gateway, which used to send the question to RECAP.
  • The MCP data query receives the object routing resolved: "give me a sample" about the conversation's dataset reached the MCP agent without the dataset name, which then sampled others. The question now carries it when it does not name it already. The classifier's failure path also applies the remembered dataset to a bare sample request (it returned before reaching that rule).
  • Live-verified on AI Studio: freshness and samples chosen from the 7-dataset list, a sample after a freshness answer, a single-dataset list answered directly.

[0.3.5.17] - 2026-10-30

Changed: value types a feature cannot have are refused before registration; datasets avoid aggregating types MAX() refuses

Measured on Vantage 20: every type goes through the ingestion's temporal MERGE, LOBs included. The limits are comparison and aggregation (tdfs4ds.feature_store.value_types).

  • Refused at registration: CLOB, BLOB, JSON and now XML. upload_features checks the source types before writing anything and returns None with the reason and what to do (extract scalar fields in the view). These values cannot be compared (Errors 5689, 3600): an unchanged value cannot be recognised, so every run would open a new period, and a history cannot merge equal periods. Before, CLOB/BLOB/JSON were refused only when the process ran, after it was registered; XML was not refused.
  • Run time unchanged: CLOB/BLOB/JSON still raise; a process registered earlier with an XML feature keeps running, with a warning.
  • Grouped datasets and lag/lead pivots read MAX(CASE ...), which Teradata refuses on periods (6976), ST_GEOMETRY and other user-defined types, XML, JSON (6945) and LOBs (5664). build_dataset / rebuild_dataset / build_grouped_dataset now read the storage tables' value types (HELP COLUMN, one per table, only when the requested form aggregates) and build such a dataset multi-join and/or with one sub-query per lag/lead column, with a warning naming the features. Before, the view creation failed with the database error. A dataset without those types is built exactly as before.
  • Live-verified on efs-dev: an XML feature refused with nothing registered; a grouped dataset and a lag dataset holding a PERIOD feature created and readable.

Added: tdfs4ds.open_session(env_file=None, database=None, token=None)

One call replaces the block every AI Studio notebook repeated (%gen_token, then the Vector Store token, the MCP bearer, load_config, create_context, connect). It loads the nearest .env (the current folder, then each parent folder, then ~/.tdfs4ds/.env; the file used is logged and returned; pass a path to choose another, False for none), takes a current token (the one given, else the platform's own, the way %gen_token gets it) and puts it wherever the file uses a token, opens the teradataml context from the TDFS4DS_TD_* settings, connects to the store (TDFS4DS_TD_DATABASE by default) and renews the token in the background. With a password .env it is load_config + logon + connect. connect=False stops after the logon (for setup()).

[0.3.5.16] - 2026-10-30

Changed: roll_out refuses dates that would add nothing to a stored history

A run at business date t writes its values from t onward, but never before the first value already stored. A process first run in real time (upload_features runs at once by default) or at a later date therefore could not be rolled out over past dates: every step rewrote the period from the first stored value onward, the history ended as one period holding the last date's value, and nothing said so.

  • Checked before step 1. roll_out reads where each process's stored values start (one MIN(ValidStart) over its storage tables, filtered on its process id) and compares it with the first date it would roll out (time_id_start). When the values start later, it raises ValueError naming the processes, the stored start and the fixes, and runs nothing.
  • roll_out(..., rebuild_history=True) deletes the stored values of those processes from the first date rolled out onward (ValidStart >= that date: in this case all of them; earlier periods are never touched), and their entity-refresh rows (logged in FS_REMOVAL_LOG as a VALUES removal), then replays every date with force_compute=True. With nothing to delete it only forces the recomputation.
  • Not checked: a process with no stored value yet, a process whose valid time comes from a source column (valid_time_column), a manager object without date_of. A start that cannot be read is a warning and the roll-out goes on.
  • TimeManager.date_of(time_id) returns the business date of a step, formatted like get_date_in_the_past(), without moving the cursor.
  • Live-verified on efs-dev: a process run in real time is refused for a roll-out from 2024-01-01; with rebuild_history=True the three monthly values match an independent reference; a second roll-out is accepted.

[0.3.5.15] - 2026-10-30

Changed: roll_out keeps processes on soft clones consistent, and finds the manager itself

  • Soft clones follow the manager. roll_out([p1, p2], tm) where p2's view reads a soft clone of tm (same table of dates, own cursor: clone_timer(clone_mode='soft'), split_time_manager) used to warn and run p2 at its own, unmoved cursor while stamping its values with tm's dates. The clone's cursor is now moved to the same step before the processes run, at each date; p2 keeps its own pairing. Verified on efs-dev: two processes with the same SQL, one on tm, one on its soft clone, store the same values at every date.
  • Another table of dates is refused, even under TIME_MANAGER_CHECK='warn', before anything runs (a known-inconsistent result used to be written with a warning). A hard clone has its own table, so it is refused too: roll it out with its own manager. A view that reads no manager is unchanged (warning under 'warn').
  • roll_out(process_list) without a manager uses the TimeManager attached to the processes (process catalog). They must share one table of dates (a manager and its soft clones; the owner of the table drives); otherwise ValueError lists the managers. roll_out(process_list, tm) is unchanged.
  • tdfs4ds.utils.time_management.point_time_manager_view (the statement update() runs, for a view whose table lives in another database). The clone log lines of clone_timer / clone_filter are ASCII (an arrow made logging fail on a Windows console).

Changed: skills carry the traps of a time-managed ML use case

fs-rollout, fs-dataset, fs-upload, fs-document, fs-tdstone2 and td-analytics-workflow gained what the propensity demo (notebook dev/propensity) ran into: month-start calendars with monthly lags (ADD_MONTHS clamps month ends), registering with run_now=False before a backfill, the train (history + lead) / score (snapshot following the TimeManager) / monitor recipe, what FULL OUTER and INNER do with lags, CAST to FLOAT before a division, a BYOM / ONNX scoring process picking the model trained at or before the business date, KPIs as a process, documenting sources first, and checking that the Script operator is licensed before choosing tdstone2.

Added: documentation knows the in-database operators a process calls

A process view that scores a BYOM model or compresses time series with td-ts-compressor was opaque to the LLM: it saw a function call and string parameters. tdfs4ds.genai.operator_context recognises known operators in the SQL and gives the documentation (upload_features(document=True), intermediate views) and the agent's EXPLAIN a fixed block written by code, as ground truth. A SQL calling no known operator gets nothing: every other prompt is unchanged.

  • BYOM (ONNXPredict, PMMLPredict, H2OPredict, DataRobotPredict, DataikuPredict, MLeapPredict, ONNXEmbeddings, ONNXSeq2Seq, ONNXClassification, any database qualifier: TD_MLDB, mldb ...): the model source is followed to its table (directly, the common case, or through views, at most 3 levels; one DBC.TablesV read decides), and the table's metadata columns are read (no LOB / byte / JSON column; 20 rows at most, the rows a simple WHERE selects first, then newest first). The block gives the input, the model source and the selection rule as written, the model rows (the selected one marked), the USING parameters and the meaning of the output columns. A refused read becomes a note, never a guess. Live-verified on efs-dev: the documentation and EXPLAIN of a scoring process name the model in use and the one it replaced, from the table's own metadata.
  • td-ts-compressor (ts_compress_to table operator; mapping ts_compress_seq_to_map, any qualifier, or a renamed mapping recognised by its USING signature; the unrelated system function TD_SYSFNLIB.ts_compress is not matched): per call the series, time and value columns, the parameters as written with their defaults, the mode they imply (visual fidelity, analytical points, fixed budget), what compression_importance / compression_distance mean, and the features feature_sql derives (compression_ratio, distance_rank_N ...). Live-verified on efs-dev on a process built from td-ts-compressor's own SQL, with and without its business-dictionary rows (imported rows are kept as written).

[0.3.5.14] - 2026-10-29

Fixed: consumer-agent answers found weak on the propensity demo (seen live on AI Studio)

  • "Which datasets are used to train the propensity models?" listed every dataset of the store. A dataset inventory now keeps its topic words and its purpose: the topic narrows the list to the datasets whose name, entities, data domain or description share a word with it, and "train" / "entraîner" keeps the datasets that carry a target (a LEAD offset column, else "train" in the name). Each filter narrows only when something matches; "what are the available datasets?" and "datasets for CustomerID" give exactly the answers they gave before. skill_list_datasets(topic=, purpose=), both optional.
  • A search answer whose documentation was trimmed for length said documented training datasets "lack documented details": the synthesis is now told that part of the documentation was left out, and never to say something lacks documentation. The trace fields dropped_unverified / dropped_for_budget are no longer given to the language model (it repeated them as "flagged ... may require verification").
  • Features of two data domains were presented as one use case: each feature match now carries its data domain (a dataset's lag / lead column takes its base feature's), and matches from several domains are grouped by domain.
  • Lag and lead columns of datasets (N_EVENTS_LAG_1MO) are known names for the agent's grounding checks: they were dropped from search answers as unverified.
  • Documented data read by the store's processes (a model table read by the scoring process) is named as their source data, with the processes that read it, instead of "outside the feature store, documented data also covers this". The line for data no process reads is unchanged.

Fixed: two package gaps found running the propensity demo

  • upload_features(run_now=False, dataset_view_name=...) dropped the dataset. The name only reached run() as an argument, so the later roll_out built no dataset and nothing said so. It is now recorded as process settings (DATASET_VIEW_NAME, DATASET_GROUPED in FS_PROCESS_SETTINGS, no DDL), and a run() that passes no dataset_view_name builds the recorded one (so roll_out, at each date). An explicit dataset_view_name still wins; a failed record is a warning naming run(pid, dataset_view_name=...). run_now=True records nothing: unchanged. A run reads the process settings once more (fail-open).
  • TimeManager.load_time_steps failed with Error 3515 ("Duplication of column BUSINESS_DATE") when the source column was named business_date (any case), or when the source had a TIME_ID column: the two columns it adds collided. Such a source now keeps only those two columns from the start (assign(drop_columns=True)); any other source gets the same statement as before.

Added

  • notebook dev/propensity/: a conversion propensity use case end to end. Simulated web interactions of 8,000 prospects; a TimeManager on month starts; two feature processes rolled out over 9 months; training on historical datasets with lags and a one-month lead as target; two models trained at two dates, exported to ONNX and stored with BYOM; each month scored by the latest model trained by then (a process); validation as a monitoring process (ROC-AUC and top-decile lift of last month's scores against this month's conversions, with a history); the agent. Live on AI Studio: out-of-time ROC-AUC 0.67 to 0.75, lift 2.3 to 3.3.

[0.3.5.13] - 2026-10-29

Fixed: two answers that depended on how the model filled its fields (seen live on AI Studio)

  • "Top 5 customers by TOTAL_SPEND" answered "No feature named 'TOTAL_SPEND' for entity 'customers'" when the classifier wrote the entity as the plain word: with one feature of that name, its own entity is now used.
  • "Which dataset gives me the RFM features?" answered "DS_CUSTOMER_RFM is not the name of a registered feature" when the classifier returned the dataset's own name: a registered dataset named exactly (any case, with or without its database) is now answered as that dataset.

Changed

  • notebook dev/agentops/notebook_easy/: each notebook opens with a diagram of what it does on the data plane (the values) and the metadata plane (what they mean), drawn as SVG (the Mermaid version clipped its labels in JupyterLab). Notebook 04 asks the agent on both planes: catalog and documentation, the top values from the stored features, and a sample through the MCP server.

[0.3.5.12] - 2026-10-29

Changed: deploy_to_agentops prepares two things notebooks did by hand

  • The pod's database address (pod_host='auto', new last parameter): an AgentOps pod cannot resolve the database name a notebook uses. On AI Studio the name is now resolved where the deploy runs (as is, then with cop1..cop3, as the driver does) and the pod's runtime_config gets the address; the session's TDFS4DS_TD_HOST is not changed. Off AI Studio, or when the host is already an address, nothing changes. A string gives the host explicitly; None keeps the session's value. The resolved value is in report["pod_host"].
  • The platform's agent_runtime: with agent_runtime='auto', a package folder without one now gets the copy the image ships (AI Studio: ~/Demo/AgentOps/agent_runtime) before tdfs4ds generates its own. A copy tdfs4ds generated on an earlier run used to be reported as the platform's and kept at its old version; it is now replaced (by the platform's copy when there is one, else regenerated). A real platform copy already in the folder is still never touched.

Changed: the app starts from the generated .env and the session token

  • generate_ai_studio_env writes TDFS4DS_SCHEMA (the user's database, where setup() creates the store). Without it the app opened on no feature store, which is why the demo patched it into .env before launching. It is not applied while the file is checked, since the store may not exist yet.
  • launch_app(token=user_token) (new last parameter): the app logs on with the session token and gives it to the Vector Store and the MCP gateway, over the token the .env holds (which expires about two hours after it was written). Nothing is written to the file. A token cannot go into the configuration once and for all: it expires, so it is passed at launch.
  • On AI Studio the app renews its own token (new tdfs4ds.utils.session_token). %gen_token calls the platform module oauth_token_helper, which asks the JupyterHub hub for the user's current token (the hub renews it every 20 minutes); any process of the user's server can call it. When the .env logs on with a token, launch_app starts the app with the current one (no token= needed) and the app renews it every 15 minutes: the stored logon used by an automatic reconnect, the Vector Store token and the MCP bearer, each only where a token was already in use. Live-verified: an app started from a .env whose token had expired 250 minutes earlier connected and started renewing. Inert elsewhere: without oauth_token_helper, or with a password logon, nothing changes. tdfs4ds reads no unprefixed variable; the platform's module does.
  • generate_ai_studio_env writes the upload_features defaults that match its models: UPLOAD_DOCUMENT and UPLOAD_EXPLAIN on when an instruct model is configured, off otherwise; UPLOAD_VECTOR_INDEXING on only when the embedding model is configured too (the index is built from the documentation); UPLOAD_OPTIMIZE_QUERY always off (it rewrites the user's view, which must be asked for). The package defaults are unchanged; an explicit argument still wins.

Added

[0.3.5.11] - 2026-10-29

Changed: the AgentOps registry entry names the feature store

  • Before: every agent deployed by deploy_to_agentops was registered as "tdfs4ds feature-store consumer agent" under the name tdfs4ds-consumer-agent-<uuid>, so with several agents deployed nothing told which store each one serves.
  • Now: the default description states the feature store, its view database, its data domains (read from the process catalog; at most 8 listed), the stores attached read-only, what the agent answers, MCP on or off, the logon mode and the tdfs4ds version. It carries no counts, since those go stale as the store grows. The default name is tdfs4ds-agent-<database>-<8 hex>, and the deployment now uses the same name as the definition (each used to get its own random one). A dry run shows both in report["definition"]; name= and description= still override them.

[0.3.5.10] - 2026-10-29

Fixed: no traces in the AgentOps Observability panel

  • Cause: AgentOps names each agent's Opik project after the agent (OPIK_PROJECT_NAME = its agent id, set in the deployment's runtime config), and the Observability panel shows that project only. tdfs4ds passed its own project_name='tdfs4ds' to the Opik tracer, and the deployment package wrote TDFS4DS_OPIK_PROJECT_NAME: tdfs4ds into config.yaml, so every trace of a deployed agent went to a project called tdfs4ds (seen live: traces accepted by Opik, nothing in the panel).
  • Now: the tracer passes a project only when tdfs4ds.OPIK_PROJECT_NAME is set; None leaves it to the Opik SDK's own configuration. The deployment package writes None when the session has the default (tdfs4ds) or nothing, so the pod traces into the platform's project; a name set on purpose is kept. Locally nothing changes: the default is still tdfs4ds.
  • An agent deployed with an earlier version keeps tracing into tdfs4ds: redeploy it (notebook 06) to see its traces in the panel.

[0.3.5.9] - 2026-10-29

Found by a multi-turn test of the agent deployed on AgentOps (notebook 08).

Fixed: an empty answer to "Is it up to date?" in a new conversation

  • Cause: with tool calling (STRUCTURED_OUTPUT_METHOD='function_calling'), a model that makes no tool call leaves the classifier with nothing: the parser returns None instead of raising, and the turn failed on it. The deployed agent answered with an empty text.
  • Now: the classification is asked once more, then treated like any unusable one. A question that only refers back ("it", "this") with nothing in the conversation goes to RECAP, which says there is nothing to refer to yet. Other questions keep the SEARCH fallback.

Fixed: "the second one" after a listing

  • "What does the second one mean?" after a listing of features asked which item was meant. An ordinal that stands for an item ("the first one", "the last feature", "le deuxième") is now replaced by that item of the previous answer: its last numbered list, else the names of its table rows. "the last month" and an ordinal beyond the list are left alone.

Changed: MCP tools that take the database apart, decided by the server's answer

  • 0.3.5.7 split a qualified table_name into db_name + table_name before every call to a tool with a db_name argument. Now the call is sent as the model wrote it, and only a 3807 naming that qualified name ("Object 'db.table' does not exist", the Enterprise server) is retried once with the database apart, in the tool's own argument (db_name or database_name). The Community server, which builds db.table itself, gets exactly the calls it got before 0.3.5.7. Live: the Enterprise retry returned the sample.

Fixed: the deployed-agent check reported a session leak

  • check_deployed_agent's isolation test passed only on a few wordings of "I don't know what you mean". The agent's "I don't have any features or datasets in our conversation context yet" was reported as a leak. A fresh session that names nothing of what the first session resolved now passes; naming it still fails. Checked live with two interleaved conversations.

Added: notebook 08 - use the deployed agent

  • In notebook dev/agentops: calls <agent_url>/invoke with the token (single questions, a conversation through session_id, streaming), runs check_deployed_agent and reads the pod log.

[0.3.5.8] - 2026-10-29

The consumer agent, from the AI Studio demo. Each change applies only where the agent failed before: a name that resolves today resolves the same way, and no LLM call is added.

Fixed: a concept the catalog knows by a dataset

  • Before: "Which dataset gives me the RFM features?" answered "No dataset exposes RFM features" (DATASET) or "Nothing in the feature store matches" (SEARCH), although the dataset ds_customer_rfm is registered. The agent's name recognition only matched exact catalog names, and the DATASET answer read every name as a feature.
  • Now: when a name matches no catalog object exactly, the recognition (node_recognize_entity, _recognize_object_type(..., concepts=True)) tries, in order, the one dataset whose name holds its words ("RFM features" → ds_customer_rfm), the one dataset whose documentation does ("recency features"), then the features whose own description matches it. The recognition is now kept in the turn's state (recognized_entities). DATASET answers with the dataset and its features; SEARCH does the same when it found nothing, using the question's acronyms when no object was named. A unique match only. Live on AI Studio: both questions answered with ds_customer_rfm.
  • A name that is no feature: "No dataset exposes X" is now said only of a real feature. A name that is no feature reads "X is not the name of a registered feature", followed by the registered datasets.

Fixed: "give me a sample" after a question on a dataset

  • Before: after a domain listing then a freshness question on ds_customer_rfm, "give me a sample" listed CUSTOMERID values: the entity left in focus by the listing won.
  • Now: a sample request that names nothing ("give me a sample", "show me some rows", "un échantillon") while the conversation is about a registered dataset reads rows of that dataset (DATA_QUERY with its qualified name). Without MCP it says that reading rows needs MCP. Live: the sample came from dm250067.ds_customer_rfm.

Fixed: numbered replies

  • Before: a reply "2", or "1 last month, 2 customers", to a numbered list in the previous answer (next steps, questions) was classified on its own and lost its meaning. Only the agent's own clarification prompts were connected.
  • Now: when no flow is waiting for a reply, such a reply is written out with the items it points at ("Which time window: last month; Which entity: customers") before classification, and the skill answers that. Every number must be an item of the last numbered list of the previous answer, and the reply must start with one; anything else is classified as before. Verified offline only.

[0.3.5.7] - 2026-10-29

More from the AgentOps demo on Teradata AI Studio. As in 0.3.5.6, each fix applies only in the situation it fixes.

Fixed: a dataset sample through the Enterprise MCP server

  • Before: "give me a sample of the ds_customer_rfm dataset" answered that the dataset does not exist. The agent names datasets with their database, and the Enterprise server's base_tablePreview (also base_tableDDL, base_columnDescription...) takes the database in db_name: given table_name='db.dataset' it looked for an object of that name (Error 3807). The Community server, used so far, answered the same call.
  • Now: when a tool has a db_name argument the model left empty, a qualified table_name / obj_name is split into the two arguments. Other calls are sent unchanged.

Fixed: dataset sections still left out

  • A section still missing after the follow-up call of 0.3.5.6 is asked alone, as plain text. Asked alone through tool calling, FEATURE_THEMES came back as an empty object every time (Ministral on Bedrock, behind LiteLLM). Live: 5 sections of 5 written.
  • Several tool calls in one reply: with tool calling, a model may spread its answer over several calls of the same tool (seen: one call per theme). Their arguments are now merged; only the first was read. A reply with one call is read as before.

Fixed: the app on AI Studio

  • launch_app on a JupyterHub server: its proxy check reported "install jupyter-server-proxy" while the proxy worked. A JupyterHub single-user server refuses the token of its own runtime file, and the proxy answers an unauthenticated request with a 404. When the server's API refuses that token too, proxy_ok is now None (not checked) with a note to open the link. Elsewhere the check is unchanged.
  • Verified on AI Studio: through jupyter-server-proxy (installed with pip install --user, see the docs) the app at <jupyter base>/proxy/absolute/8000/ reports connected with the token, the Report and Lineage panels load, and the chat streams (SSE) when the request sends Accept: text/event-stream, as a browser does.

Changed: smaller items

  • check_configuration, LLM transport: a LiteLLM gateway was reported as INTERCEPTED: GET <base> -> 200 text/html. LiteLLM answers a bodiless GET on /chat/completions with a 500, so the probe went on to the base URL, which serves LiteLLM's web console. Any error status from /chat/completions now counts as the API answering; a redirect or a 200 page there is still reported.
  • markdown is now a dependency (install_requires). The chatbot, the query-optimizer report and the admin panels render Markdown with it; it was declared in the serve extra only, so a plain install showed raw Markdown there.

[0.3.5.6] - 2026-10-29

Found by running the AgentOps demo on Teradata AI Studio. Each fix applies only in the situation it fixes; everywhere else the behaviour, the SQL and the requests are unchanged.

Fixed: the app logs on with a token

  • Before: with a token logon (TDFS4DS_TD_LOGDATA='token=…', TDFS4DS_TD_LOGMECH=JWT, no password), as on AI Studio, the app started disconnected ("missing connection field(s): password").
  • Now: when no password is set and TDFS4DS_TD_LOGDATA is, the app opens its connection with it, the way the automatic reconnect already did. A password, when set, is used as before.
  • launch_app no longer says the app starts disconnected when the only missing setting is TDFS4DS_TD_ENCRYPT: the app connects without wire encryption.

Fixed: agent questions answered "nothing matches" behind tool calling

  • When: STRUCTURED_OUTPUT_METHOD='function_calling' (for example LiteLLM in front of Bedrock).
  • Cause: a few agent calls ask for a JSON list, and a tool's parameters must be an object. Bedrock refused the request, and SEARCH answered that nothing matched.
  • Now: a list is sent as the single property of an object and taken back out of the reply. Requests asking for an object are unchanged.

Fixed: grouped datasets documented with only some of their features

  • The dataset documentation found its features with a pattern that fits datasets built without grouped=True only. A grouped dataset of 6 features was documented as having 2.
  • Now every feature is found, whatever the dataset's shape. Datasets built without grouped=True get the same documentation as before.
  • Sections left out by the model: a reply that held only some of the five sections (seen with tool calling on Bedrock, which does not enforce required keys) is completed by one more call asking for the missing ones. A complete reply makes no extra call. A section returned as a list or an object is stored as text, one item per line.

Fixed: an expired token reported as a trust problem

  • The AgentOps preflight and generate_ai_studio_env test the token in TDFS4DS_TD_LOGDATA. An expired token was refused by the database and reported as "Vantage has to be configured to trust this token's issuer".
  • Now the token's expiry is read first: an expired token is reported as expired ("mint a fresh one with %gen_token"), and no logon is attempted.

Fixed: the agent and names written in plain words

  • A dataset named in words: "the customer RFM dataset" now resolves to ds_customer_rfm when no closer name exists (only a single match counts).
  • A follow-up naming its own object: a question that names something other than the object of the previous turn is no longer answered about that previous object.
  • A feature named by its meaning: "how is the recency feature calculated?" now lists the documented features whose description contains the word (here DAYS_SINCE_LAST_TXN, "most recent transaction") instead of only saying it was not found. The vector index only orders them: on its own it ranked spend features first. The agent does not switch to one by itself.

Changed: smaller items

  • Statistics: the first statistics collection on a new feature table no longer logs a warning (Error 3624 "no statistics defined" is expected there). Other failures show the whole message.
  • Documentation: upload_features(document=True) no longer has the LLM document the same process a second time while documenting its dataset.
  • Data domain: select_data_domain logs a warning, not an error, before raising for a domain that does not exist yet.

[0.3.5.5] - 2026-10-28

Added: attached objects in the admin app

  • New panel, Documentation › Attached objects:
  • the list of attached views and tables, with Sample and Detach on each row;
  • Check for changes and Apply the changes (refresh_attached_objects);
  • Attach, a preview first, then the write. Each object is attached on its own, so one that cannot be attached shows its reason without stopping the others;
  • Suggestions: the views of the given databases that read a dataset or a process view.
  • Documentation › Objects: the detail of a view or source table has Attach to the feature store… or Detach from the feature store.
  • Explore › Dataset builder: next to an entity's datasets, the views attached to them are listed, each with a sample.

Added: a sample is read only after its EXPLAIN is checked

  • Why: SAMPLE applies to a view's result, so the whole view is computed.
  • What is checked: the panel reads the EXPLAIN of SELECT * FROM <view> SAMPLE 20 first, with the new pure parser tdfs4ds.utils.explain_cost. It reads the per-step estimated times, the intermediate result sizes, the *** overflow, the product joins and the confidence.
  • Above the limits: an estimate above 60 s (tdfs4ds.SAMPLE_MAX_SECONDS) or above 10 GB of intermediate result (tdfs4ds.SAMPLE_MAX_SPOOL_GB).
  • A confident estimate is refused: the plan is shown, and Sample anyway runs it.
  • A low- or no-confidence estimate (statistics missing) only warns, and the sample runs.
  • Cancellable runs: a sample above the limits runs on its own database connection, with its elapsed time and a Cancel button. Cancel stops the query in the database (connection.cancel(), Error 3110; verified live: the session was idle on the AMPs right after). A sample still running after 10 minutes is cancelled (tdfs4ds.SAMPLE_JOB_MAX_SECONDS). Without the TDFS4DS_TD_* settings, a sample above the limits is refused, and Sample anyway reads it on the app's connection, which cannot be cancelled.
  • Estimates can be far off without statistics: on efs-dev, a dataset view of 9 categories was estimated at millions of hours and read in 2.3 s.
  • What can be sampled: only an attached object or a suggested view.

Changed: column names of several versions side by side

  • A feature read under several process versions now gives columns named after the bare process view: amount_FEAT_V1. They used to be amount__DB___FEAT_V1_, because the catalog stores the view name qualified and quoted.
  • This matches the names the admin app's dataset builder already gave.
  • Existing dataset views keep their columns until they are rebuilt with build_dataset. rebuild_dataset and migration keep the names recorded in the view.

[0.3.5.4] - 2026-10-28

Changed: datasets from feature names alone

  • build_dataset and build_historical_dataset accept the feature names alone: selected_features=['amount', 'nb_tx']. A feature computed by one process is resolved to that version.
  • When to give a version: only for a feature that several processes compute. Write it in a dict with None for the others: {'amount': '<process uuid>', 'nb_tx': None}. {'amount': {'alias': 'spend'}} names the only version.
  • Errors: such a feature named without a version raises tdfs4ds.FeatureSelectionError (a ValueError). The message lists the processes with their ids and gives the dict to paste, for one version or for all of them side by side. A name not registered for the entity in the current data domain, or a feature no process computes, raises the same error. All the problems are reported at once.
  • A full dict ({feature: process uuid}, the earlier form) gives the same view as before.
  • A version that no current process has is now logged as a warning. The values of a removed process may still be stored, so the build goes on.
  • One catalog query: the selection is resolved with one query whatever the number of features. A build used to read the catalog two or three times.
  • Verified live on efs-dev, on a single-version feature set, a feature with three versions, a history and lags.

[0.3.5.3] - 2026-10-28

Fixed: valid_time_column with a PERIOD feature

A process with valid_time_column and a PERIOD feature (PERIOD(TIMESTAMP(0) [WITH TIME ZONE])) failed at every run with Error 6976 "Period data types are not allowed with Window function": the history write computes each period with window functions, and Teradata refuses a PERIOD value in such a query. The staged column types are now read once (TYPE() on one staged row), and a PERIOD feature goes through the same window queries on its BEGIN / END bounds, the value being rebuilt in the outermost SELECT. Same periods and the same change rule (equal consecutive values merged, NULL-safe). Other types issue exactly the statements they did.

[0.3.5.2] - 2026-10-28

Views and tables attached to the feature store

A registered dataset joined with a source table in a user view that no process reads, or a table brought in by hand, was "outside the feature store" for the agent, and a full rebuild_lineage_graph() closed the edges an extra_views= call had recorded. attach_objects declares such objects part of the feature store.

  • API (exported on tdfs4ds): attach_objects(objects, data_domain=None, *, document=True, refresh_lineage=True, refresh_index=True, author=None, dry_run=False), detach_objects(objects, dry_run=False), attached_objects(data_domain=None) (DATABASE_NAME, OBJECT_NAME, KIND, DATA_DOMAIN, DOCUMENTED_AT), refresh_attached_objects(objects=None, *, dry_run=True, document=True, refresh_index=True, author=None) and suggest_attachments(databases, *, include_processes=True) (VIEW, READS, ATTACHED; read only, never attaches). A process view, a dataset, an FS_* / *_HIDDEN object or a name not found in the database is refused.
  • Marker. The reserved business-dictionary section ATTACHED (JSON: kind, optional data domain, fingerprint, documentation time). No new table; every section reader skips it (RESERVED_SECTION_NAMES). The reads and detach_objects live in tdfs4ds/attached_registry.py, which imports nothing from genai, so lineage, dependencies and the workflow assessment can use it.
  • Lineage. The edges of an attached view are recorded at attach, and every full rebuild_lineage_graph() seeds attached views. An attached table that no view reads has no edge until an attached view reads it.
  • Documentation. The pass-through columns of a view (a plain column of one FROM / JOIN relation, renamed or not, t.* too) take the description of the source column (PROPAGATED). Computed columns and the object description are written by the LLM (skipped without an instruct model). A table is documented by its owner (set_object_documentation, dictionary import). IMPORTED and MANUAL parts are never overwritten.
  • Refresh, also run by rebuild_documentation. The fingerprint is the normalised view text plus one hash per column (select item and source), or the column types of a table. dry_run=True by default. States:
  • unchanged: skipped, no write, no LLM.
  • sources_changed: a source's documentation changed after the last pass; pass-through columns are copied again, no LLM.
  • changed: the view text or columns changed; added and changed columns are documented, the rows of removed columns are ended, the LLM runs only when a computed column or the sources changed, lineage is recorded again.
  • broken: the object exists and its EXPLAIN fails; reported, nothing written.
  • disappeared: the object is gone; marker ended, lineage closed, documentation ended (readable at a past date), a person's parts kept and reported, a row written to the removal log.
  • Agent. Attached views and tables are inside the feature store (kinds "attached view" / "attached table"). A fixed EN/FR sentence says "Attached to the feature store: X is an attached view ... It reads DS (dataset), T". A dataset or process view read by attached views is described as "used by the attached views ...". Search answers show attached objects in their own block before the outside block.
  • Guards. Removing a dataset or a process read by an attached view is refused (DependentObjectsError names it). The removal plan has a detach_object step; the view itself is never dropped. The network check of the workflow assessment counts attached views as consumers: they are not reported as detached, and the processes they read are not reported as unused.
  • Guarded by tests/test_attached_objects.py. Live-verified on efs-dev: a view joining a registered dataset with a documented table attached with the table; pass-through columns documented from their sources (PROPAGATED), lineage edges recorded, the agent's sentences in both directions, the removal of the dataset refused with a detach step, refresh states unchanged / sources_changed / changed / broken / disappeared, suggestions, detach. Not exercised live: the LLM part (no model configured) and a full rebuild_lineage_graph().

[0.3.5.1] - 2026-10-28

Types of features

The text length of a feature was fixed by the first run that registered it, a re-registration with another type was ignored silently (values cast or truncated), and a stored type could not be changed. This release makes the length a process setting, compares types by whether the values fit, and adds the migration of a stored type with its history.

  • force_varchar_length is a process setting. Recorded in FS_PROCESS_SETTINGS (FORCE_VARCHAR_LENGTH) at registration. upload_features(force_varchar_length=None) (the new default) keeps the value recorded for the view's process, else 1024, so a re-registration never resets what set_process_options recorded; 0 or 'none' records exact lengths. run() / roll_out() take force_varchar_length=None, meaning "as recorded" (1024 when nothing is recorded, i.e. a process registered before 0.3.5.1; before, run() applied no rounding, so a feature first registered by a run got an exact-length table). set_process_options(pid, force_varchar_length=N) or 'none' changes it. get_process_info(pid)['FORCE_VARCHAR_LENGTH'] shows it. Migration carries it over.
  • One VARCHAR type per process and character set. A new text feature takes the stored type of a registered text feature of the process that holds it (it joins the process's text table); otherwise the new text features share VARCHAR(ceil(longest / N) * N), the longest taken over the new features only. LATIN and UNICODE stay apart, so a process has at most two text tables, and never more tables than before (with 1024, texts up to 1024 land in the usual VARCHAR(1024) table). Registered features never change type here, and each one is checked against its own source type (a short column is never reported because another one grew).
  • No empty storage tables. Registered features keep their catalog type when the storage tables are ensured. A run used to create an empty exact-length table (seen live).
  • A changed type of an existing feature is compared by whether the values fit, not by equality. Fitting: a shorter VARCHAR of the same character set or into UNICODE, an integer into a wider integer, FLOAT or a large enough DECIMAL, a smaller DECIMAL, a lower TIMESTAMP precision. A fitting type changes nothing.
  • A type that does not fit is refused at registration. upload_features registers nothing and returns None. The log explains three alternatives, with code: (1) CAST the column in this view to the stored type (the select item is quoted when found); (2) widen the stored type for every process computing the feature: tdfs4ds.plan_type_change(types={...}, entity=[...], data_domain=...), then write_relocation_script, dry run, --apply (the other processes whose type would not fit the new one get their CAST snippet; the others are listed as "no change needed"); (3) incompatible types (text vs number, ...): give the column another feature name. A migration only widens to the narrowest type holding every producer's type, so processes do not undo each other. run() / roll_out() log the same text as a WARNING and write with the stored type.
  • Truncation warning. When a registered VARCHAR feature's source is longer than its stored type, each batch reads the longest staged value (one aggregate) and warns: it fits, or N values are too long (truncated in Teradata mode, refused in ANSI mode). The batch writes and stays COMPLETED.
  • Migration of a type. plan_relocation(..., value_types={feature: type}) (no tag needed; an existing tag is kept), or plan_type_change(process_id) / plan_type_change(types=, entity=, data_domain=). A new FS_T_ table with the new FEATURE_VALUE type (same primary index and partitioning), the full history copied and verified, catalog FEATURE_TABLE / FEATURE_VIEW / FEATURE_TYPE updated, datasets rewritten and checked, statistics, old rows deleted last. value_conversion blocks when the stored values do not convert (a text narrowing is allowed when the longest stored value fits). verify_relocation checks the catalog type.
  • valid_time_column may still be changed freely: no table definition is involved.
  • Compatibility. A process registered before 0.3.5.1 keeps 1024. Behaviour change: a re-registration with a type that no longer fits is refused instead of silently cast or truncated.
  • Guarded by tests/test_type_changes.py. Live-verified on efs-dev: one text table per process and character set (LATIN and UNICODE apart, a non-Latin value round-trips), the table name unchanged from earlier versions, no empty table after run(), a grown column refused alone with the alternatives, the run() and truncation warnings (a 1500-character value cut to 1024), a VARCHAR migration with its history and a dataset, INTEGER to FLOAT, an incompatible change, a shared feature without ping-pong, the narrowing block, a recorded setting kept by a re-registration, exact lengths with 0.

[0.3.5.0] - 2026-10-28

Valid time read from a column of the source (valid_time_column)

A source that already holds a history (one row per entity and valid time) needed one run per date: a TimeManager loaded from the distinct dates, then roll_out. A process can now read the valid time from the data.

  • Option. upload_features(..., valid_time_column='EVENT_TS') (last parameter, default None). Recorded as the process setting VALID_TIME_COLUMN in FS_PROCESS_SETTINGS (no DDL). Change it with set_process_options(pid, valid_time_column='COL'), switch it off with valid_time_column=False. Read it with get_process_info(pid)['VALID_TIME_COLUMN']. Migration carries it over.
  • Meaning. The column is a DATE or TIMESTAMP of the source (neither an entity column nor a feature). A row's value is valid from that time until the entity's next row (last row: 9999-01-01). One run writes a whole history, and the source may hold several rows per entity. The unique key is (entity, column): a NULL valid time or a duplicate (entity, time) refuses the batch, and the error names the column.
  • Rules. Equal consecutive values are merged (NULL-safe). Each entity's history is replaced from its earliest valid time in the load (t0) onward and kept before it. A re-run is a no-op, an incremental load rewrites only from its own t0, and a backdated correction works with a full load. An incremental load can leave two adjacent periods with the same value at its start; a later full load merges them, and history datasets NORMALIZE them.
  • Valid time of the run. FEATURE_STORE_TIME and the TimeManager date no longer set the values' valid time (a TimeManager can still bound the rows the view returns). The follow-up row is unchanged (no FS_FOLLOW_UP column).
  • With the other options. close_absent_entities: an entity absent from a load has no value from the load's earliest valid time on. track_entity_refresh records the load's latest valid time. The process SQL version is captured at the load's earliest valid time.
  • Types. DATE becomes midnight. TIMESTAMP(6) is truncated to the second, through its text (a CAST from TIMESTAMP(6) to TIMESTAMP(0) is Error 7454). Without a time zone, the session time zone applies.
  • Refusals. Not for a UAF ART view. Registration returns None and registers nothing for an entity column, a feature, a missing column, a non-date type, or a store without FS_PROCESS_SETTINGS. The read is fail-closed: if the settings table exists but cannot be read, run() raises (a history load stamped with the run time would be wrong).
  • Write path. Per storage table, one multi-statement request (one transaction): a NONSEQUENCED DELETE of the rows starting at or after each entity's t0 (joined to the volatile table TDFS_VT_KEYS), a NONSEQUENCED UPDATE trimming the row that spans t0, and one NONSEQUENCED INSERT per feature of the computed periods. Code in feature_store/valid_time_column.py.
  • Compatibility. Opt-in. Without it every statement is unchanged. A client older than 0.3.5.0 running such a process ignores the setting: several rows per entity are refused at staging as duplicates, one row per entity is stamped with the run time. The earlier route (a TimeManager from the distinct dates and roll_out) is still valid.
  • Guarded by tests/test_valid_time_column.py (55 offline tests). Verified on Vantage 20 (efs-dev): first load equal to an independent reference, idempotent re-run, incremental load, backdated correction, close_absent_entities, track_entity_refresh, build_dataset(as_of=) and build_historical_dataset, the refusals, DATE and TIMESTAMP(6), FilterManager segments, and a control process without the option.

[0.3.4.7] - 2026-10-27

Lineage stops behind exposing views are recognised; one grouped warning

Field log on 0.3.4.6: a process reading ten tables through views that expose them still warned about all ten, each with the full Teradata message and its own GRANT advice.

  • Exposing views recognised in the lineage. The lineage graph is built without view texts, so 0.3.4.6 had nothing to check and recognised no exposing view. record_process_lineage now reads, with SHOW VIEW, the views right above the objects it could not read (only when there are such objects, each view once). Tables read through a view that exposes them give one info line and no warning.
  • One warning for the rest. Objects refused with the same message are listed once with the message once, and the GRANT advice is grouped per database, with a note that no grant is needed when they are read through views that expose them.
  • Guarded by tests/test_exposed_views.py (a graph shaped like the builder's, without view texts, and ten stops).

The agent compares documented columns of tables and views

Field report: "is the MCE_NUMBER of HCF_ACCESS.HCF_FIBER_VW similar to the MACHINE_NR of the Historian?" answered "No documentation found" for both columns, although MCE_NUMBER alone had been answered the turn before.

  • Several named items were looked up in the feature catalog only. Names that are not features are now looked up as documented columns (one read for all), before the "did you mean" correction so a column is never turned into a similarly named feature. The owner named in the question wins (a qualified name, or a word of it: "the Historian"), then the table remembered from an earlier turn, else every owner; owners repeating the same text (a view exposing its table) are shown once. Each column gets the "Outside the feature store" line, as a single column does.
  • Join evidence. When a process reading both tables joins the two columns (f.MCE_NUMBER = h.MACHINE_NR), the answer gives it as the deciding fact; without it, the documented descriptions are compared and anything beyond is labelled an interpretation.
  • Routing. The classifier sent such a question to LINEAGE with one name (seen live) or to DEFINITION. A comparison shape (EN/FR) naming two or more documented items is now routed by code to DEFINITION with all the names, never split into a compound question.
  • The trace shows a documented column as "documented column of DB.TABLE" instead of "not found".
  • Guarded by tests/test_definition_multi_columns.py. Verified on Vantage 20 through the agent with a live LLM: the comparison was routed to DEFINITION, both owners were found and the process join was given as the answer.

[0.3.4.6] - 2026-10-27

A view that exposes a table takes the table's documentation

Field report: users read a historian table through a LOCKING ROW FOR ACCESS view in another database, and have no right on the table. Its documentation (layout, tags, units) was on the table, and the LLM pass that documented the view kept only the physical columns, so the tags never reached the processes reading the view (verified live: units and instruments were missing from the feature descriptions).

  • Exposed views. A view exposes a table when every output column is a column of that one table, unchanged or renamed. Row restrictions do not stop it: WHERE (a filter on USER), joins to an entitlement table, QUALIFY, DISTINCT, TOP, SAMPLE mark it filtered. Aggregation, GROUP BY, set operations and columns from two tables do not qualify. The mapping is read from the view's SHOW VIEW text: no right on the table is needed.
  • Documentation copied, no LLM call. The table's description, sections, columns (through the renames), historian layout and tags are written on the view as PROPAGATED; a part a person wrote on the view is kept. The pair is the reserved business-dictionary section EXPOSES of the view (no new table).
  • Automatic in the documentation walk (process, dataset, rebuild_documentation) for a view over a documented table with no computed column, including a view already documented by the LLM. tdfs4ds.EXPOSED_VIEW_AUTO_ATTACH = False turns it off. Editing the table with set_object_documentation / set_deep_table_documentation copies the new text onto its views and lists the processes to re-document.
  • API: attach_exposed_view, detach_exposed_view, exposed_views, refresh_exposed_views, suggest_exposed_views (read-only).
  • Lineage. A lineage stop at a table read through a view that exposes it is an info line, without the GRANT advice. The entitlement tables of such views are not reported as undocumented.

Fixes

  • The lineage-stop warning printed the driver's stack trace (about 20 Go frames). It now keeps the message only.
  • "does not have any access" gave the advice GRANT ANY ON ...; it is now GRANT SELECT.
  • "N intermediate view(s) could not be documented (LLM failure)" was logged for views that had been documented: the function returned nothing on success. It now returns True / False.
  • Entity columns of an intermediate view (e.g. the drum id of a historian view) were documented by the LLM and not stored.
  • Guarded by tests/test_exposed_views.py and tests/test_lineage_access.py. Verified on Vantage 20 with a filtered LOCKING ROW FOR ACCESS view joined to an entitlement table on USER.

[0.3.4.5] - 2026-10-26

A missing right is reported as "not verified", never as a broken view or a silent end of the lineage

  • Where the lineage stops. The lineage builder could not read the definition of a view the user has no right on, stopped there, and stored nothing to say so (only edges are kept): everything below was missing from lineage, impact analysis and the checks. record_process_lineage now logs a warning naming each such object and the GRANT that would lift it, and the builder keeps the refusal instead of the unrelated SHOW TABLE error that followed it.
  • Workflow assessment, lineage network. A view of the lineage graph with nothing recorded below it is probed (SHOW VIEW): lineage_stops_no_access (definition refused, with the GRANT), lineage_not_recorded (readable, its inputs missing from the graph: rebuild its lineage) or lineage_stops_unreadable. Each names the processes and datasets whose lineage ends there. Objects absent from DBC.TablesV are probed too: that view only lists objects the user has a right on.
  • EXPLAIN refused for a right. The network check reports view_not_verifiable (warning) instead of "does not compile"; the integrity check process_reads_inaccessible_object (warning) instead of "no longer compiles"; check_downstream_processes returns them under not_verifiable instead of broken.
  • Removals still refuse when a dependency cannot be read.
  • Guarded by tests/test_lineage_access.py. Verified on Vantage 20: an EXPLAIN refused with Error 5315 is classified as not verified.

[0.3.4.4] - 2026-10-26

Dataset documentation finds the descriptions of its process features

Field report: documenting a dataset over two documented processes logged "feature X not found in process P documentation" for every feature, and the process views were documented again as plain intermediate views.

  • Process views recognised. The documentation walk matched the lineage graph against the process catalog with the view name as stored, quotes included ("DB"."VIEW", the name of a tdml.in_schema DataFrame). Names are now compared unquoted, so a process view is documented as its process.
  • Feature descriptions filed under their features. A first upload_features documents the process before its run registers the features, so the feature descriptions could not be filed under their FEATURE_IDs and none was written (no message); they were stored on the process view's columns in the business dictionary only. upload_features now links them after the run, and tdfs4ds.link_feature_documentation(process_id) does it for any process (no LLM; existing descriptions are never changed).
  • Existing stores work at once. Dataset documentation falls back to the business-dictionary description of the process view's column when a feature has no description row, so re-running build_dataset(..., document=True) or document_dataset_incremental is enough.
  • Guarded by tests/test_dataset_doc_propagation.py. Verified on Vantage 20 with a quoted process view name.

[0.3.4.3] - 2026-10-26

A storage definition the database refuses stops upload_features before anything is registered

Field report: partitioning= with RANGE_N on a DECIMAL entity column (Error 3732) was only refused when the first storage table was created, after the process was registered. The error was logged and skipped, the features were registered anyway, and every later run failed with "Object 'FS_T_...' does not exist", even once the partitioning was fixed.

  • Checked first. upload_features tries the entity columns, primary index and partitioning on a throwaway volatile table before registering anything (tdfs4ds.feature_store.storage_definition). A refusal is logged with the whole database message and what to change (RANGE_N on a DECIMAL column: cast it to INTEGER in the view; a TIMESTAMP column needs TIMESTAMP bounds; only entity columns), the view created from sql_query is dropped, and the call returns None.
  • No table, no registration. A storage table that cannot be created now raises instead of being logged and skipped, so its features are never registered against a table that does not exist. run() tries the definition first too.
  • Missing tables are created again. Each run creates the storage tables that are missing for every feature of the process, not only for new features (one DBC read), so a store left in the state above repairs itself on the next run.
  • A changed definition is no longer ignored silently. Registering a view again keeps its recorded primary index and partitioning (its tables were created with them); passing other values now logs a warning naming plan_relocation(...).
  • Entity key types keep their precision. DECIMAL, NUMBER and TIMESTAMP keys are created with their exact type (DECIMAL(18,0), TIMESTAMP(0) WITH TIME ZONE, ...); the bare names meant DECIMAL(5,0) (a key above 99999 overflowed, decimals were lost) and TIMESTAMP(6) without time zone. Existing tables keep their columns; table names do not depend on the types.
  • Guarded by tests/test_storage_definition.py. Verified on Vantage 20: both refusals reported, nothing registered; a dropped storage table re-created by the next run (20 rows, COMPLETED); the re-registration warning.

[0.3.4.2] - 2026-10-26

Log lines coloured by level, on stdout, root logger left alone

import tdfs4ds called logging.basicConfig on the root logger. Every line went to stderr, which Jupyter shades pink whatever the level, so an INFO looked like an ERROR; and basicConfig does nothing once anything else has configured the root logger, so in some kernels the lines came out plain instead. The import order decided which, and the level never did. It also switched on INFO for every other library (httpx's HTTP Request lines).

  • The tdfs4ds logger now has its own handler (tdfs4ds/log_output.py): stdout, the same line format, and the level as a colour: DEBUG grey, INFO plain, WARNING orange, ERROR and CRITICAL red. The root logger is no longer touched, so other libraries' INFO lines no longer appear.
  • tdfs4ds.LOG_COLOR (TDFS4DS_LOG_COLOR): None / auto (default) = colours in a notebook or a terminal that shows them, never in a file or a pipe, never when NO_COLOR is set; True / False to force.
  • tdfs4ds.LOG_PROPAGATE (TDFS4DS_LOG_PROPAGATE, default False): tdfs4ds lines are no longer passed to the root logger, so they never print twice. If you collect tdfs4ds logs through your own root logging (a file, a collector), set it to True or call tdfs4ds.configure_logging(propagate=True): tdfs4ds then prints nothing and the lines go to your handlers, as before. python -m tdfs4ds.serve does this, so the server log keeps one format.
  • DISPLAY_LOGS and the admin app's live log are unchanged.
  • Guarded by tests/test_log_output.py.

[0.3.4.1] - 2026-10-26

Fix: a dataset following a TimeManager built on a timestamp column could not be read (Error 7454)

build_dataset(time_manager=tm) and the dataset_time_manager option of upload_features / run() filter every feature read with the manager's business date cast to TIMESTAMP(0) WITH TIME ZONE. A TimeManager loaded from a timestamp column stores TIMESTAMP(6) WITH TIME ZONE, and Teradata refuses CAST(TIMESTAMP(6) AS TIMESTAMP(0)) even with zero fractions (Error 7454, DateTime field overflow): the view was created, every read failed, the dataset was never registered.

  • The manager scalar is now cast up: CAST(BUSINESS_DATE AS TIMESTAMP(6) WITH TIME ZONE) (constant tdfs4ds.dataset.temporal.TM_SCALAR_TYPE), in the period predicate and in the lag/lead instant. A DATE manager casts up as well, and the TIMESTAMP(0) valid-period columns compare against a TIMESTAMP(6) value unchanged. Current and AS OF datasets are byte-identical. Views written with the old text still parse (the parser accepts any precision); recreate them (rebuild_dataset(..., replace=True)) to make them readable.
  • build_dataset: the warning "could not apply the known entity/feature split" and the "Dataset build failed" errors of run() / upload_features give the whole database message (describe_db_error), not its first line.
  • run(): after an ingestion that ran but did not create the dataset, the log no longer says ingestion was skipped.
  • Guarded by tests/test_tm_business_date_cast.py.

[0.3.4.0] - 2026-10-26

The dataset created by upload_features follows the process's TimeManager (dataset_time_manager)

The dataset that upload_features / run() builds from dataset_view_name= was always CURRENT VALIDTIME, even for a process driven by a TimeManager.

  • upload_features(..., dataset_time_manager=False) (last in the signature), run(..., dataset_time_manager=None), roll_out(..., dataset_time_manager=None) and set_process_options(pid, dataset_time_manager=): a process setting (DATASET_TIME_MANAGER in FS_PROCESS_SETTINGS, named-column write, never in METADATA; None = as recorded). When on and the process has a TimeManager (time_manager=, else the recorded one), the dataset is built with build_dataset(time_manager=...) (grouped or not): tm.update(i) moves it, no re-run.
  • On without a TimeManager: one warning, current dataset as before. A dataset that already exists at CURRENT VALIDTIME (or in another mode) is not rebuilt silently: a warning names tdfs4ds.rebuild_dataset(<name>, time_manager=..., replace=True); one already on that manager is left as it is. Migration carries the setting.
  • Off (default): the build_dataset call and SQL are unchanged. Guarded by tests/test_upload_dataset_time_manager.py.

Track when each entity was last confirmed (track_entity_refresh)

The ingestion MERGE writes nothing when a value is unchanged, so a feature row's ValidStart is the time of the last change, not of the last confirmation, and FS_FOLLOW_UP (per run and segment) cannot tell an entity the source dropped from one a run recomputed.

  • upload_features(..., track_entity_refresh=False) (last in the signature), run(..., track_entity_refresh=None), roll_out(..., track_entity_refresh=None) and set_process_options(pid, track_entity_refresh=): a process option (None = as recorded). When on, each batch that staged rows records, right after the values and before the follow-up closes, that it produced each of its entities, in an admin table FS_R_*: entity columns, PROCESS_ID, REFRESH_TIME (the run's valid time), LOAD_TIME, RUN_ID, valid-time period, primary index = the process's. One FS_R_* table per (data domain, entity key, primary index, placement tag), named by refresh_table_name(...) = the FS_T_ naming rule plus the placement tag, so processes in tables of their own never share one. Created lazily. Not a feature: in no catalog or FS_V_* view; the EFS map lists it as Entity refresh (admin). Fail-open: a failed write is a warning with the full database message and the run stays COMPLETED.
  • New optional table FS_PROCESS_SETTINGS (valid time, key/value PROCESS_ID, SETTING_NAME, SETTING_VALUE, every INSERT names its columns; created by setup() / connect()). FS_PROCESS_OPTIONS is not altered: older clients write it with a column-less positional INSERT, so a new column would break their close_absent_entities=True with Error 3812. read_process_options merges both tables and works when either is missing; later options need no DDL.
  • Readers: tdfs4ds.entity_refresh(process_id, entities=, as_of=, history=) and tdfs4ds.entity_refresh_summary(process_id, as_of=) (entities by last refresh; those the latest run did not confirm; not tracked (older client or failed write) when COMPLETED runs left no refresh row).
  • The parallel planner adds the FS_R_* table to the write set of a tracked process; remove_process deletes its refresh and settings rows and footprint counts them; migration copies the setting, not the tables; get_process_info reports TRACK_ENTITY_REFRESH.
  • Off by default and byte-identical: the ingestion statements with the option off are asserted against SQL hashes captured before the change. No ALTER of any existing table.

Relocate features to their own storage table

Features of one (data domain, entity, value type, primary index, partitioning) share one FS_T_* table, so the parallel planner serialises their processes (one MERGE lock per table), and a table cannot be reorganised for one process. A feature can now have a table of its own: FS_T_<hash>__<TAG>, with its FS_V_ view.

  • Placement: optional table FS_FEATURE_PLACEMENT (valid time, one row per (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME) with a TABLE_TAG), created by setup() / connect() like FS_PROCESS_OPTIONS, and followed by the naming rule: get_feature_store_table_name(..., table_tag=) appends __<TAG>; register_entity creates the tagged tables; register_features keeps the tag (also across a type change) and warns when the catalog was pointed back at the old table. Nothing used, nothing changed: without a tag or without the table every name and statement is byte-identical (tests compare against SQL hashes captured from the previous version); one cached existence check per store. No ALTER of any existing table.
  • upload_features(..., storage_tag='SALES') starts a process in its own table (refuses a feature already registered elsewhere or shared, with the fix).
  • plan_relocation(features=, process_id=, table_tag=, include_shared=, primary_index=, partitioning=) (read only), relocate_features(plan, dry_run=True), verify_relocation(plan), generate_relocation_script / write_relocation_script (dry run unless --apply): create the table (a clone of the source: same primary index and partitioning), copy the rows with their history, check the counts per (FEATURE_ID, FEATURE_VERSION), write the placement and point the catalog at the new table, rewrite the datasets (EXPLAIN and equal row count, the view is put back if it differs), lineage, statistics, clear caches, delete the old rows last. The plan states rows, perm, free perm (a copy that does not fit is blocking), whether the copy redistributes, and refuses a shared feature, a producer in RUNNING, an unparseable dataset and a view reading the old per-table view. A new primary index / partitioning moves the whole process and updates FOR_PRIMARY_INDEX / FOR_DATA_PARTITIONING.
  • rewrite_dataset_tables(dataset, mapping=None, dry_run=True): rebuilds a dataset view (snapshot, grouped, history, lags and leads) so that some features read another table, from its own specification; name, DATASET_ID and columns are kept; only the features concerned move. It is the repair of dataset_reads_stale_table.
  • Workflow assessment: new storage-level finding relocation_reverted (rows of a relocated feature in the old table, or a catalog row pointing back at it, left by a client older than this version that registered the feature again); running the relocation plan again moves them with a sequenced merge. The stale-table fix names rewrite_dataset_tables; the app links findings to a plan, a dry run and the script (it never executes one).
  • Precondition: every client that runs a relocated process must be at least this version. An older client running a registered feature keeps writing to the tagged table (ingestion reads the catalog); it points the catalog back only when it registers the feature again (type change, or removed and registered anew).
  • Migration does not copy placements (the physical layout is a decision of each store).
  • Fixed while testing live: the catalog update of a relocation is nonsequenced, in place. A sequenced update leaves the old table in a closed version of the row, which FS_V_FEATURE_CATALOG returns to a session whose temporal qualifier is ANSIQUALIFIER; the next run then wrote to both tables.

Workflow assessment: do the dataset features exist?

A dataset view whose feature is gone still compiles (EXPLAIN passes), and its feature reads return nothing: an INNER dataset is empty, a FULL OUTER column is all NULL. The assessment now checks that the features are there.

  • Dictionary level, with the cheap integrity check: dataset_reads_stale_table (the catalog stores the feature in another table than the view reads; a type change moves a feature on the next run, with no history copy) and dataset_reads_missing_table (the table no longer exists). A feature missing from the catalog stays dataset_reads_removed_feature.
  • Storage level, on demand, new check dataset_rows (run by the Scan entity keys button and by assess_workflow; dataset_rows=False skips it): dataset_feature_no_rows when the table holds no row for the feature's (FEATURE_ID, FEATURE_VERSION) at the valid time the dataset reads. INNER dataset = error, FULL OUTER = warning; the follow-up says whether the process never completed or the rows are gone. The report's Lineage section says whether the storage level ran, and when.
  • dependencies.dataset_table_reads(ddl) returns the storage table and join type of each read (dataset_reads is unchanged); scan_storage_orphans(..., by_version=True) counts per FEATURE_ID and FEATURE_VERSION (default output unchanged).

[0.3.3.29] - 2026-10-23

Dataset builder: the Python that creates the dataset

  • Show Python, next to Show SQL, gives the tdfs4ds.build_dataset / build_historical_dataset call that Create dataset makes for the same form, preceded by the connection and the data domain, with one explanation per argument (grain, versions and aliases, join type, valid time or window, lags and leads with the resulting column names). A Copy button copies it. Nothing is run.
  • No password in the code: the host and user are filled in from the app's connection; the code asks for the password with getpass.
  • Same call, checked: a test runs the generated code and Create against the same stub and compares the arguments, for every kind of dataset and both join types. Live on efs-dev, the code run as written created a view whose SQL equals Show SQL.

Workflow assessment: the lineage network

The lineage check only compiled the processes reading an object missing from the data dictionary, and said nothing of the network's shape; the dead branch of a gap appeared only inside a finding's text. A new check, network (Lineage dimension, run by default):

  • Every view is compiled. EXPLAIN on every process view, dataset view and view the lineage graph holds. Only the root failures are reported (a view whose own inputs compile); the views failing because of it are listed under it, with the processes and datasets it stops. An error the database did not give (no session) is listed under Not verified, never as a broken view.
  • Is the network in one piece? The connected networks with their processes, datasets, views and tables; lineage_split (info) when there are several, lineage_isolated for a process or dataset with no lineage link. Names of the graph that are not database objects (SQL tokens) are left out, so they cannot join two networks.
  • Dead branches. process_not_read (a process no dataset or process reads), detached_branch (views and tables no registered process or dataset reaches), and the branches the lineage gaps leave.
  • Shown in the report. The Lineage section always carries a Lineage network block (HTML and Markdown), also when nothing is found. The gap check reuses the compilation instead of running its own EXPLAINs.
  • Live on efs-dev: 29 views compiled in an 18 s assessment; a dropped table under two views was reported once, at the first view, with the second failing through it; 10 historical datasets built without their lineage showed as isolated networks.

Skills

  • fs-workflow-assessment describes the network check and its five finding kinds.
  • fs-dataset describes the admin app's Dataset builder (lags / leads and column-name fields, Show SQL, Sample, Show Python, Create). fs-agent describes an app started with no feature store selected (Error 3707 before 0.3.3.28, and what 0.3.3.28 changed).

[0.3.3.28] - 2026-10-23

Lags and leads in datasets: feature_offsets

  • build_dataset(..., feature_offsets=) / build_historical_dataset(..., feature_offsets=) read a feature at other instants than the dataset's own, as many times as needed. For example {'amount': ['0', '-1 day', '-7 days'], 'churn': ['+1 month']} gives amount, amount_LAG_1D, amount_LAG_7D and churn_LEAD_1MO.
  • Sign: - is a lag (the past: autoregressive inputs), + a lead (the future: a forecast target), 0 the instant. A value without a sign is refused.
  • Modes: every dataset mode: current, as_of, a TimeManager (tm.update() moves every offset column), and histories.
  • Units: year, month, week, day, hour, minute, second, and ISO 8601 ('-P1D'). Months are calendar months (ADD_MONTHS). Fractions of a second are refused, since valid periods are stored to the second.
  • A lead is NULL until its instant is reached (database time): an open stored period is not an observation of the future. A current snapshot cannot have a lead: refused with join_type='INNER', a warning with 'FULL OUTER'. The cut-off is real time: on a store whose business dates are in the past, a lead after the last business date reads the last stored value, so end a training window one horizon before it (documented, and done in demo 06).
  • Latency:
  • offset_strategy='PIVOT' (default) reads each storage table once for all its offset columns. Measured on Vantage 20 with 8 features × 4 offsets: parse 0.35 s and a 23 ms single-entity lookup, as fast as 8 plain grouped columns.
  • 'PER_OFFSET', one sub-query per column, parsed in 31 s on the same columns.
  • Partition elimination is kept: the shift is applied to the constant side (ValidStart <= t - L), so every storage read keeps it (EXPLAIN steps read in every mode, single-entity lookups single-AMP).
  • Catalog and documentation:
  • Each offset column is a row of the dataset's feature list.
  • The new optional table FS_<catalog>_OFFSETS (tdfs4ds.get_dataset_offsets()) records its base feature, direction, offset and strategy.
  • rebuild_dataset, restore_dataset_catalog and migration keep the offsets.
  • Dataset documentation gives each offset column its feature's description, with a fixed sentence on the instant it reads (no LLM).
  • Admin app, Dataset builder: a Lags / leads field per chosen feature, with presets, live column names and the lead check (POST /ui/dsbuild/columns).
  • Compatibility:
  • Without feature_offsets every statement is unchanged.
  • Earlier versions cannot read the offsets back from a view: rebuild or restore datasets with offsets using 0.3.3.28 or later.

Column names for feature versions (aliases)

  • selected_features accepts an alias on a version:
  • {feature: {'process_id': pid, 'alias': 'spend'}} for one version;
  • {feature: [{'process_id': p1, 'alias': 'amount_old'}, {'process_id': p2, 'alias': 'amount_new'}]} for several versions side by side.

The alias is the column name. Offsets are suffixed to it (spend_LAG_1D). - Rules: an invalid alias, or one that repeats another column, is refused, never renamed. Without an alias the names are unchanged. - Kept everywhere: - the dataset catalog registers the alias; - rebuild_dataset, restore and cross-store migration keep it. Migration now rebuilds a dataset from its view with the source catalog's feature names, so every column keeps its name. Before, version names <feature>_<view> were lost. - Documentation: each aliased or version column gets its own version's description, "amount, version computed by FEAT_V2 — …". Before, such columns got none. - Admin app, Dataset builder: a column-name field per chosen version; names are previewed as you type, and an invalid or repeated one is shown in red and blocks the creation.

Demo

  • 06 - Datasets in time gains two sections: lags and leads (a lagged dataset checked against the dataset one month earlier, the offsets in the catalog, a training history for a forecast) and a column name of your own (alias).

Workflow assessment: the lineage check says what it checked

  • The Lineage integrity section used to read "nothing to report" with an empty body. It now says how many processes, datasets, features and dependency links were checked, and lists the gaps it looked for.
  • The panel's description names the lineage check.

Fixes

  • A FULL OUTER dataset with a single sub-query could never be created. The outer select wrote COALESCE(A1.<entity>) with one argument, which Teradata refuses (Error 3706, checked live). This hit a one-feature dataset, or a grouped one whose features all share one table. It now writes COALESCE(A1.<entity>, A1.<entity>).
  • HELP COLUMN cuts column names at 30 characters ("Column Name"; seen live: a 46-character name came back as its first 30). The benchmark's schema probe, the query optimizer's column qualification and the plotting column types read "Column Dictionary Name" instead, which is complete.
  • Admin app connected but with no feature store selected (seen at a site). The .env set the connection (TDFS4DS_TD_*) but not TDFS4DS_SCHEMA, so every panel built its SQL with an empty database name (FROM .FS_FEATURE_CATALOG) and showed Error 3707, a syntax error that names nothing. The Parameters list displayed the only store found as if it were chosen, because a select always shows its first option.
  • The default database is used. A logon with a default database (the Connect page's Default database, or TDFS4DS_TD_DATABASE) that holds a complete feature store now selects it, as import tdfs4ds does after create_context. Another database is left alone.
  • Panels name what is missing. With no store, every panel reading the store answers "No feature store is selected. Choose one in Settings → Parameters, or set TDFS4DS_SCHEMA …" and runs no SQL. The store list and the execution-log switches still work, since they are how a store gets chosen.
  • The startup log warns and the index build is skipped instead of failing.
  • Parameters shows "no store selected" as the first option until one is applied, and Apply stores refuses an empty choice.

[0.3.3.27] - 2026-10-22

Removals refused when other objects depend on them; ordered removal scripts; lineage gaps

  • Behaviour change: remove_process, remove_feature and remove_dataset(..., dry_run=False) now refuse when other objects depend on the object, and raise tdfs4ds.DependentObjectsError. The dependents are:
  • datasets reading the process's values (FEATURE_VERSION) or the feature;
  • process views reading the process view or the dataset;
  • what lies downstream of those.

The error lists them in the order they must be removed. The removal is also refused when they cannot be read (a dataset view that cannot be read, an empty lineage graph). force=True restores the previous behaviour, and leaves the dependents broken.

Why: a process could be removed while datasets still read its values. Those datasets kept returning the last values and were never refreshed, and a lineage rebuild then dropped the link. - New tdfs4ds.plan_removal(objects, purge_values=False, drop_views=False) (dependencies.py): the removal steps for processes, datasets and features plus everything depending on them, in order. - tdfs4ds.write_removal_script(plan) writes a .py script that is a dry run unless started with --apply, and stops at the first failure. - purge_values=True also deletes the removed versions' stored values. - Nothing is executed by the plan. - New FS_REMOVAL_LOG (optional table, created at setup() / connect(); tdfs4ds.get_removal_log()). Every removal and every refusal is recorded: object id and name, user, time, forced or not, dependents, origin (api / script / app), outcome. remove_process checks that its catalog rows are gone before recording COMPLETED, since its decorator prints a failed DELETE instead of raising. - Removals are verified. After --apply, the script checks, step by step, that nothing is left behind, and exits with code 2 otherwise. It looks for: - catalog rows; - business dictionary rows, process documentation and provenance; - stored SQL versions; - vector-index entries: counted by id with Chroma; with the Teradata Vector Store, every collection must be ready after its refresh.

tdfs4ds.dependencies.verify_removal(plan) runs the check on its own; the dry run shows the same counts beforehand, and the app's Check what is left runs it after the script. - Workflow assessment, lineage integrity: new check integrity, area lineage, and a Lineage column in the executive summary. Kinds: - dataset_reads_removed_process (named from the removal log when possible); - dataset_reads_removed_feature and feature_without_process; - process_view_missing and dataset_view_missing; - process_reads_missing_object and process_view_broken, judged by EXPLAIN on suspect processes only; - process_reads_unregistered_dataset and lineage_edge_to_unknown.

Each error names its dead branch and the plan_removal call that removes it. - Admin app: Operations → Remove objects. - You choose processes, datasets and features; the page shows the ordered plan, runs a dry run (read-only counts) and downloads the script. - It never removes anything; a source-level test guards that. - A lineage gap's Remove dead branch… link in the workflow assessment opens it with the branch already chosen.

Dataset builder: features computed by several processes

  • A feature computed by several processes (its versions: FEATURE_VERSION = the producing process) shows one checkbox per process.
  • Each line gives the process view, its SQL version, its last run and the datasets already reading it; the tooltip adds the managers and the last SQL change.
  • The default is the latest run.
  • Ticking several gives one column per version, named <feature>_<process view>. These are the names build_dataset already gives a version list ({feature: [{"process_id", "process_view_name"}, ...]}); the builder's create route used to turn every choice into a single id.

Fixes

  • remove_feature / delete_feature failed with Error 3706 ("expected something between a string ... and the word 'USTOMER'") when entity_id was a list printed as text (e.g. "['customer']"): the text, quotes included, was pasted into the SQL.
  • entity_id is now normalised: a list, a tuple, a dict (its keys), a comma string, or a list / dict printed as text; a type after a name is dropped. Anything else gets a clear ValueError.
  • The values are escaped (sql_literal).
  • tdfs4ds.get_process_info is exported at the top level, as the documented removal recipe uses it (AttributeError before).

[0.3.3.26] - 2026-10-22

Workflow assessment: executive summary, exports, demo

  • Executive summary: the verdict (processes with an error / a warning / information / nothing to report), the most frequent issues, and a health grid: one row per process, grouped by execution wave (the planner's layering), one cell per dimension (entity keys, order and time, inputs in time, runs, process SQL, storage and pruning) with the worst severity, ok, or not checked. Findings carry the processes they concern (pids).
  • Admin app: the data-domain box is a list with a refresh button; exports in HTML (standalone page), Markdown and CSV (/ui/workflow/download?fmt=html|md|csv). tdfs4ds.write_workflow_report(model, out_path=None, fmt='html'|'md'|'csv') writes the same files.
  • Demo: notebook A3 - Assessing the workflow (an entity key with one id too many, the primary index against the source, a downstream process left stale, filters the optimizer can prune on).

[0.3.3.25] - 2026-10-22

Workflow assessment: is the pipeline well built?

  • New tdfs4ds.assess_workflow(processes=None, data_domains=None, checks=None, entity_scan=True, days=90) (workflow_assessment.py). The consistency checks ask whether the catalogs and the storage line up; this one asks whether the workflow is well implemented. Every check is deterministic (counts, the data dictionary, the lineage graph, follow-up rows, stored SQL hashes): no language model is used, and a test guards the imports. Nothing is changed; every finding names the command that fixes it.
  • Entity keys. For each entity of two columns or more, the largest storage table is read once per round (distinct entity tuples, with each key column left out, over the whole history and now):
  • redundant_key_column: the other key columns are unique without this one over the whole history, so it adds nothing to uniqueness and, being in the primary index, prevents a lookup by the real key from using it;
  • key_column_changes_over_time: determined by the others now, but it changed: each change started a new entity and split its history;
  • the fix names one key for both (entity_id=[...]), checked unique now;
  • a dependency observed in the data is not a rule: the finding says how many tuples it covers and asks for the business to confirm. Also: entity_key_superset (one entity key containing another in a domain), entity_column_type_mismatch (an entity column typed in different families across tables: joins convert), storage_skew (DBC.TableSizeV).
  • Order and synchronisation, per direct dependency of the process DAG (lineage), business date by business date: ran_before_upstream, upstream_recomputed_after (the process is stale), completed_on_failed_upstream (failed or COMPLETED,0 rows upstream), upstream_missing_at_date and ahead_of_upstream (same timeline only), mixed_time_modes, different_time_drivers. Real-time pairs are only checked for overlapping runs (different cadences are normal).
  • Point-in-time leakage: a TimeManager-driven process reading a dataset built at the current valid time (point_in_time_leak), frozen at a date (fixed_as_of_input) or on another TimeManager (input_other_time_manager).
  • Primary index and partitioning against the sources (check distribution). Each entity column is followed through the view SQL to its source column (sql_context.column_origin / column_lineage, unchanged columns only), the sources' primary indexes and partitioning are read from the data dictionary, and the staging step of ingestion is EXPLAINed with the current and the aligned index (EXPLAIN only):
  • pi_not_aligned_with_source: a warning when EXPLAIN shows the rows redistributed now and local with the index of the largest source, nothing when it shows no difference;
  • entity_pi_inconsistent, partition_not_aligned_with_source, fs_partition_not_on_filter;
  • segments_rescan_source: a FilterManager whose columns do not follow the largest source's partitioning;
  • pruning judged in two steps: the filter names a partitioning column (dictionary), then the plan with the manager's value known (DYNAMIC EXPLAIN under DynamicPlan=SYSTEMX, utils.query_band. forced_incremental_planning, session band restored). segment_filter_defeats_pruning and time_filter_defeats_pruning: the filter matches but the plan still reads every row, so the view SQL prevents pruning. time_filter_by_join / time_filter_not_on_partitioning: favour a scalar filter on the date partitioning column (ACC-scored views excepted, Error 6881);
  • recommendations are per entity: the index is part of the storage table name, and changing it is a re-registration (registering a view again keeps the recorded index).
  • Runs: segments_missing, segment_failed, hollow_run, stuck_running. Process SQL: sql_drift (live SQL different from the stored version) and history_mixes_versions.
  • A check that cannot run (lineage graph empty, DBC unreadable, versioning off) is listed in unverifiable with its reason, never reported as passing.
  • Admin app: Operations → Workflow assessment (/ui/workflow, entity scan on demand over SSE /ui/workflow/scan, CSV export /ui/workflow/download), and a Workflow assessment section in the Report.
  • New skill fs-workflow-assessment.
  • process_sql.chain_drift() is now the one drift computation (the feature catalog panel calls it).
  • Found while verifying live: a FilterManager with a BUSINESS_DATE column logs only its other columns in APPLIED_FILTER, so segments are counted per run when the logged filter repeats; and run numbers from execution_run_ids follow RUN_ID text, not time, so "the latest run" is chosen by start time. EXPLAIN names a table by its correlation name (DB_SOURCE.A in view ...). A static plan shows an all-rows scan of a source filtered on its own partitioning column (FilterManager or TimeManager value = a parameter, request below the incremental-planning cost threshold); with incremental planning forced, the same views read 304 or 365 partitions, and a PTI source "a time range of" it. A filter on one partitioning level is enough. Later plan steps name the table as a grouping column while scanning a spool: only steps reading FROM it count. process_storage_tables matches features by name within a domain, so it also returns another entity's tables; the check keeps the process's own entity only.

[0.3.3.24] - 2026-10-21

Skills reviewed against the code

  • Fixed: td-explain called tdfs4ds.document_process (it is tdfs4ds.genai.document_process); td-analytics-workflow said connect(create_if_missing=True) does not exist (it does); fs-setup offered STORE_FEATURE='UPDATE_INSERT' (broken, keep 'MERGE') and described FEATURE_STORE_TIME as time travel only; fs-lineage-graph described rebuild_lineage_graph() as a wipe (it is diff-based); fs-tdstone2 overwrote tdfs4ds.SCHEMA instead of connect() and linked into the package source; a ca-plot classifier example carried placeholders the renderer does not fill.
  • Added: failure handling (on_failure, check_upstream, check_upstream_health, wave_on_failure) in fs-rollout; retrieve_batch and run_benchmark in fs-dataset; storage statistics, cleanup_temp_tables and process SQL versions in fs-inspect; Chroma collections per store and index_freshness in fs-vector-store; the MCP settings in fs-agent; Demos: lines naming the notebooks; the README lists every skill.
  • New test tests/test_skills_consistency.py (offline): every tdfs4ds call in a skill's code blocks must exist and take the arguments shown, entity_id is never a dict, classifier examples use only filled placeholders, the README lists every skill, and links survive install_skills().

[0.3.3.23] - 2026-10-21

remove_feature can be scoped to one process

  • New process_id= parameter. A feature is the triplet (data domain, entity, name) and has one catalog entry whatever the number of processes computing it. remove_feature(name, entity_id, data_domain, process_id=pid) removes the feature as produced by pid:
  • computed by pid alone: removed entirely, as before;
  • also computed by another process: the catalog entry is kept, and only pid's documentation of the feature goes (its FS_PROCESS_DOCUMENTATION_FEATURES rows, the business-dictionary column of pid's view, the provenance and vector-index entries). The other process's documentation is untouched.
  • A pid that is no longer registered is accepted: the decision rests on the other producers. When the producers cannot be read, nothing is removed (None is returned and the error logged).
  • With process_id the call returns {"mode": "full" | "process", "others": [...], "listed": bool}.
  • Unchanged without process_id: the feature is removed for every producer and None is returned.
  • Removing a process with its features is documented in Inspect & monitor: impact first (downstream objects), then remove_feature(..., process_id=pid) per feature, then remove_process.

[0.3.3.22] - 2026-10-20

Entities a process no longer produces lose their validity (close_absent_entities)

  • New process option. upload_features(..., close_absent_entities=True) declares that every later run() / roll_out() of the process ends the validity of the stored entities its view no longer returns. Before, a run only wrote the entities present, so a customer that left the source kept its last values open (UNTIL_CHANGED) and current datasets kept showing it.
  • The rule is the MERGE's. A run states the truth from its valid time t onward (the TimeManager date under roll_out, FEATURE_STORE_TIME, else the run's real-time instant): absent entities have no value from t. The period before t is kept, an entity that comes back gets a new period (the absence is a gap), a re-run at the same date changes nothing, and an ordered replay gives the same history. As for the values, re-running an earlier date alone rewrites everything from that date: replay from the first date (verified live: the ingestion MERGE already rewrites the later periods of the entities present).
  • Safety. A batch that staged 0 rows closes nothing (an empty or failed upstream must not end every entity). A failed close records the batch FAILED with the database error, and on_failure applies. With a FilterManager a segment closes only its own entities: every filter column must be an entity column, else the registration is refused (nothing registered, None returned).
  • Stored in a new optional table, FS_PROCESS_OPTIONS (valid time, one row per process; the place for later per-process options), not in the descriptive METADATA column. It is created at setup() and by connect() when missing; a refused CREATE is one warning. FS_V_PROCESS_CATALOG does not join it, so its text and the registration SQL are unchanged, and a store without the table runs with the option off.
  • API. tdfs4ds.set_process_options(pid, close_absent_entities=...) changes the option of a registered process; run(..., close_absent_entities=) and roll_out(..., close_absent_entities=) override it for one call (None = as recorded). get_process_info returns CLOSE_ABSENT_ENTITIES; remove_process deletes the row; migrate_objects carries the option over.
  • SQL shape (Vantage 20). A sequenced DELETE refuses every subquery (Error 9347 for NOT EXISTS, NOT IN and IN), so the close stages the absent keys in a volatile table (NONSEQUENCED VALIDTIME SELECT) and runs the sequenced DELETE joined to it, over the MERGE's own period clause.
  • Backward compatible. Off by default; ingestion SQL unchanged when off. A run reads the option with one fail-open query.

[0.3.3.21] - 2026-10-19

Charts: "the latest week", historian tables read from their column documentation

  • Relative windows. "the latest week", "last 7 days", "past 2 months", "la dernière semaine", "les 3 derniers jours", "depuis 2 jours" zoom a chart on that span, ending at the database's current time (months and years go back on the calendar). The phrase is no longer taken for a valid time: the answer used to start with "Could not read the date 'latest week'".
  • Historian tables without a DEEP_LAYOUT section. A table whose columns are documented (a timestamp, the sensor name, the measured value, identifiers such as the drawing tower) is drawn as a historian table: the roles are read from the column types and descriptions (infer_historian_layout), stated in the answer, and used for that chart only (plotting.using_layout). set_deep_table_documentation still makes them permanent.
  • Sensors and identifiers found by their wording. "the drawing length sensor" finds the tag DRAWING_LENGTH, DrawingLength or Drawing length (m) among the tags of the table (match_tags); several equal matches are asked. "per drawing tower" finds the id column described as the drawing tower. A historian table is found from its documentation, or from the word "historian" when there is only one.
  • The chart request survives "I could not tell what to plot". That answer now keeps the request, so a reply such as "it is in the historian" completes it; it lists the documented historian tables and their tags. Before, the reply was classified on its own and could be sent to the MCP tools.
  • MCP tools that cannot be imported. No module named 'langchain_core.messages.content' (a 0.x langchain-core under 1.x packages) is reported as such, with the installed versions and the %pip install -U line, in the chat answer and in check_configuration. setup.py requires langchain_core>=1.0.

[0.3.3.20] - 2026-10-16

launch_app asks for the feature store; query band switch in Parameters

  • Parameters → Execution log: query band and history: switch QUERY_BAND / EXECUTION_RESOURCES for the app, with the tables created and a check that this user's runs can be measured. A store not selected is now said as such ("choose one in Parameters") instead of "disabled".
  • launch_app asks for the feature store when TDFS4DS_SCHEMA is not set (lists the stores visible with the logon; the choice can be saved) and checks the query band and DBQL with the same logon (a note, nothing asked). One check, query_band.query_band_capability, also behind check_configuration. find_feature_stores(run=) reads through another connection.

[0.3.3.19] - 2026-10-16

Execution log, consumption history, defective datasets

  • Resource usage kept beyond DBQL's retention. Two optional tables, created by setup() / connect() when missing: FS_EXECUTION_RESOURCES (one row per execution: the follow-up columns, the summed DBQL figures and PROCESS_VERSION, the process version in force at the execution's valid time) and FS_EXECUTION_STATEMENTS (one row per statement). tdfs4ds.harvest_execution_resources() copies, in the database, the executions DBQL has finished writing; it also runs at the start of run() (at most every 10 minutes, never fatal). New readers read_execution_resources(process_ids, days, as_of) (stored + live join; says when DBQL is unreadable or logs nothing), execution_statements(execution_id) and the pure process_run_statistics(frame, by_version=). TDFS4DS_EXECUTION_RESOURCES=false turns it off.
  • Runs, not follow-up rows. A FilterManager run writes one row per segment and roll_out one per business date; frequency and per-run figures now group them (same RUN_ID, rows within 10 minutes). The Report's refresh cadence uses the same rule (process_followup.execution_starts).
  • Admin app, Operations › Execution log: each execution with its process version, segment, status (whole error), duration, CPU, skew, I/O and spool; its statements on selection; per-process and per-version consumption with a chart over time; XLSX export. Maintenance › Execution history copies from DBQL on demand.
  • Feature catalog card: Runs and consumption for the 30 days up to the catalog time viewed; the versions table marks the version in force at that time and gives each version's median run and CPU. The export adds PROCESS_VERSION_AT, RUNS_30D, RUN_FREQUENCY, MEDIAN_RUN_SEC, MEDIAN_RUN_CPU_SEC to Processes.
  • Defective datasets: in the Dataset builder's list, a failed sample marks the dataset Defective with its cause (its view dropped while the catalog still lists it, an object it reads gone, access refused, no longer compiles); Check all finds them without reading data. Remove from catalog (admin view, primary store) previews then removes. New tdfs4ds.remove_dataset(name, drop_view=False, dry_run=True): catalog rows (history kept), lineage, business documentation, provenance and vector-index entries; the view only on request, and nothing to drop when it is already gone. drop_dataset() gains close_lineage=.
  • followup_resource_usage_sql(process_ids=[...]); with days=, EXEC_SEQ is now numbered before the time bound (an older run's rows were renumbered and could be paired with the wrong executions). Windows over 99 days no longer produce an invalid interval literal.

[0.3.3.18] - 2026-10-16

launch_app asks for missing connection settings

  • launch_app() checks the database connection before starting the app: host, username, password (or a JWT token) and TDFS4DS_TD_ENCRYPT, resolved the way the app reads them (the env_file given, which wins over the environment, else the first of ./.env and ~/.tdfs4ds/.env), and tests the logon with the raw driver (the notebook's connection is untouched). When one is missing, malformed or refused, and someone can answer (a terminal or a notebook), it asks: type them (password hidden; the username and password are asked again after a refusal), use another .env found, or start without them (Connect page). Typed values can be saved for the next launches in a .env found or a new one (default ~/.tdfs4ds/.env, created readable by the user only), edited line by line; the password is asked separately. New ask= (None = when someone can answer, False = never) and check_logon=. tdfs4ds.connection_settings(env_file) reports what the app would use and where each value comes from, never a secret. The returned connection lists the variables typed and where they were saved.
  • Fixed: the app crashed at start when no feature store was selected (TDFS4DS_SCHEMA unset): the background index build received None as the store. It now builds nothing and says to choose the store in Parameters.
  • The logon check (utils.reconnect._credentials_probe) reports the driver's whole message (it kept the first line, where the cause of an Error 169 is not).

[0.3.3.17] - 2026-10-16

Fixed

  • TDFS4DS_TD_ENCRYPT broke the connection on recent drivers. teradataml's create_context upper-cases every string connection parameter (for TMODE), so TDFS4DS_TD_ENCRYPT=false reached the driver as "FALSE", which teradatasql 20.0.69 refuses: [Error 169] Unable to parse JSON connection parameters ... trying to unmarshal "FALSE" into bool, and the app could not start. The setting (and the Connect page's Encrypt field) is now read by utils.reconnect.encrypt_setting (true/false/1/0/yes/no/on/off, any case) and passed to create_context as a bool and to the raw driver as 'true'/'false'. Any other value is refused with a message naming the variable. Workaround on older versions: remove the TDFS4DS_TD_ENCRYPT line (encryption off is the default).

[0.3.3.16] - 2026-10-16

The agent searches documented data outside the feature store; the documentation exports and imports back

  • Inside or outside the feature store, one rule. tdfs4ds.documentation_export.object_scope decides it: process views, datasets, Time/FilterManagers and FS_* / *_HIDDEN objects are in the feature store; every other documented table or view is outside it (documented data, not governed features: no data domain, no entity, no refresh history, column relationships not certified). The agent and the export use it.
  • The consumer agent finds tables and columns by meaning. "Which table contains data about stock prices?", "is there any data dealing with transactions?": the vector hits on documented objects outside the store were dropped until now (only feature, dataset and process names passed the grounding filter). They are now reported under their own heading, Outside the feature store, written by code (never by the LLM), with the matching columns and the feature processes that already read each object (lineage). The feature-matching LLM never sees them, so a source column is never presented as a feature. A question about features that found some gets one line pointing to the outside data; a question about data, or one that found no feature, gets the full section after the In the feature store part. Such a question is routed to SEARCH whatever the classifier said, unless it names a documented table or column; "what is this table?" / "lineage of this table" then refer to the best match. DEFINITION of a documented source table or column opens with the same fixed sentence. Without a vector index, outside objects are matched on words. Vector hits now carry DATABASE_NAME.
  • tdfs4ds.export_documentation(path, scope=, databases=, objects=, as_of=, include_missing=True) writes the business dictionary as XLSX (About / Objects / Columns / Sections, Excel tables) or one HTML page: every documented table and view, its description, columns and sections, each row tagged with its side and kind, with the provenance of each text. What is not documented yet is listed too, in yellow (STATUS='missing'): process views, datasets and lineage tables/views without a description (existence checked in DBC), their undocumented columns in table order, and the standard sections a dataset lacks. Admin app: Documentation › Coverage › Export the documentation (scope, XLSX, HTML).
  • tdfs4ds.import_documentation(path, dry_run=True, author=) writes the filled-in workbook back: only new or changed texts, through the hand-documentation API (MANUAL provenance, kept by LLM runs; a feature's text also reaches the process documentation and the datasets). An emptied cell deletes nothing; read-only stores, managers, internal objects and process-view sections are skipped with a reason. Live: an unedited export re-imports as 250 unchanged texts and nothing to write.

[0.3.3.15] - 2026-10-15

Several feature stores in the admin app; a status bar; Chroma indexes every store

  • Status bar. A bar at the bottom of the admin app names the feature stores the session reads and what runs in the background, with its progress: the vector index build, Maintenance runs, documentation, migration, evaluation and benchmark runs. details lists the jobs of the last minute, with the whole error of a failed one. New route GET /ui/status; the registry is tdfs4ds.serve.activity (no database access).
  • The index is built in the background. At startup the server used to build the vector index before opening its port, so the first operations waited with nothing on screen. It now answers at once and builds in the background (one job per store; a second request for the same store is queued, never run alongside). The Parameters actions that rebuilt the index (store, vector store, embedding model) return at once too, and their routes no longer block the server while they run. A background build reads the business dictionary through a raw teradatasql connection of its own (from the TDFS4DS_TD_* settings): reading on teradataml's single shared connection while the app served requests broke that connection for the rest of the process (seen while developing this release). Without those settings, and on the Teradata Vector Store backend (which works through teradataml), the build runs in the foreground as before.
  • Several feature stores in Parameters. Pick the primary store (read-write) and any number of stores to read from (read-only, connect(databases_readonly=...)). A store that cannot be read is named with the reason, and the panel says how many features exist in several stores (the assistant treats each as one, the primary's). Stores set with TDFS4DS_SCHEMA_READONLY are now validated at startup.
  • Chroma: one set of collections per store. tdfs4ds__<database>__objects / _sections / _columns. Switching the primary store no longer wipes and re-embeds the index (it used to run force_rebuild=True); a search covers every attached store and tags each hit with FEATURE_STORE, as the Teradata Vector Store backend already did. A read-only store is read through its public FS_V_BUSINESS_DICTIONARY_* views; nothing is written to it. build_vector_index(force_rebuild=False, progress=None, stores=None): stores picks the stores (default: every attached one), progress receives each step. Upgrade: the first build re-embeds the primary store once; the old collections tdfs4ds_objects / _sections / _columns are left unused (delete them with chromadb.PersistentClient(path=tdfs4ds.CHROMA_PATH).delete_collection(name) when no older version shares the index).
  • Chroma: an indexing date per store, and removed rows leave the index. Each build records, per store and collection, the database's time when it started (in a small tdfs4ds__index_state collection) and the embedding model. The next build of the primary store then reads only the documentation written since (ValidStart at or after that date) plus the ids of every current row, so removed documentation is dropped from the index; until now nothing ever left it. Live: an unchanged store re-reads nothing, one new row embeds one row, its deletion removes it. tdfs4ds.agent.vector_index.index_freshness(store) says whether the index is behind (up_to_date / behind with the number of rows / never / model_changed / unknown) without embedding or scanning Chroma; Maintenance shows it for each store. Another embedding model, a force_rebuild, or a read-only store (its public views carry no ValidStart) use the full row-by-row comparison, as before.
  • Maintenance re-indexes the stores you tick, including a store that is not attached (read through its views). force vector rebuild re-embeds only those. The Teradata Vector Store backend still indexes the primary only.
  • Run history and documentation per store. With read-only stores attached, the Report, EFS map, consistency checks, feature-catalog export and documentation coverage used to read runs and documentation from the primary only, so another store's processes showed as never run and its features as undocumented. They now read each store's FS_V_FOLLOW_UP, FS_V_PROCESS_DOCUMENTATION_*, FS_V_DOCUMENTATION_PROVENANCE and FS_V_MANAGER_REGISTRY (new capability probes at attach; a store without one is skipped for that read). The three FS_V_PROCESS_DOCUMENTATION_* views were declared but never created; connect() now creates them. With a single store every read keeps its SQL. Past valid times, stored SQL versions and the EFS map's metadata objects stay primary-only.
  • Writes stay primary-only. Coverage and Objects list another store's objects with 🔒; they cannot be documented or edited from the app (the server refuses it too). The Execution plan lists and plans the primary's processes only (run() executes those alone).
  • Fixed: picking a source in the Migration panel left that store attached for the rest of the session, so every later read crossed stores.
  • SageMaker start script: the settings reach notebooks and terminals. sagemaker_lifecycle_script() used to give the env file to the app only. It now links it as ~/.tdfs4ds/.env (read by import tdfs4ds in every notebook) and writes ~/.tdfs4ds/env.sh, quoted export lines sourced from ~/.bashrc, for terminals. Both are readable by ec2-user only; nothing goes to /etc/profile.d, which every account can read (and which a kernel never reads). share_env=False restores the previous script.

[0.3.3.14] - 2026-10-15

Documentation edits no longer fail with Error 3807 on a staging table

  • Fixed: upload_business_dictionary_objects, _columns and _sections staged their rows in a volatile table and read it back with a second statement. A volatile table exists only in the session that created it, and the admin app's threads share teradataml's pooled connection, so the MERGE could run in another session (or after a reconnect) and fail with [Error 3807] Object '<user>.BIZ_DICT_OBJECTS_STAGING' does not exist. An edit made in the app failed while the same upload from a notebook worked.
  • Each upload now stages in a regular table named tdfs_tmp_<12 hex> in the store schema and drops it afterwards; cleanup_temp_tables() already recognises that name. The MERGE statements are unchanged. The account needs CREATE TABLE in the store schema, as for feature registration.

[0.3.3.13] - 2026-10-15

Removals clean up after themselves; process versions ignore managers; the Connect page says why

  • Managers are not versions. A TimeManager / FilterManager replaces its own view at every update(). Manager views are never steps of a version, and every manager the chain reads is replaced by a placeholder before hashing, so moving to the next date or segment, or repointing a process onto another manager, makes no new version.
  • Removing a process or a feature removes what it left behind.
  • remove_process also removes:
    • the provenance of its documentation;
    • its vector-index entries: Chroma by id. The Teradata Vector Store collections are refreshed, because their embeddings are keyed by row position; refresh_index=False skips that;
    • its stored SQL versions;
    • its FilterManager / TimeManager catalog rows;
    • the lineage edges of its view once the view is gone. The new drop_view=True drops it; a view kept after its process is removed is still a real object, so its edges stay.
  • remove_feature also removes its feature documentation (process feature rows, the business-dictionary column of each producing process view, its provenance) and its vector-index entries. A feature is not a node of the lineage graph, so there is nothing to remove there.
  • Unchanged: the stored values (FS_T_*) are kept, as before.
  • Module: removal.py.
  • Two process-version bugs found by the new demo notebook (0.3.3.12 is affected):
  • Version 1 never closed. A version captured in real time runs to 9999-12-31. The next capture at a business date deleted only up to 9999-01-01 and left a sliver of it, so its period read "valid to 9999-12-31". Deletes and updates now run up to UNTIL_CHANGED, as the ingestion MERGE does.
  • A spurious version 2 at the first roll-out. A TimeManager attached at registration is recorded only after the first capture, so the neutral text changed between the two captures. The managers now come from the chain itself as well as from the catalog. Object names are compared unquoted.
  • Demo notebook 10b - The SQL beneath a process.
  • What it shows: an upstream view edited between two roll-outs; versions 1 and 2 at their business dates; the diff; the feature version unchanged; a history with its _PROCESS_VERSION column; the export columns; the backfill.
  • Notebook 10: removes its second process with remove_process(drop_view=True).
  • The Connect page showed only TDML_2000 Failed to connect to Teradata Vantage. teradataml raises it from the driver's error, which names the fault (password, LDAP, host, encryption). The page now shows the whole chain.
  • Credentials typed in the Connect page now serve the automatic reconnect. It reads only TDFS4DS_TD_*, so a session opened from the form could not be re-opened after a drop. They are kept in the server process's environment and never written to a file.

[0.3.3.12] - 2026-10-15

Feature values of given entities; process versions from the SQL beneath a process

  • The values of a feature for given entities. "What is sum_Transaction_Amount for customer 33357?" and "show the features of TICKER NVDA" are answered by the VALUES skill with a table written by code, the valid time it was read at, and each feature's process version.
  • tdfs4ds.feature_values_for(features, entity_values, entity=, data_domain=, as_of=) and tdfs4ds.entity_profile(entity, entity_values, ...) (every feature of the entity, at most AGENT_LIST_MAX_ITEMS).
  • The lookup compares the key columns untrimmed with typed literals, so Teradata reads one AMP and one partition (checked in EXPLAIN). Values the store does not hold are named in the answer, never guessed.
  • Process versions. A process keeps its PROCESS_ID (it belongs to the view name), but the SQL beneath the view - the view and every view it reads, down to the source tables - can change. That SQL is now versioned.
  • A new optional temporal table FS_PROCESS_SQL (view FS_V_PROCESS_SQL) stores the chain of each version: one row per view and 32,000-character chunk, with a hash per view and for the chain.
  • upload_features and run() read the chain and compare its hash with the stored one. A change (whitespace, case and comments ignored) records a new version and moves the process catalog's PROCESS_VERSION to it. Inside a roll_out, only the first date checks. Measured: about 1 s per run when nothing changed.
  • Valid time: a version is stamped at the valid time the process writes its values at (FEATURE_STORE_TIME / the TimeManager's date, else the run's real-time pin), like the catalog row and the values. The version that computed a stored value is the one in force at its ValidStart.
  • The feature version is unchanged: FEATURE_VERSION in the FS_T_* storage tables stays the PROCESS_ID. Datasets, histories and plots read the same rows as before.
  • API: tdfs4ds.process_sql_versions(pid), tdfs4ds.process_sql(pid, version=None, as_of=None), tdfs4ds.process_sql_diff(pid, v1, v2), tdfs4ds.capture_process_sql(all_processes=True, dry_run=True). tdfs4ds.PROCESS_SQL_VERSIONING = False (env TDFS4DS_PROCESS_SQL_VERSIONING) switches it all off: upload_features and run() then send exactly the statements of 0.3.3.11.
  • Histories know the versions. build_historical_dataset warns when the window spans several process versions of a feature, and tdfs4ds.history_versions(dataset) lists them. process_version=True also creates a companion view <view>_PV with a <feature>_PROCESS_VERSION column per period; the dataset view itself is unchanged. The chat states the versions of a history dataset, and a history chart notes them.
  • Feature report (admin app and export). The process detail lists its versions (validity, changed views, runs that used each, who captured it). A version opens its stored SQL, two versions can be compared (diff), and a SQL changed since the last capture is flagged with a Capture now button. At a past valid time the SQL shown is the version in force then. The export adds SQL_VERSIONS and LAST_SQL_CHANGE per process and, at a past valid time, the stored SQL.

Upgrading an existing feature store

  • connect() (or setup()) creates FS_PROCESS_SQL and its view when missing. A user who may not create them gets one warning and versioning stays off for that session; nothing else changes. No existing table is altered.
  • Existing processes have no stored SQL. Their first capture (next run() / upload_features, or the backfill) stores the current SQL under the catalog's current PROCESS_VERSION, without changing it. Backfill: tdfs4ds.capture_process_sql(all_processes=True, dry_run=False), or Maintenance > Process SQL versions in the admin app. No history exists before that capture: earlier runs show as "before versioning".
  • An older client sharing the store ignores the table. A SQL change made through it is recorded at the next capture by a newer client; the feature report flags it meanwhile.
  • A migrated process starts again at the target store's own first capture (versions are not copied).

[0.3.3.11] - 2026-10-15

Entity values and feature extremes in the chat; no database lock in the app

  • The values of an entity. "Give me 5 TICKER values" and "What are the TICKER available?" returned the list of TICKER's features, then the reference ticker NVDA. A new VALUES skill answers with how many distinct values the entity has in the feature store and a random sample (tdfs4ds.feature_values.entity_sample: one COUNT over the union of the distinct keys of its storage tables, then SAMPLE n, all in the database).
  • The ends of a feature's distribution. "Which customers have the highest sum_Transaction_Amount?", "top 5 ...", "outliers of ..." give the statistics (min, quartiles, max, mean, std), the n highest and n lowest entities, and the outliers by Tukey's rule with the n farthest from the median (feature_values.feature_extremes: one aggregate pass, then TOP n reads). A distribution chart of a feature lists the same under the chart.
  • A feature name shared by several entities or domains: the entity named in the question, even in plain words ("customers" for CustomerID), the domain written in it, then the conversation's domain choose; otherwise the agent asks with a numbered list and the reply ("2", "CustomerID") completes the request.
  • Routed by code, so the classifier cannot send these questions to the feature list.
  • "What could you tell me about DB_SOURCE.MARKET_PRICES" answered that nothing was documented. The business dictionary lookups matched the whole db.name against the object name; they now split it and keep the rows of that database (a bare name still reads every database).
  • "Which features are available for CustomerID?" after a turn on another domain answered "the data domain CustomerID" with that other domain's features, then "0 features". A name that is not a data domain of the store no longer labels the answer, and a domain the question does not write, which does not hold the entity named, gives way to the entity: its features in every domain holding it.
  • Admin app: the database lock of 0.3.3.10 is removed (Denis: the database deals with concurrency). The configuration check keeps its streamed rows (each check shows as it finishes, with its duration) and its corrected hint for a broken session. The errors that motivated the lock came from the client side - one teradataml connection shared by the app's threads, replaced by a reconnect under the others - and can come back under concurrent use; one connection per request thread is the follow-up.

[0.3.3.10] - 2026-10-14

Superimposed curves aligned on their mean or median; one database request at a time in the app

  • Align the level on the mean or the median. rebase= (on plot_aligned_periods, plot_historian, panel) now takes True / 'first' (100 at the first value, as before), 'mean' or 'median'. The last two give 100 at the mean / median of each whole extracted curve, margin included. The x alignment (days from each period's start) is independent: rebase=False aligns on x only.
  • How: Teradata has MEDIAN as an aggregate only, and not in a sub-query either (MEDIAN(...) OVER is Error 3707, MEDIAN in a derived table Error 3706, both measured). So the reference of each curve is grouped once into a small volatile table and joined back to the staging table (plotting.rebase_update_sql). A curve whose reference is 0 is left out. Checked live: every curve's median reads back exactly 100.
  • In the chat: "align them on the median", "the mean instead", "sur la médiane". A short reply that only changes the previous chart (reference, "raw values", "separately", a window) is now routed to the chart by code; the classifier sent the first two to OUT_OF_SCOPE and EXPLAIN.
  • Admin app: one database request at a time. teradataml has one process-wide connection, and the app serves requests from several threads. Two requests on it at once, or one reconnecting under the others, failed with N is not a valid connection pool handle (Benchmark dataset list) or 'NoneType' object has no attribute 'cursor' (configuration check: Temporal and Query band reported as failing although they work).
  • The fix: a lock (serve/db_access.py). Every /ui/* and /v1/* request takes it in a middleware; every streamed job (chat turn, benchmark, maintenance, evaluation, migration, documentation, storage counts) takes it in its worker thread for the whole job. A chat turn takes it before the turn lock, the order the dataset builder uses.
  • Waiting: a request that cannot get it within TDFS4DS_SERVE_DB_WAIT seconds (default 60) is answered "The database is busy (benchmark run, 42 s)" instead of hanging; a chat turn says it is waiting.
  • Checked live: 8 concurrent panel requests, twice, all answered with no error.
  • Not covered: the Gradio chat at /chatbot and the LangServe routes.
  • Configuration check: when the database never answered (a broken client session), the Temporal and Query band hints now say so instead of asking the DBA to enable Temporal.
  • langchain-mcp-adapters is named in the serve extra (it was already a dependency).

[0.3.3.9] - 2026-10-14

Superimposed periods in one chart

  • One chart, not one per stock. plot_aligned_periods now draws every stock and every period of the feature in a single chart per tag. Each curve is one stock and one period (legend NVDA | 2017-07-03), each indexed to 100 at its own start. Every curve is drawn: the series cap is raised to stocks x periods.
  • Before: one panel per stock, so the stocks could not be compared on the same axes.
  • One panel per stock remains available: per_id=True, or "separately" / "one chart per stock" in the chat.
  • TD_PLOT has 10 colours: beyond 10 curves, colours repeat, and the legend labels tell the curves apart.
  • A legend with many curves no longer covers them (every chart, not only this one).
  • TD_PLOT's limits: it puts the legend inside the plot area only ('outside' is Error 9153). It takes no legend font size or column count, one COLOR for every series, and no NULL value (measured on Vantage 20).
  • Up to tdfs4ds.PLOT_LEGEND_INLINE_MAX curves (6): unchanged.
  • More curves: the image grows taller until every entry fits. On a numeric x axis (the superimposed periods), the x range is extended on the right by the legend's width, sized from the longest label; the legend sits there, 'upper right', clear of the curves. A time axis takes no XRANGE and no empty point, so its legend stays 'best' in the taller image.
  • Above tdfs4ds.PLOT_LEGEND_MAX curves (40): no legend, and a note says why.
  • New pure helper plotting.legend_plan(...). A panel's legend can be a TD_PLOT location string.
  • "Superimpose them" after a windowed chart redrew the whole history as base 100, without the window or the periods. The classifier's sticky object name made the reply look like a new request. A modifier-only reply is now matched on the words typed, so it redraws the previous chart with the periods aligned.

[0.3.3.8] - 2026-10-14

Duplicate keys shown when a process is refused; datasets with a DATE entity key

  • A process that returns duplicates now shows them. The refusal (Invalid process: the process generates duplicates) is now a DuplicateEntityKeysError, still a ValueError.
  • Its first line gives the number of duplicated key values (roll_out prints only that line).
  • Its message shows the rows of up to 3 duplicated keys and names the columns whose value differs between them. When nothing differs, it says the view repeats rows (a join fan-out, a UNION ALL, a missing DISTINCT).
  • A query listing every duplicated key of the process view is included.
  • On the exception: e.sample (DataFrame), e.differing, e.query, e.n_keys.
  • Before: only the count, and in DEBUG_MODE an unexecuted query text.
  • Exact copies: they never counted. The check reads a SET volatile table, where exact copies are already dropped, and the listing query counts distinct rows to match.
  • Fixed: building a dataset failed when an entity key is a DATE (Error 3707 "between the word 'YYYY' and '-'", seen from the chat).
  • Cause: the catalog registration pasted the column type, DATE FORMAT 'YYYY-MM-DD', between quotes without escaping it.
  • Fix: every value of those catalog writes is now escaped.
  • Recovering a dataset that failed this way: its view and catalog row exist without entity and feature rows. Repair it with tdfs4ds.restore_dataset_catalog(dry_run=False, datasets=['<name>']).
  • Fixed: a dataset name given as 'database.view' was taken whole as the view name. This applies to upload_features(dataset_view_name='DB.DS_X'), build_dataset and build_historical_dataset.
  • Effect: the view was created as DB."DB.DS_X" (one name containing a dot) and registered as DB.DB.DS_X.
  • Fix: the database is now taken from the name; a different schema_name given as well is refused.
  • Cleaning up a dataset created this way:
    1. Run the process again to build DB.DS_X properly.
    2. Drop the old view with DROP VIEW DB."DB.DS_X".
    3. Close its catalog row with tdfs4ds.restore_dataset_catalog(drop_orphans=True). Run it with the default dry_run=True first, then with dry_run=False.

[0.3.3.7] - 2026-10-14

A local Teradata MCP server from the tdfs4ds credentials; the admin app on SageMaker Studio

  • tdfs4ds.start_mcp_server(port=8001, profile='eda') starts the Community teradata-mcp-server on 127.0.0.1, logged on with TDFS4DS_TD_HOST / _USERNAME / _PASSWORD (database: TDFS4DS_TD_DATABASE, else tdfs4ds.SCHEMA). It points tdfs4ds.MCP_SERVER_URL at it (Community edition, streamable HTTP), so the agent's MCP skills (DATA_QUERY, ALGORITHM) work. stop_mcp_server() and mcp_server_status() go with it.
  • The password is never on a command line: the database URI goes through the server's environment, with the user name and password percent-encoded (the server hands them to SQLAlchemy, which decodes them, so @ : / # in a password are safe).
  • Checked before starting: the logon is tried with the raw driver, and a refused logon starts nothing. A port already used by another program is refused rather than shared.
  • Not a dependency. The server runs in its own process: install it in the kernel's environment (%pip install teradata-mcp-server) or in its own (pipx install teradata-mcp-server).
  • Limit: a token logon (TDFS4DS_TD_LOGDATA) cannot drive it, because the server drops LOGDATA from its own connection. Use TD2 or LDAP.
  • launch_app(mcp=True) starts that server with the app and gives its address to the app, through the new pass-through variable TDFS4DS_SERVE_MCP_URL. The app applies it after loading its --env file, so an MCP_SERVER_URL in that file cannot hide it. If the server fails to start, the app starts anyway and the failure is in the notes. sagemaker_lifecycle_script(mcp=True) does the same at instance start.
  • The admin app on SageMaker Studio.
  • Wrong Jupyter server: launch_app() took the newest Jupyter runtime file. On Studio that was another service's (http://localhost:3001/mcp), so the app was started with the prefix /proxy/absolute/8000 while Studio forwards /jupyterlab/default/proxy/absolute/8000/, and every page answered 404. The launcher now prefers the server that is an ancestor of the kernel, then one whose URL path is its base path. On Studio it falls back to /jupyterlab/default/.
  • root_path= sets the prefix explicitly.
  • Proxy check: a 404 written by the app itself is now reported as a wrong prefix, not as a missing jupyter-server-proxy.
  • No invented address: the "Full address" line (a *.notebook.* host) is no longer printed on Studio, whose address cannot be worked out from inside the space.

[0.3.3.6] - 2026-10-14

Plots of source data: historian tables, periods, a feature's period as the window

  • tdfs4ds.plot_historian(table, tags, id_values=) draws tags of a historian (deep) table as time series.
  • Reading the table: its documented layout gives the columns: the tag is filtered, the payload is the value, the timestamp is the axis, and there is one line per id.
  • Panels: one panel per tag.
  • Matching: tags and ids are matched without regard to case. An unknown tag is refused with the list of tags; an unknown id is named in the notes.
  • Also new: historian_panels, historian_tags, historian_layout.
  • Periods. A PERIOD column, or two date/timestamp columns named as bounds (X_START / X_END, BEGIN_X / END_X, ValidStart / ValidEnd), is a period.
  • Values over a period are drawn as steps. A historical dataset is now drawn this way by default.
  • kind='periods' draws the periods themselves: 1 while a period holds, 0 outside. The 1-second ramps sit inside the period, so adjacent periods meet cleanly.
  • Before: PERIOD columns were dropped.
  • feature_window(feature, entity_value, as_of=) reads the period a feature stores for one entity value, to use as the time window of a series.
  • It takes a PERIOD feature, or either bound of a ..._start / ..._end pair.
  • feature_periods_panel(feature, entity_value) draws every distinct period a feature has held over its history.
  • Its NONSEQUENCED VALIDTIME qualifier stays inside the derived table; over one it is Error 9330.
  • As an extra_panels of plot_historian, it spans the series' time range.
  • The chat plots documented source objects. A table or view from the business dictionary or the lineage graph, or a tag of a historian table, can be named:
  • "plot the close of NVDA and MSFT" (CLOSE is a tag, NVDA / MSFT are ids);
  • "plot NVDA close during its last stress episode" (window from last_stress_start / last_stress_end);
  • "plot NVDA close with the periods of last_stress_start" (a second panel of the episodes);
  • "plot the transactions table".
  • Never targets: feature-store internals. A lineage-only name is checked in DBC.
  • Answers: they say the data is source data, not features. The LLM spec can target a source object, and every name is checked.
  • An incomplete request is completed by asking. "Plot all the stocks during the stress period" names no table, tag or feature.
  • The table is found from the documented descriptions ("daily prices of five US stocks").
  • What the agent asks: which measure (the table's tags), and which period (the pairs of date features sharing a word with the question, each with the dates it stores).
  • The reply completes the request, which the agent keeps between turns: "the close, period 1, around it" (a number names an offered period).
  • Nothing is guessed: with no tag named, the agent asks instead of drawing every tag; "all tags" draws them all.
  • "around" widens a stored period by its length on each side, at least 7 days.
  • Periods superimposed in time. tdfs4ds.plot_aligned_periods(table, tags, period_feature, id_values=, margin_days=, rebase=True) draws one curve per period that a feature has held (every stress episode).
  • Axes: x is the number of days from the period's start, so periods months apart overlay. The values are an index, 100 at the start (rebase=False keeps them).
  • Panels: one panel per stock when several are drawn.
  • In the chat: "superimpose them", "each episode", "align", "same start", "day 0" do this when a period feature is in play. "raw values" keeps the prices.
  • Without periods: "base 100" / "rebase" indexes the series of one window (rebase=True on plot_historian).
  • Query plans of these charts (read in EXPLAIN).
  • Literal bounds per period. The periods are read first (one small query), and the series query has one UNION ALL branch per period with literal TIMESTAMP bounds. A PTI historian then reads a time range per branch, where a join to the periods would scan it.
  • Plain comparisons. Tag and ids are compared without TRIM.
  • No NULL test on an expression of x. Staging no longer tests x IS NOT NULL when x cannot be NULL (new x_known). The test on the day-offset expression was pushed into every branch, and turned the time-range read back into an all-rows scan (measured).
  • An invalid literal removed. INTERVAL '7' DAY(4) is Error 3706, so it is no longer emitted.
  • A period stored identically for every entity value (per-stock copies of one episode) is one window.
  • Chart answers are a bullet list (they were one paragraph in the app).
  • No point limit by default. PLOT_MAX_POINTS is now None: TD_PLOT draws large requests, and its whole error comes back when it fails. A number still refuses requests above it.
  • Default value of a plot: an identifier column (CustomerID, customer_id, *_KEY) is no longer chosen as the value.
  • Demo notebook T2 ends with the NVDA close and its stress episodes, drawn in the database.
  • Live-checked on DB_LAB: the six chat questions above, French included, and every library call. Guarded by tests/test_plot_sources.py.

[0.3.3.5] - 2026-10-14

Four gaps found by running the agent evaluation and the serving benchmark from the admin app on the demo lab store.

Fixed

  • "Where does that object come from?" after a question on a column. The agent found the column's table but positioned it in the lineage graph by its bare name, looked up under the store's database. A source table lives in another database (DB_SOURCE.MARKET_PRICES), so lineage answered "could not be located".
  • skill_lineage now positions the qualified owner.
  • A bare name outside the store's database is accepted when exactly one lineage node carries it.
  • A DEFINITION answer about a non-feature column records its table, so the follow-up keeps it.
  • In the evaluation run, both such turns had scored 1 out of 5. Live, the follow-up now lists the 4 objects that depend on MARKET_PRICES.
  • "What features are available for CustomerID?" asked which data domain instead of answering. An entity named in the question, without a data domain, now scopes the list:
  • One data domain has the entity: its features in that domain.
  • Several have it: one section per data domain.
  • The conversation's data domain has it: that domain only.
  • Benchmark report inside the app: the section titles were navy on the report's dark panels, and unreadable. The report now gives them its own text colour.

Evaluation

  • Columns of a historian table. A name documented on a deep (historian) table is a tag, not a column, and the agent says so. The judge did not know the table's layout, and scored that framing as invented.
  • The evaluation now reads the table's layout, and the column facts state that the name is a tag, with the layout sentence and the layout's column names.
  • The judge is told that this framing is documented.

[0.3.3.4] - 2026-10-13

The admin app from a notebook (SageMaker, JupyterHub, AI Studio)

  • tdfs4ds.launch_app(env_file=None, port=8000) starts python -m tdfs4ds.serve in a separate process and shows a link ("Open the tdfs4ds app").
  • In a notebook, the link goes through Jupyter's own proxy, <jupyter base>/proxy/absolute/<port>/. The base is read from the Jupyter server's runtime file, and the app's path prefix is set to match.
  • Outside Jupyter, the link is http://localhost:<port>/.
  • What it starts and skips: it skips the standalone Gradio chat and the docs server. It passes --env only when the file exists, and says so otherwise (the app then starts disconnected).
  • Waiting and checking: it waits for /health, reuses an app already running on the port, and shows the log tail when the app stops at start. It asks Jupyter's proxy whether it reaches the app, with an install hint when it does not.
  • tdfs4ds.app_status() and tdfs4ds.stop_app() use a state file (~/.tdfs4ds/app_<port>.json). stop_app never stops an app it did not start.
  • tdfs4ds.sagemaker_lifecycle_script(env_file=, version=None, start_app=True) returns the SageMaker "Start notebook" script. It installs the latest tdfs4ds (or a fixed version) in the python3 conda environment and starts the app behind the proxy, all in the background.
  • Checked on this machine with SageMaker's layout: a Jupyter server with jupyter-server-proxy in its own environment, separate from the kernel's.
  • Page, stylesheet, GET and POST requests (Connect, create preview) and the chat stream all work through the proxy.
  • Not run on a SageMaker instance.
  • Panels: the feature catalog and dataset builder say "No feature store is selected" instead of showing Error 3707 (FROM .<table>) when no store is chosen.

[0.3.3.3] - 2026-10-13

The chat keeps the conversation's entity and topic

Seen live: "build a dataset for customer id" was built on Category, "what are these datasets?" listed all 13 datasets instead of the 5 on CustomerID just mentioned, and "I meant for CustomerID?" was answered as a definition.

  • Choosing the entity of a dataset build:
  • Order: the entity named in the question first, then the entity of a named feature, then the conversation's entity, then the only entity there is. Otherwise the agent asks, with a numbered list. It used to take the entity with the most features without saying so.
  • Spellings: "customer id", "customer_id", "customerid" and "CustomerID" all name CustomerID. The exact entity wins over a wider key (Category,CustomerID).
  • Replies: the answer to "which entity?" stays in the flow even when it ends with "?". Naming another entity instead of "ok" restarts the proposal on that entity.
  • Fixed text: the feature proposal and the entity question are written by code. The LLM rendering announced the user's entity over another entity's features and invented "additional features".
  • Scoped dataset lists: "datasets for CustomerID" and "what are these datasets?" (the conversation's entity) list that entity's datasets only (skill_list_datasets(entity=, data_domain=)), with "N of M" in the header.
  • Narrowing follow-ups: a short follow-up that only narrows ("I meant for CustomerID?", "what about Category", "pour CustomerID") redoes the previous request (dataset list, build, feature listing, or the same question on a feature) for that entity, without a classifier call. A longer question is classified on its own.
  • Better logging:
  • _dataset_kind_tags logs a failed read of the build table as a warning. Every dataset showed "time unknown" with no trace of why.
  • agent/graph.py defines the logger its except branches used. It was never defined, so reaching one raised a NameError.

Dataset builder: existing datasets and samples

  • An Existing datasets card lists the registered datasets (by default those on the selected entity), with their kind, feature count and entity.
  • Sample 20 rows works on any registered dataset, and on the dataset being defined before it is created: the generated query runs with SAMPLE 20 and nothing is created. Only registered datasets can be sampled by name. Routes: GET /ui/dsbuild/datasets, GET|POST /ui/dsbuild/sample.

[0.3.3.2] - 2026-10-13

Dataset builder panel (admin app, Explore)

  • The flow: pick an entity (grouped by data domain, with its key columns and existing datasets), then its features, then a view name, then the kind of dataset; then create it.
  • Feature definitions on hover: the description, the producing process, the last run and the source of the description. A feature computed by several processes has a list to pick one (by default, the one that ran last).
  • Kinds:
  • snapshot: current, as of a date, or following a TimeManager;
  • historical: start / end / duration / delay, or the full history;
  • join: inner or full outer;
  • documentation: optional, and only when an instruct model is configured.
  • Name check: Create dataset stays disabled while the name is taken, either by a registered dataset or by any table or view in the view database. The server checks again before creating, so nothing is ever replaced. Show SQL previews the view. A selection that matches an existing dataset's features is flagged.
  • One creation path: creation goes through the chat's skill_build_dataset, which gains join_type, document, the history delay and the snapshot kind time_manager. Existing calls are unchanged.
  • Routes: /ui/dsbuild/entities, /features, /name and /create (with ?preview=true for the SQL only).

Feature catalog: the SQL chain down to the sources

  • The SQL of a process is now its chain. It starts with the process view (the result), then every view it reads, each before the views it reads, down to the source tables. Each step names the step(s) that read it.
  • Where it stops: at source tables, dataset views (feature store), Time/FilterManager views (including a soft clone over another manager's _HIDDEN table) and FS_T_* / FS_V_* objects. These are listed as inputs.
  • Where it comes from: the persisted lineage graph, read once per export, with one SHOW VIEW per distinct view. A process absent from the graph is walked from its SQL, with a hint to run rebuild_lineage_graph().
  • Phantom nodes: lineage nodes that are not database objects (SQL tokens such as APPLY, 1) are left out and named.
  • Colours. Each step has its own colour, and each input kind has its own colour.
  • In the app, references are tinted and clickable (they open the step).
  • In the HTML export, they are links.
  • In Excel, they are bold rich text in the target's colour, the line links to the step, the step header rows are filled, and there is a legend on About. openpyxl>=3.1 is now required.
  • New functions: catalog_export.process_sql_chain (pure), read_sql_chain, chain_text, reference_spans. highlight_sql_html(refs=) gains the reference marks.
  • Panel fix: long values in the side panel's detail tables now wrap instead of scrolling the panel sideways.

[0.3.3.1] - 2026-10-13

Feature catalog panel and export

  • Horizontal scroll bar with the side panel open (reported issue). The table used to shrink into the space left by the side panel, and when it did overflow, its scroll bar was under the last row, below the window. It now keeps a minimum width (1,100 px) and scrolls inside a box sized to end above the bottom of the window (refitted when the side panel opens or closes, and on resize), with a sticky header.
  • Filter by process. A Processes multi-select (the processes of the selected domains, with their feature count) and a Show only this process button in the side panel. The exports follow the filter.
  • export_feature_catalog(..., processes=) and feature_catalog_documented(..., processes=): a PROCESS_ID or view name (qualified or not, any case), or a list. An unknown one raises with the existing views. The About sheet / HTML header names the processes kept and counts what is kept. None (the default) gives the same output as before.

[0.3.3.0] - 2026-10-12

Charts drawn in the database by TD_PLOT, from Python and from the chat: line, step history, bar, histogram, scatter, panels and correlation heatmap, with entity selection, zoom and download. Feature registration no longer leaves a tdfs_tmp_* table behind.

In-database plotting (tdfs4ds.plotting)

  • plot_feature(name, kind='distribution'|'history', entity=, data_domain=, entity_values=, process_id=, as_of=), plot_dataset(view, y, x=, series=, kind=, agg=), plot_in_database(source, x, y, ...) and show_plot(result), exported on tdfs4ds. The chart is drawn by the UAF function TD_PLOT; only the image (tens of kB) leaves the database, whatever the number of points.
  • How: count the rows (refused above PLOT_MAX_POINTS, 1,000,000), copy them to a volatile table (x/y/id, no CREATE right needed), keep the first PLOT_MAX_SERIES (12) series and say so, run TD_PLOT, drop the tables. Requests of at most PLOT_INTERACTIVE_MAX_POINTS (2,000) points also get an interactive Plotly version; above that the rows are never read back.
  • Feature plots read the storage table (FS_T_*, FEATURE_ID + FEATURE_VERSION): a histogram at the valid time, or the history (a daily feature on its date column, otherwise on ValidStart). A name used by several entities or domains is refused with the candidates; with several producing processes the most recently written one is plotted, and the result says so.
  • Dataset plots take their defaults from the dataset catalog (y = a feature, series = the entity columns that are not dates, x = the date column or ValidStart).
  • Found live on Vantage 20, built into the SQL: a volatile table is a valid TABLE_NAME; dates print only with XFORMAT('YYYY-MM-DD') (%Y-%m-%d is Error 9153, no format prints 1, 2, 3); a statement-level temporal qualifier over derived tables only is Error 9330, so a temporal histogram first copies the raw values; one series per category draws unreadable bars, so a category bar chart is one series over positions 1..k with the names in the axis label. 1.5-2 s per chart end to end; TD_PLOT drew 200,000 points in 0.5 s.
  • PLOT_PLOTLYJS = 'cdn' (default): the interactive version loads plotly.js from its CDN (inlined, every chart would weigh 4.8 MB in the chat's conversation memory); 'inline' for offline use. The TD_PLOT image does not depend on it.

Panels, series choice, step histories

  • Several panels in one image (one TD_PLOT call, one SERIES_SPEC per panel, LAYOUT(columns, rows), CELL(column, row); line and bar panels mix - verified live): plot_features([...]), plot_dataset(y=[...]), plot_panels([panel(...)], layout=). At most PLOT_MAX_PANELS (6); a panel with nothing to draw is left out and named, the others are drawn.
  • Which series: max_series= and series_order='largest' (default: by latest value), 'smallest', 'alphabetical', 'most_points'. The default was alphabetical, which drew an arbitrary 12 of 1,958 customers.
  • Which entities: entity_values=[...] (a value matches the whole key or one of its columns); feature_entity_values / dataset_entity_values say which words of a question are real values.
  • A history on valid time is a step: each value from ValidStart to ValidEnd (to now when open). It used to join the starts only, so a value that never changed after March looked as if it stopped in March. Each end point is one second early: with the end of one period and the start of the next at the same instant the line went back and forth (a sawtooth, seen live).
  • The chat frame follows the image height (two stacked panels were cut off).

Correlation, zoom, download

  • plot_correlation(view, columns=None): all Pearson correlations in ONE pass (CORR), the k x k matrix built from that row, drawn by TD_PLOT as a mesh heatmap over a MATRIX_SPEC (colour bar -1..1, verified live); r['matrix'] holds the values. In chat: "correlation matrix" / "heatmap" / "matrice de corrélation".
  • Zoom: xlim= / ylim= on every plot function. A time window filters the rows in SQL (TD_PLOT refuses XRANGE on a time axis, Error 3706, measured); a numeric one also sets XRANGE; step histories are cut at the edges; YYYY-MM as an upper bound includes the month. In chat: "between A and B", "from A to B", and a follow-up "zoom from A to B" redraws the previous chart (remembered as _last_plot). Window dates no longer become the conversation's valid time (they did, and the dataset catalog was then read before the dataset existed - seen live).
  • Download and actual size: each chart page has Download PNG, Download SVG (an SVG copy for requests up to PLOT_SVG_MAX_POINTS, 20,000) and Actual size; the chat frames are now sandboxed allow-scripts allow-downloads (still no same-origin). save_plot(r, path) writes .png / .svg / .html.
  • TD_PLOT limits measured and handled: text arguments over 127 characters (a 7-column correlation label) and images under 400 x 400 pixels are refused with a bare Error 3706 - texts are cut, sizes raised; a date and a time cannot share one XFORMAT (Error 9153), so the format follows the span (HH:MI within a day with the date in the label, YYYY-MM-DD, YYYY-MM over ~13 months) - a history written in one day printed the same date on every tick.
  • Tests no longer reach the configured instruct model (the repo .env sets one; a test went through the LLM fallback and was flaky). 92 offline plotting tests.

PLOT skill in the chat

  • New ca-plot skill (intent PLOT): "plot the distribution of X", "show X over time for the Y entity", "plot A by CATEGORY from DS", "plot A against B in DS", "trace l'évolution de X". The request is read by code (named dataset or feature, its named columns, plot words) and every column is checked against the object; the answer is fixed EN/FR text. An ambiguous feature name lists its entities and shows how to ask.
  • A question with a plot word and no lineage word is routed to PLOT by code, whatever the classifier says.
  • MCP plot_* tools are no longer bound (mcp_connection.without_client_side_plot_tools): they read the rows out of the database to draw them client-side.
  • Charts carry a height; both chats size their frame with it.
  • "plot A and B over time" draws panels; "top 5", "the 3 smallest", "les 5 plus grands" choose the series; entity values in the question restrict the chart.
  • A request that names nothing is read by the instruct model against the store's datasets and features (JSON with a schema); invented names are dropped, and the answer says how the request was understood. No model: the agent asks for a name.
  • A name shared by several data domains with the same entity asks for the data domain (it asked for the entity, which the question already gave).

Leftover tdfs_tmp_* tables of feature registration

  • GetAlreadyExistingFeatureNames dropped its temporary table with tdml.execute, which teradataml does not have; the AttributeError was swallowed, so every feature registration left one tdfs_tmp_<12 hex> table in the store's database (250 in the demo store). Fixed (tdml.execute_sql, a failure logged at DEBUG); live-checked: a registration no longer leaves one.

  • tdfs4ds.cleanup_temp_tables(dry_run=True, database=None, min_age_hours=1) lists those tables (exact name pattern only, older than an hour) and drops them with dry_run=False; a refused DROP is reported per table. Guarded by tests/test_temp_table_cleanup.py.

  • Guarded by tests/test_td_plot.py (67 offline tests). Live-verified on DB_LAB through the agent: 7 plot questions (dataset line, bar by category, scatter, feature distribution with two producers, daily feature history in French, ambiguous name, history for a named entity) and a lineage question still routed to LINEAGE.

[0.3.2.5] - 2026-10-12

The chat creates, registers and documents a dataset, asking its kind and parameters; dataset lookups match features by store; the feature catalog panel handles a feature computed by several processes and highlights its computation in the process SQL.

Datasets from the chat

Seen live: "where can I find X?" counted features of an entity name across domains; "what are the available datasets?" was swallowed by a waiting dataset flow and answered with a feature list; that flow never built anything ("ask your data team"); and the MCP sample query used bare dataset names (Error 3807).

  • The agent creates, registers and documents the dataset, only when the requested one does not exist. The database decides permissions: a refused CREATE VIEW is shown with the database's own message.
  • It asks the kind and its parameters: a current snapshot, a snapshot at a date (build_dataset(as_of=)), or a history over a window (build_historical_dataset: "last 90 days", "les 6 derniers mois", "from … to …", "all"). A missing parameter is asked again by name.
  • Timestamps: ISO dates and times, T, Z/UTC/GMT, +02:00/+0200, and textual dates in English and French. No offset means UTC, and the answers say "UTC". Slash dates (03/04/2026) are refused as ambiguous, and an impossible date is refused. A date typed as a parameter never becomes the conversation's valid time.
  • Name check: taken in the dataset catalog, or any object of that name in the view database (DBC.TablesV); an unreadable DBC stops before building.
  • The result: registration is read back from the catalog, and the documentation status is reported. The vector index is not rebuilt (it took minutes live); the answer says the dataset reaches semantic search at the next refresh. build_dataset failures that are logged rather than raised are captured and reported.
  • Lookup by identity: datasets are matched on (FEATURE_DATABASE, FEATURE_ID), since a FEATURE_ID is unique only within one store. A dataset reading another store's features no longer matches a local feature with the same id (found live; the catalog export had the same defect, also fixed). Found datasets show their entity and domain; a failed read is reported as such, never as "no dataset".
  • Dataset inventory: "what are the available datasets?" (EN/FR) lists every dataset with its entities and kind, the kinds read in one query (5 s live, was 55 s).
  • Leaving the flow: a new question while a dataset step waits is answered and the flow dropped; "create it" inside the lookup flow moves to the build flow.
  • MCP: the sub-agent gets qualified dataset names and a dataset → feature map, and a tool error (without driver frames) goes back to the model, which corrects and retries once instead of the turn failing.
  • describe_db_error also drops SQLAlchemy's "Background on this error" link.
  • Guarded by tests/test_agent_dataset_build.py (52 offline tests). Live-verified on DB_LAB (as-of and history datasets built, registered and documented, then removed); the MCP path was not verified live.

Feature catalog panel

  • A feature computed by several processes has one row per process, each marked "1 of N producing processes"; the side panel names the others, and each opens its own SQL.
  • The selected feature in its process:
  • the side panel shows the process's data domain and entity;
  • a "Computed as" block gives the feature's expression at each level (outer SELECT item, then the derived tables / CTEs it reads) and the grain;
  • in the SQL the computation is underlined and every mention of the name is marked; a link scrolls to the computation. The name is also marked in the process's feature list and business logic.
  • This comes from sql_context.computation_context, whose levels now carry their character span, and from catalog_export.highlight_sql_html (pure, escaped).
  • One catalog read at a time. A page load and a filter change building at once (two threads through teradataml's shared pool) left the panel loading. The panel's reads are now serialised, and the page ignores a response older than the last request.

[0.3.2.4] - 2026-10-12

Finding the databases that hold a feature store, connecting to a store you may only read, the classic chat matching the app chat, suggested questions read from the connected store, updated documentation skills, and the feature catalog exported with its processes and documentation.

Feature catalog export (XLSX, HTML) and panel

  • tdfs4ds.export_feature_catalog(path=None, *, fmt=None, as_of=None, data_domains=None, with_sql=True) writes each feature joined with its producing process and its documentation:
  • XLSX sheets: About (store, valid time, domains, counts, links to the sheets), Features (one row per feature × producing process), Processes and SQL.
    • Features and Processes are Excel tables. Features are grouped by domain, entity and process, with the first three columns frozen.
    • A process name links to its SQL. The SQL sheet holds each process's SQL one line per row, with links back to its first feature and to its process.
  • HTML: one self-contained page, the features table linking to each process and its SQL.
  • DataFrame: tdfs4ds.feature_catalog_documented(...) returns the Features table.
  • New module tdfs4ds/catalog_export.py; openpyxl is now a dependency.
  • Columns: feature (domain, entity, name, id, type, storage table), its description with provenance (manual / imported / propagated / generated, editor, date), the process (id, view, type, FilterManager, TimeManager, business logic, entity description), the datasets that use it, the last run and its status, and the EXPLAIN scores.
  • Missing values: a value that is not available reads unknown (e.g. no EXPLAIN score without an LLM); a feature in no dataset reads none, a process without a run never run.
  • Link to the process: a feature is linked to its processes by triplet through catalog_counts' producers, never by name within a domain. A feature with several producers gives one row per process, one with none a no producing process row.
  • Filters: data_domains = all, one or a list (case-insensitive; an unknown name raises with the existing ones). as_of reads the catalogs, business dictionary, process documentation and provenance at that valid time (base tables, VALIDTIME AS OF), and the last run up to that time.
  • The SQL is today's text, since view definitions are not versioned; an export at a past date says so.
  • Long SQL: one line per row in the SQL sheet. A single line longer than an Excel cell (32,767 characters) continues on the next rows, marked continued, never cut.
  • Reads: batched (one query per documentation table, 500 names per IN-list), plus one SHOW VIEW per process view. Live on the demo store (67 features, 13 processes): about 6 s.
  • Admin app, Explore › Feature catalog:
  • 20 features per page, filtered by data domains (multi-select), date and text;
  • a row opens a resizable side panel with the feature, its process and the process SQL, read on demand;
  • Export HTML / Export XLSX download the file for the current filters;
  • visible in the user view too.

Chat

  • The classic (Gradio) chat now matches the app chat: the app's colours (light and dark), the whole page width, the conversation in a centred column with no box around it, the suggestions as pills under the heading, a single-pill composer, a full-width answer bubble and the plain orange avatar. 0.3.2.3 had the same parts but Gradio's own layout (a narrow page, a boxed conversation, suggestion tiles at the bottom).
  • Suggested questions come from the connected store and follow a switch. They used to be read from the agent's per-turn cache, which outside a turn could still hold the previous store, and the app fetched them once. chat_suggestions() (in agent/chatbot.py, shared by both chats) now runs three small reads (current processes, last completed run per process, documented columns of their views, about 0.7 s) and names a documented feature whose process ran most recently, with its data domain; "Is X up to date?" is offered only when that process has run, "Which data domains exist?" only when there are several. The app reads them again each time the chat is shown and after a store switch in Parameters or Connect; the Gradio chat at each page load.

Finding feature stores, and connecting to one you may only read

  • tdfs4ds.find_feature_stores(complete_only=False, details=True, databases=None) lists the databases that hold a feature store, as a DataFrame: DATABASE_NAME, VIEW_DATABASE (found from the view's text for a store set up with database_view), COMPLETE / MISSING (the tables connect() needs), and with details READABLE, N_PROCESSES, N_DATA_DOMAINS, ERROR (the whole refusal). One read of DBC.TablesV, so only what the session may see, plus one count per store (about 1 s for four stores, live). New module tdfs4ds/store_discovery.py.
  • Parameters panel: a list instead of free text. The stores are looked for once at app startup (and after a login in Connect); Refresh list searches again. Each entry shows its process and domain counts; a store connect() would refuse (incomplete, or its catalog unreadable) is shown but cannot be chosen, with the reason as its tooltip. Free text comes back only when the search itself failed.
  • connect() works on a store the user may only read. It refreshes the business-dictionary views with REPLACE VIEW, which needs the DROP right; a refusal raised, so a lab account could not connect to the production store it may read (seen live on DB_PROD). The existing views are now kept with a warning (is_access_denied_error, matched on the phrase "does not have ... access to", never on a code). The refresh of objects that already exist (COMMENT ON, REPLACE VIEW after an idempotent CREATE) no longer logs ERROR for the same refusal.
  • connect() reads DBC with raw SQL. DataFrame.from_query over DBC.TablesV / DBC.ColumnsV creates a view, which needs WITH GRANT OPTION on DBC (Error 3523 for that account), and the fallback then listed the whole database.
  • A refused store switch in the app leaves the session where it was. connect() sets SCHEMA before it can fail, so a failed switch used to leave the app half on the new store; the previous settings are now restored. The message no longer carries the driver's Go stack frames, and a failed index rebuild after a good switch is reported as such instead of as a failed connection.
  • Skills. fs-document says which way to document (at registration, one process, a dataset, a plain view, the whole store with rebuild_documentation, after editing views, by hand) and which of them refresh the vector index; its document_view_incremental example had the wrong signature (it takes the database and the view). fs-vector-store describes the Teradata refresh as it is since 0.3.1.169 (in place, then destroy + recreate, never add_datasets) instead of an unsolved limitation, with a table of what refreshes the index. fs-setup shows find_feature_stores().

[0.3.2.3] - 2026-10-12

Chat fixes, documentation of deep (historian) tables, and the time-series demo track.

Chat fixes

  • Table headers readable in the light theme. A model writes bold headers (| **date** |), and the chat's bold text is navy, the colour of the header background in the light theme, so the headers disappeared (dark theme was fine). Header text is now white whatever it holds, in the admin app's chat and in the other panels that render model Markdown.
  • Light theme background. --td-soft referred to itself (var(--td-soft)), which is invalid: zebra rows, quotes and the edit log had no background in the light theme.
  • Resizable context graph. Drag the left edge of the panel (it keeps the chat at least 420px wide); the width is remembered per browser, a double-click restores 440px, and the graph refits after a resize.
  • Copy chat. A header button copies the whole conversation as Markdown (questions, answers and errors). Copy falls back to a hidden text area where the browser clipboard is unavailable (an http:// page behind a proxy), which also fixes the per-answer Copy button there.
  • The classic (Gradio) chat looks like the app chat, at /chatbot, on the standalone port and in notebooks (launch_chatbot, launch_chatbot_with_index): the same header buttons (Context graph, Copy chat, Clear, New chat), bubbles, avatar, suggestions in the empty conversation, pill composer with a Send button, and the context graph in a side column. Its style is now actually applied: Gradio 6 ignores css/theme on gr.Blocks, so they are passed to launch() and mount_gradio_app() (chat_style_kwargs). Gradio's own copy-all button is on. It follows the system light or dark theme. Behaviour (thread, MCP toggle, trace) is unchanged.

Deep (historian) tables

A historian stores one row per id, timestamp and tag (sensor, signal), the value in a payload column. Its tags can now be documented as if they were columns of the pivoted table (tdfs4ds/genai/deep_tables.py). No new table, no ALTER; a table without a layout behaves exactly as before everywhere.

  • set_deep_table_documentation(database, table, *, id_columns, tag_column, payload_column, timestamp_column, timestamp_timezone='UTC', description, columns, tags, ...). The layout is checked against the table's columns and stored as the reserved business-dictionary section DEEP_LAYOUT (JSON), the FEATURE_GROUPS precedent; it is required once and read back afterwards. Each tag is a FS_BUSINESS_DICTIONARY_COLUMNS row of the table, with the usual provenance (a tag list loaded with source='IMPORTED', a correction MANUAL), log, index and protection from LLM runs. Tags come from a dict or a DataFrame with an optional unit column. A tag that is also a physical column, longer than 128 characters or not Latin is refused with the reason, nothing written.
  • deep_table_tags(database, table, since=, until=, documented=) lists the tags present in the table (one GROUP BY, bounded on the timestamp), marked documented or not. It reads the table, so it is opt-in.
  • The LLM receives only the tags a process's SQL names. When documenting a process or an intermediate view that reads a deep table, its context holds the layout, the physical columns and the tags whose value appears as a literal in the SQL, whatever the shape (CASE WHEN, IN, PIVOT, join condition). When none is named, at most DEEP_CONTEXT_MAX_TAGS (40) are listed. A tag compared with the tag column but undocumented is logged. The prompt for a wide table is byte-identical (tested).
  • A tag edit's stale list is the processes whose SQL (or an upstream view's) names the tag; with regenerate_dependents=True only they are re-documented.
  • Agent: a DEFINITION of a tag starts with a fixed sentence (EN/FR) saying it is a tag of the historian, its readings in the payload column, by the id columns and the timestamp.
  • Admin app: Objects > Edit on a deep table shows the layout, the columns and a Tags group (filter, pages of 200, Add a tag); each tag is corrected, protected or released on its own, with the usual preview and confirmation. The coverage panel says "Deep table · N tags documented".
  • Reserved sections are excluded by every reader through one list (RESERVED_SECTION_NAMES): the BD sections view behind the Teradata vector store, the Chroma index, the coverage panel. On a store without a layout row the results are unchanged. An older client sharing a store with a deep table shows the layout JSON as a section and gives its LLM every tag.
  • Fixed on the way: a business-dictionary OBJECT_TYPE reads back padded ('T '), so source tables were shown to the LLM as "Object" instead of "Table", and legacy source-table documentation was labelled "generated or unknown (before 0.3.2.1)" instead of "imported (before 0.3.2.1)". It is now stripped at the read.

Demos: time-series track (--timeseries)

A risk desk watches contagion: when NVDA goes through a stress episode, how did the other holdings behave over the same sessions? Five stocks (2013-2018) from the Kaggle dataset "S&P 500 stock data" (CC0), in demos/data/stock_prices_daily.csv with its source in demos/data/README.md, loaded by Setup.ipynb into a deep PTI table DB_SOURCE.MARKET_PRICES (historian layout: PRIMARY TIME INDEX (TIMESTAMP(0), ..., DAYS(1), COLUMNS(TICKER, TAG)), UTC timecode). Data domain Market Stress Episodes.

  • T1 reads the table and documents it as a deep table (tags imported, one corrected, the undocumented one found).
  • T2: a TimeManager starting in the past; process 1 finds episodes with DELTA_T (CTE, LAG, start/end flags, as a historian session view); process 2 reads the stored episode from a dataset attached to the TimeManager and measures every stock over it (min, max, average, std, drawdown, return from the close before the shock, relative to the reference); roll_out over 14 trading days; values equal a pandas reference at sampled dates.
  • T3 (instruct model): the LLM's context holds only CLOSE of the five tags; the agent frames a tag.
  • Diagrams and plots. Three generated diagrams (demos/diagrams/make_diagrams.py): historian (deep rows read as a pivoted table), delta_t (NVDA's real August 2017 returns, the start/end events and the restart rule), stress_pipeline (the two processes, the feature store and the TimeManager). Three matplotlib plots, drawn from data read back from the database (dc.plot_market_signals, dc.plot_stress_episodes, dc.plot_contagion): the five stocks over 2017 with NVDA's event days, the stored episodes on NVDA's closes, and every stock over the last episode with its return relative to NVDA.
  • Learned on the database, recorded in the notebooks: DELTA_T((WHERE ...), (WHERE ...)) is the syntax; a later start before an end restarts the episode; under GROUP BY TIME a timecode predicate must be a literal; a PTI cannot also be row-partitioned; a PTI reads a time range only with literal bounds (run-time bounds give a scan, a period join gives a product join); DELTA_T's TIMESTAMP(6) WITH TIME ZONE bounds refuse a CAST to TIMESTAMP(0) (Error 7454).

[0.3.2.2] - 2026-10-11

Release of the 0.3.2.1 work below (0.3.2.1 was not published): documentation written by hand, the provenance and documentation-log tables, LLM runs that keep human text, the admin app's edit with preview, feature descriptions filed under the right entity, and broken Vector Store collections recreated. No other change.

[0.3.2.1] - 2026-10-11

Writing documentation by hand

Features, processes, datasets and source tables can be documented by hand, from Python or from the admin app, with no LLM. With an LLM the LLM writes first and a person corrects; the correction survives the next LLM run. Without an LLM this is the documentation. An LLM configured later documents only what nobody wrote.

  • API (tdfs4ds/genai/manual_documentation.py, exported at top level):
  • set_feature_documentation, set_process_documentation, set_dataset_documentation, set_object_documentation;
  • propagate_documentation: dataset columns filled from the feature documentation, no LLM;
  • get_documentation, mark_documentation_manual, release_manual_documentation, documentation_log.
  • Where a feature edit is written. A feature's description is written in both places it is read from: the process documentation, which datasets inherit, and the business dictionary on the process view, which the agent, the index and the app read. It is copied to every dataset exposing the feature.
  • What each call returns. A report of what was written and copied, the human copies left alone, the LLM-written documentation that the edit makes stale (regenerate_dependents=True regenerates it when a model is configured), and the vector index refresh. dry_run=True returns that report without writing.
  • Feature identity. A feature is resolved as its (data domain, entity, name) triplet. A shared name raises with the candidates; when data_domain is omitted, tdfs4ds.DATA_DOMAIN decides.
  • Who wrote what, in a new optional VALIDTIME table FS_DOCUMENTATION_PROVENANCE:
  • values: IMPORTED (a dictionary load: the right tag for source data), MANUAL (a person's edit), PROPAGATED (a copy). No row means the part was written by the LLM, or before 0.3.2.1;
  • author: the database USER and a declared editor name.
  • What was done, in a new optional FS_DOCUMENTATION_LOG. One row per action (manual edit, import, propagation, LLM documentation, protect, release, index refresh) records:
  • its status (a failure keeps its whole error) and where it came from;
  • the RUN_ID / PROCESS_ID that join it to FS_FOLLOW_UP.

Documentation events are deliberately not written to FS_FOLLOW_UP: every reader of that table treats a row as a process run, and run() skips a computation that already has a COMPLETED row. - No ALTER. Both tables are created by setup() / connect() when missing, and the six documentation tables are untouched. A user who cannot create them still documents; the edits are then neither protected nor logged, and a warning says so once. - LLM runs keep human parts. - Writers: upload_documentation, _store_dataset_doc and _store_intermediate_view_doc leave IMPORTED / MANUAL parts as they are. A protected feature is not expired by the "features no longer documented" step. - Prompt: document_process_incremental gives the human text to the LLM as authoritative context, and lays it over the LLM output so the change detection compares the final text. - Overriding: overwrite_manual=True (new keyword) on document_process_incremental, document_dataset_incremental, rebuild_documentation and cascade_redocumentation lets the LLM replace them. - upload_business_dictionary_* gain source= (default 'IMPORTED') and author=. source=None records nothing, and the LLM pipelines and the migration use it, so their SQL is unchanged. migrate_objects carries the provenance rows of the objects it promotes. - Admin app (Documentation › Objects › Edit documentation, admin view): - one field per part, each with who wrote it; - a required editor name; - a preview dialog (what is written, copied, kept, made stale, re-indexed) before Confirm and save; - options to copy, overwrite, re-index or regenerate with the LLM (disabled with the reason when there is no model); - Protect / Release per field, and the object's history.

The Coverage panel says who wrote the documentation, and the Report has a Documentation activity section. - Demos: - 01 loads a data dictionary for the source table; - 02 documents features and a process by hand; - 03 documents a dataset without an LLM; - G1 corrects what the LLM wrote and shows the correction kept.

01–03 were run with and without an instruct model.

Fixed

  • rebuild_documentation(force_update=False) documented only the gaps in name. It re-documented every process that was a graph root or absent from the graph, whether or not it was documented. Documented processes are now skipped, as its docstring said. This matters for a store documented by hand before an LLM is configured.

  • The coverage panel's hint named tdfs4ds.upload_business_dictionary_objects, which was not exported at top level; it is now, and the hint names set_object_documentation.

  • upload_documentation filed a feature's description under the other entity's FEATURE_ID too. It resolved FEATURE_ID by name and data domain only, so a domain holding one name for two entities got the description under both ids. Read back by name, the extra row could hide a hand correction. The MERGE and the expire step are now restricted to the process's own ids (name, domain and entity, compared order-insensitively); when they cannot be resolved the SQL is unchanged. The expire step then closes the misfiled rows of each process it re-documents.

  • Existing stores: tdfs4ds.genai.repair_feature_documentation_entities(dry_run=True) lists those rows, and dry_run=False closes them (history kept). It found 11 on the demo store.
  • A Teradata Vector Store collection whose own backing object is missing (Object 'collection_<name>_index' does not exist) was reported as "service unreachable or not enabled … check licensing". _is_broken_collection_error now classifies it. build_teradata_index recreates such a collection, or says that an admin must when the destroy is refused. Seen on the demo store: all three collections had lost their _index table; they were recreated and search works again.

[0.3.2.0] - 2026-10-11

A chat built into the admin app, a user view, and Opik tracing

  • Native chat panel. The admin app's Chatbot panel used to be the Gradio chat in an iframe. It is now part of the app (serve/chat.py, /ui/chat/*, SSE). It has:
  • streamed answers with a live status line;
  • Markdown rendered server-side, with charts inside the answer;
  • Copy, and a folded Sources line;
  • suggestion chips named after the connected store's own objects;
  • Clear vs New chat;
  • one conversation per browser.

Turns are answered one at a time, as Gradio's queue did, because a turn sets process-wide state. The Gradio chat stays at /chatbot and on its own port. - User / Admin view. A switch in the top bar, remembered per browser. The first view comes from TDFS4DS_SERVE_DEFAULT_VIEW (admin by default, so nothing changes for an existing deployment). - The user view shows the conversation and the Chatbot, Report, Lineage and Objects panels. - The admin view adds, under each answer: intent, skill, domain, timed steps, MCP tool calls, tokens and the full trace. It also has a context-graph side panel, and every panel. - The switch is not access control. - Dark theme for the shell (system preference, or the ◐ button), and a phone layout. - Opik tracing (tdfs4ds/utils/tracing.py), opt-in: - Enable it with TDFS4DS_TRACING=opik and pip install tdfs4ds[opik]. - An OpikTracer callback in the LangGraph run config of the consumer agent and the query optimizer gives one trace per turn, with a span per node and per LLM/tool call, grouped by thread_id. - Opik's own settings decide where traces go. tdfs4ds never reads OPIK_*. - The Opik SDK's analytics (stats.comet.com) and Sentry reports are switched off unless TDFS4DS_OPIK_VENDOR_TELEMETRY=true. Seen live: with its defaults the SDK posted to stats.comet.com on the first traced turn. - Tracing never breaks a turn. - check_configuration has a new Agent tracing row. - AgentOps packages trace with Opik. config.yaml carries TDFS4DS_TRACING: "opik" unless the session says 'off', and opik in requirements.txt is now used. The entrypoint stays a bare coroutine: the tracer is a callback, not a decorator.

Fixed

  • Token counts were always 0.
  • The cause: consumer_agent() / consumer_agent_stream() read input_tokens / output_tokens from TokenTracker.totals(), which returns total_input / total_output, and re-added the whole session's by_skill on every turn.
  • Missing calls: only _llm_call recorded usage, so the intent classifier and every structured-output call were never counted.
  • The fix: a turn now records every model call through a callback, labelled by graph node. _llm_call tags its own call so it is not counted twice. Measured on the demo store: 3.8k tokens for a turn that showed 0.
  • A list that opens indented ("Total\n - a\n - b", the shape of the data-domain answer) rendered as a code block in the admin panels; it is a list again.
  • The Gradio chat's Clear / Reset now also clear the context graph.

[0.3.1.241] - 2026-10-11

Agent evaluation grounded in the store, and challenged on time

Scenario generation filled fixed templates with the first catalog rows: every skill asked about the same feature, six intents had no scenario, FEASIBILITY asked healthcare-fraud questions whatever the store held, and FRESHNESS, EXPLAIN, USAGE and feature LINEAGE carried no fact the judge could check. The facts were frozen at generation and not covered by the scenario hash, and the counts were computed by a second definition.

  • Questions chosen from data (agent/eval_scenarios.py): a feature is picked because it fits the skill (documented, produced by a process that ran, read by a dataset, registered under several entities), deterministic for a seed. Every intent is generated or listed in meta["coverage"] with the reason (BUILD_DATASET needs include_writes=True, ALGORITHM needs MCP). SEARCH and FEASIBILITY questions are built from the words of the store's own documentation.
  • Facts from the Report's authorities (agent/eval_facts.py): counts from catalog_counts, freshness in business dates with the TimeManager's date and the dates behind, lineage from the persisted graph, dataset membership by FEATURE_ID, EXPLAIN grain and filters from the view's SQL. Each turn keeps a fact_spec; run_evaluation(refresh_facts='check') rebuilds the facts and flags turns whose facts no longer match the store ('refresh' judges on the rebuilt ones). The scenario hash covers the facts.
  • Time scenarios (category temporal), with dates taken from the store's history: the catalog at a date between two registrations, before anything existed, a date kept by a follow-up then cleared by "now", a feature that did not exist yet, freshness and lineage at a past date, a dataset's own time, a year alone.
  • A judge with the agent's time rules. Each turn carries time_context, computed with the agent's own _resolve_as_of; the agent's trace gains an **As of** line and the two are compared (as_of_match). Names that exist now but not at the turn's date (absent_names) are violations even though they are real. Checks done by code: names, counts (digits or words, EN/FR), dates (ISO or textual), violations. Faithfulness is capped at 2 on a violation, completeness at 3 when a dated answer does not state its date, robustness at 2 when the agent answered at another valid time. The report gains a Time section, the coverage list and the STALE flags.
  • The admin Evaluation panel passes the time, objects/columns, MCP and seed options; the generator before this release is removed.

Agent - EXPLAIN states the grain and the filters

An EXPLAIN answer gave the formula but not the aggregation it sits in nor the rows it is computed on. agent/sql_context.computation_context reads them from the process view's SQL by code (reusing the sql_qualify tokenizer): GROUP BY keys or the window, WHERE / HAVING / QUALIFY, inner joins (rows without a match dropped), semi-joins, outer joins, a WHERE on a LEFT join's optional side (which makes it inner), and the TimeManager / FilterManager the view reads, following the expression through derived tables and CTEs. Every EXPLAIN answer ends with a fixed "Computed at / on which rows" block; the columns used to filter or join are listed and drillable.

Lineage at a past date, manager roles, datasets always in the graph

  • load_lineage_digraph / get_ancestors / get_descendants / get_persisted_graph accept as_of: the base table read VALIDTIME AS OF, primary store only, never the on-the-fly build (today's DDL). The agent's LINEAGE answers at a past date from it, and says "no lineage recorded at that date" for an object absent then.
  • TimeManager / FilterManager views and their tables are tagged with a role when the graph is loaded (stored OBJECT_TYPE unchanged); the agent names them as drivers.
  • drop_dataset closes the dataset's lineage edges (lineage_store.remove_object_lineage; history kept, purge removes it). restore_dataset_catalog records the lineage of the datasets it registers or repairs, and reports catalogued datasets absent from the graph. The consistency model gains dataset_not_in_lineage. A guard test checks that no package path builds a dataset with record_lineage off.

Admin app

  • Lineage panel: a new default Flow layout replaces the Sankey (unreadable on large workflows): left to right, zoom and pan, the whole path of a clicked object highlighted and its documentation shown below, search, filters (hide source tables, hide managers, processes and datasets only), managers drawn as drivers. Offline (vis-network inlined from pyvis). plot_lineage_sankey remains in Python.
  • EFS map: a Time drivers section, the same as the Report's.

Agent defects found by the new evaluation (live, demo store), fixed

  • USAGE invented datasets ("_Live", "_History") for a feature no dataset reads, and matched datasets by feature name across entities. Membership is now matched by FEATURE_ID and storage table, and the prompt says explicitly when no dataset exposes the feature.
  • SEARCH answered "0 features registered" when the session's DATA_DOMAIN was not a real domain (the database name): such a domain now counts as none and every domain is searched.
  • A feature that exists in the catalog but not in the active domain (or, at a past date, without a producing process yet) was answered "not found - did you mean ?". The answer now lists where it is.
  • At a past date, the agent's whole-store scans were memoized without the valid time in their key.
  • A pronoun ("what does it mean?") reached the skill as the object name "it" and answered about another object; pronouns are now a reference to the conversation.
  • LINEAGE fell back to the SHOW VIEW walk (describing storage tables) when the persisted graph had nothing in the asked direction; that graph now answers ("nothing depends on X"), "what feeds it" counts as upstream, and the sources footer names the persisted graph (and its date) instead of an empty SHOW VIEW walk.
  • At a date before any registration, the empty-scope answer named the database instead of the data domain asked.
  • DEFINITION of a dataset now states its own data time (frozen date, TimeManager, history window), given to the model as authoritative context (the stored documentation alone made it say "no fixed period" for a history).
  • A LINEAGE answer from the persisted graph could be a bare view name: the neighbours are now listed by code whenever the model's wording leaves one out, and only real TimeManager / FilterManager objects are called drivers.

Found, not fixed

  • Feature-catalog rows registered during a TimeManager roll-out carry the business date as valid time, processes the real registration time: on a 2023 date the demo store has features but no process. The evaluation reports it as it is.

Fixes

  • Business-dictionary writes turned French accents into e? (_sanitize_for_teradata decomposed with NFKD before the Latin-1 round trip). Text is now composed (NFC); only characters outside LATIN are mapped (œ -> oe, € -> EUR, others to their base letters or ?). Rows written before keep their ? until re-documented.
  • Migration logged "lineage refresh failed" when a dataset of the plan was deferred, skipped or failed: Phase F named it and resolve_objects aborted the whole refresh. Only created objects are refreshed now, and rebuild_lineage_graph(objects=..., require_exists=False) skips unknown names with a warning.

Docs

TimeManager: "Several processes, one calendar: share or clone?" (when to clone, what it solves, the cautions: a soft clone follows the original's dates, a hard clone drifts, diverging cursors mix business dates in a dataset, how to bring processes back). Mirrored in the fs-rollout skill and notebook 05b. Consumer agent: EXPLAIN context, lineage at a past date, evaluation. Skills ca-explain, ca-lineage, eval-inspect updated.

[0.3.1.240] - 2026-10-10

One count of processes and features, in the app and in the agent

A feature is the triplet (data domain, entity, feature name) and a process is a distinct PROCESS_ID. The Report, the Dashboard, the EFS map and the documentation coverage counted process-catalog rows in some places and distinct ids in others; the consumer agent counted the producers of registered features in one data domain, kept the first producer of a feature made by two processes, and let the model write the number below the listing cap. store_consistency.catalog_counts is now the one definition, used by every panel and by the agent, with the same notes (features without a producer, with several producers, processes never run).

Agent - feature inventories rendered by code

The same question on the same store gave a different answer each time. An inventory is now rendered by code: - the counts first (above); - up to AGENT_LIST_MAX_ITEMS features, a table per entity, Business meaning | Technical name (the first sentence of the documented description; an undocumented feature says so); - above it, semantic groups (label, one sentence, technical names). The grouping is proposed once by the instruct model, checked against the catalog (no invented or missing name), stored in the business dictionary (section FEATURE_GROUPS of __SCOPE__<domain>[__<entity>], left out of the vector indexes and of documentation coverage) and reused until the feature set changes. rebuild_documentation() regenerates the groups. Without an instruct model, the features are grouped by producing process, as before.

Agent - questions at a past valid time

A question that names a date ("which features existed on 30 June 2023?", "au 31 octobre 2023") reads the catalogs, the business dictionary and the dataset build tables as they were then (VALIDTIME AS OF on the base tables). The date stays in force for the following turns until "now"; every answer at a past time starts with it. Inventories, definitions, datasets, usage, search and freshness honour it; the other intents say they answer on the current state. With read-only stores attached, a past date is read from the primary store only, and the answer says so. The date never goes through FEATURE_STORE_TIME.

Agent - freshness in business dates

FS_FOLLOW_UP is read as an event log: followup_business_dates() gives, per process and segment, the last business date completed (VALIDTIME_DATE, or the run time for a real-time run), when it ran, and the last failure. FRESHNESS now answers "computed up to , last run ", compares it with the process's TimeManager ("up to date" / "N business date(s) behind"), names the manager and how many other processes share it, and is rendered by code.

TimeManager attached like the FilterManager

  • upload_features(..., time_manager=tm) records the pairing at registration (FS_TIME_MANAGER, the table roll_out already writes), so run(process_id) computes at the manager's business date with no argument. Without time_manager= the registration statements are unchanged.
  • Before run(..., time_manager=), roll_out and upload_features(..., time_manager=) use a manager, the lineage graph is read (through intermediate views) for the manager the process view really reads. New setting TIME_MANAGER_CHECK: 'warn' (default) runs as before, warns and records no pairing for a manager the view does not read; 'error' refuses before anything runs (a view just created from sql_query is dropped again); 'off' skips the check. A process the graph does not know is a warning naming rebuild_lineage_graph(objects=[...]). A real-time run of a process whose view reads an undeclared manager logs a warning.
  • time_manager_dependency(pid) (declared vs read) and link_time_managers(dry_run=True) (record or correct the pairing of processes registered without it).
  • Optional registry FS_MANAGER_REGISTRY / FS_V_MANAGER_REGISTRY: one row per TimeManager / FilterManager object, its table, and what it was cloned from. Written best effort by the managers' load / clone / take-ownership methods, ended by _drop; a failed create is a warning and is not retried. register_managers(dry_run=True) backfills it.
  • split_time_manager(process_ids, name, clone_mode='soft', dry_run=True) gives processes their own cursor (a clone, views rewritten onto it); the inverse of unify_time_managers. Both now refresh the lineage of the views they rewrite.

Parallel scripts - lockstep on a shared TimeManager

The generated execution script ran a roll_out per worker: two processes sharing a TimeManager moved one cursor from two workers at once, so one could compute at the other's date. Processes of a wave that share a manager now run in lockstep: the script moves the cursor one date at a time, runs them in parallel at that date with run(pid, time_manager=tm), and waits for all of them before the next date. A failed process is not run at the later dates.

Report - Time drivers

A new section lists, per process, the declared and the read TimeManager (OK, UNDECLARED, MISMATCH, DECLARED_UNUSED, NONE, UNKNOWN), the manager's current date and the last business date computed, and the synchronisation groups: one manager view (synchronised), soft clones (same dates, own cursors, with whether they point at the same date), hard clones (copies), independent. Each finding names its fix.

Fixes found by running the demos live

  • A failed storage MERGE ended the run COMPLETED. _store_feature_merge ran its MERGE through execute_query, which logs and swallows errors by default: a run whose values were never stored still recorded COMPLETED (seen with Error 3810). The MERGE now raises, and the run is recorded FAILED.
  • Two first runs in parallel could mix up their features. A new FEATURE_ID is the catalog's MAX(FEATURE_ID) + 1 read by the client, so two processes registering features at the same time get the same ids; one process's values were routed to the other's storage table. The parallel planner now puts every process whose features are not all registered into one serialized unit ("feature registration (first run)"); they parallelise once their features exist.
  • The planner serialised processes that only share feature names. It found a process's storage tables by (data domain, feature name), so a customer process and a category process computing sum_Transaction_Amount were each given the other's tables and never ran in parallel. The lookup now includes the entity when the catalog names it; a feature registered only under another entity still counts as a first registration.
  • The generated script reported only the first line of an error. teradataml re-raises a database error as "Encountered problem with the query. {}"; the script now reports the error and its causes (without driver stack frames).
  • Agent: freshness of a process named by its view ("is FEAT_X up to date?") said "not registered"; that answer is now rendered by code (the model, given an empty list, had invented a dataset name); business dates are no longer listed as FilterManager segments; a process computed beyond its manager's current date is not called "up to date"; dataset freshness names the process views; the data-domain detail gives the process count; a data domain named in the question is not narrowed by an entity remembered from earlier turns, also at a past date when the domain did not exist yet.
  • Demos: demo_common.connect also clears the administrator's TDFS4DS_TD_DATABASE, so a reconnect (or a worker of a generated script) opens the persona's session in its own database (Error 3524 otherwise).

Demos

05 attaches the TimeManager at registration, runs without the argument and shows a refused unrelated manager. New 05b - Synchronising processes: a shared manager, a manager read through an intermediate view and recorded with link_time_managers, a soft clone with split_time_manager and back, a hard clone. 08 attaches both managers. A2 runs two processes on one manager with a generated script and compares their values with the sequential roll_out of notebook 05 at every business date. G2 asks the counts, a past date and a freshness question; G3 checks the Time drivers section. New diagrams time_sync and lockstep.

[0.3.1.239] - 2026-10-09

Docs - entity_id is a list of column names

The README, the documentation, the demos and the skills said entity_id had to be a dict of SQL types ({'CUSTOMER_ID': 'BIGINT'}) and that a list was rejected. That form comes from early versions: upload_features reads the types from the source, with their exact length and character set, and build_dataset only uses the column names. Every example now passes a list (['CUSTOMER_ID']). A dict is still accepted, as an override of the inferred types.

Docs - skills updated from the demos; new fs-migrate and fs-federation

The package skills (tdfs4ds/skills, installed with tdfs4ds.install_skills()) are the reference for coding agents. They now carry what the rebuilt demos verified on a live system: - fs-rollout: run(time_manager=tm) computes one date (the manager's current one) and roll_out is the loop; processes on one manager share the ingestion time and the visibility; a history is built forward from its first date, and rebuilt with roll_out(..., time_id_start=1, force_compute=True). Without force_compute, the dates already completed are skipped and nothing changes (checked live). - fs-filter: the segments must split the entity; manager filters are scalar subqueries; what the execution plan reads, measured (a single partition only when every partitioning level is pinned). - fs-dataset: why a training set must be read at its label date, and the four time modes with real outputs. - fs-upload: sql_query= + target_view_name= as the preferred form, run_now=False, the .env switches of the AI steps, on_failure. - New fs-migrate (plan, approve, migrate, rebuild the history) and fs-federation (read other stores read-only). - Every skill connects with tdfs4ds.reconnect_teradata() from the TDFS4DS_TD_* variables; the unprefixed TDML_* and FS_API_* variables are gone, and the data domain is activated with select_data_domain.

Demos - a notebook on the FilterManager alone

07 - Segmented processing with FilterManager now uses a FilterManager without a TimeManager: one category at a time, one follow-up row per segment, and a check that the segments together equal one unsegmented computation (8,858 rows, 0 differing values). The former 07 to 09 become 08 (FilterManager with a TimeManager), 09 (hybrid) and 10 (process versions). The 19 notebooks pass live from empty stores.

[0.3.1.238] - 2026-10-09

Fixed - migrating a process driven by a TimeManager or FilterManager failed

Seen live while promoting two TimeManager-driven processes from a lab store to production. The production views were never created, and the only error was a later run() failing with Error 3807 ... does not exist.

Why. - Name prefixes. rewrite_references matched a qualified name without checking where the name ends. - A manager is a view X over a table X_HIDDEN, so rewriting LAB.BUSINESS_DATE also hit the start of LAB.BUSINESS_DATE_HIDDEN and produced "PROD"."BUSINESS_DATE"_HIDDEN. - The manager view could not be created, and neither could any process view reading it. - The same flaw sat in scheduling.unify._rewrite_view_reference, where repointing a process from filter view F also rewrote F_COPY. It now uses the same pattern. - Silent failure. migrate_objects ignored a failed process-view creation. - It registered and ran the process anyway. - The database error was visible only through the progress hook.

The fixes. - A qualified name is now matched as a whole name, never inside a longer one. - A process view that cannot be created is reported in report['errors'], and its process is not registered.

Fixed - a migrated TimeManager or FilterManager did not find its own table

Both managers found their backing table with ddl.split("\n")[-2].split(".")[1]. - What broke: a migrated manager view qualifies names with quotes ("DB_PROD"."BUSINESS_DATE_HIDDEN"), so the table name kept its quotes. The manager did not find its table and reported nb_time_steps = None, and roll_out failed. - The fix: both managers now read the table after the view's FROM with the quote-aware splitter (backing_table_from_view_ddl, shared by both).

Fixed - read federation cut the store name in FEATURE_STORE

The provenance column was an untyped literal, and a UNION ALL takes each column's type from its first branch. - What broke: with the lab first, 'DB_LAB' made the column six characters wide, and DB_PROD read back as DB_PRO. - The fix: the literal is now cast to VARCHAR(128).

Fixed - a federated process catalog failed on every live system

Teradata refuses a set operation over a JSON column, even a UNION ALL (Error 3600 "Comparisons are not allowed for the type JSON"). - What broke: the process catalog has two JSON columns (METADATA, ENTITY_NULL_SUBSTITUTE), so process_catalog() failed as soon as a read-only store was attached. The offline tests never ran the SQL. - The fix: JSON columns are now read as CLOB in every branch, detected from the primary view's column types.

Fixed - plan_migration left the source store attached

plan_migration attaches the source store read-only for its own reads, and did not detach it. - What broke: every later read of the session was federated with the source. In the production session, a build_dataset then referenced a view of the source store, and failed with Error 3523 because the production database has no WITH GRANT OPTION on the lab. - The fix: the caller's read-only stores are restored when the plan returns, including after an error. migrate_objects reads the source by explicit schema names and does not need the attachment.

Fixed - a migrated process ran in real time instead of at its TimeManager's date

The migration recreated a process's TimeManager but did not record it as the process's TimeManager. - What broke: the recompute that follows (run()) ran in real time and stored today's value. A history replayed afterwards with roll_out over past dates left nothing before it (see Known limitation below). - The fix: the migrated TimeManager (the placement of the source manager's view) is now recorded for each migrated process, so run() uses its business date, as in the source store.

Added - data_strategy='none' for plan_migration / migrate_objects

Registers the processes without computing them, for a primary that replays its own history with roll_out. Datasets on such processes are reported in report['deferred'] and built afterwards, because their features only reach the feature catalog at the first run.

Fixed - a dataset on another store's features vanished from the lineage graph

When the lineage builder expands a dataset, it replaces the dataset's edges to its storage tables with edges to the processes found in the store's catalog. - What broke: for a dataset built on a read-only store's features, no process is in that catalog. The dataset was left without any edge, and the graph stores edges only, so it disappeared. - The fix: when no process resolves, the edges to the storage tables are kept.

Changed - run(..., on_failure='raise') raises for an unknown process

  • Default: a process_id missing from the primary catalog still logs an error and returns None.
  • With on_failure='raise': it now raises LookupError. The process may, for example, belong to a read-only store, which run() never executes.
  • Why: with DISPLAY_LOGS off, the None looked like a run that had gone through.

Fixed - the agent described a dataset with features of other domains

The agent matched a dataset's features to the active domain's catalog by name, and kept the first match. - Seen live: for a one-entity dataset, the answer said "exposes 2 entities (Category: 2, CustomerID: 2)" and cited another domain's processes, because the same feature names exist there under two entities. - The fix: the features are now resolved by FEATURE_ID, which identifies the data domain, entity and name. - Process: the process is the one of that domain and entity whose feature list names the feature. - Several candidates: the dataset view's FEATURE_VERSION decides. - Fallback: the name match is still used when a feature id is unknown. - Guarded by tests/test_dataset_feature_identity.py.

Known issues - Teradata Vector Store (observed while building the demos, not changed)

  • Shared collection names. Collections are named <TERADATA_VECTOR_STORE_PREFIX>_objects/_sections/_columns for the whole service. Two stores on one service with the default prefix share them.
  • Seen: a migration's index refresh on a second database failed to refresh tdfs4ds_columns in place, and the collection was recreated on that database. The first store's column index was lost, and its temporary tables were left behind.
  • Workaround: give each store its own prefix.
  • Refreshing a collection on another database. Refreshing a collection whose target database is not the default database of the Vector Store login failed with Object 'collection_<name>_index' does not exist.
  • That error is reported as "service unreachable or not enabled", which it is not.
  • build_vector_index(force_rebuild=True) (destroy and recreate) works.

Known limitation - a history cannot be replayed behind an existing value

This was observed live, not changed. - Scenario: a process has a value stored from instant T, and is then run at earlier valid times (for example a roll_out over past dates). - Result: each merge only updates the part of its period that overlaps the existing row. Nothing is inserted before T. - Workaround: build a history forward, from its first date, on a store with no values yet for those features.

Fixed - write_migration_script dropped the per-table data choice

A source table can be copied into production (the default) or read in place (data_choice='reference', set on the reviewed plan). The generated script plans again before it runs, so it silently went back to copying. The script now records each table's choice and re-applies it after planning.

Fixed - a misleading warning on the first TimeManager.load_time_steps

The first load logged WARNING Error dropping table ... does not exist. There is nothing to drop on a first load, so this is now a debug message; any other failure to drop is still a warning.

Changed - demo notebooks rebuilt for the current API

demos/ is rewritten: - a setup notebook creates the personas (data scientist, data engineer) and the databases (source, lab, production), and loads a partitioned source table; - core notebooks 00 to 09; - a production track (promotion, read federation); - an operations track and an agentic track; - generated concept diagrams, and assertions that fail the run on a wrong result.

See demos/README.md.

Guarded by tests/test_migration_rewrite.py, tests/test_migration_planner.py, tests/test_manager_backing_table.py, tests/test_read_federation.py and tests/test_failure_reporting.py.

[0.3.1.237] - 2026-10-09

Fixed - "what are the available features?" answered about a data domain the store does not have

Seen live. On a store with a single data domain, the agent listed that domain, then answered the next question with "No features are registered in the claims_anomaly_detection data domain." No such domain exists in that store.

Why. - Copied example. The classifier prompt carried a fixed example taken from a demo store: "What features are available for claims_anomaly_detection?" → domain=claims_anomaly_detection. The model copied that domain into its answer to a question that named none. - No check against the store. - A domain the store does not have still counted as a scope. - So did a remembered domain (a thread that started on another database) or a session DATA_DOMAIN written for another database. - So the single-domain shortcut was never reached, and the answer named the invented domain.

The fixes. - Examples come from the connected store. - Every classifier example (the built-in ones and the ## Classifier Examples of each ca-* skill) now uses placeholders: <domain>, <domain2>, <feature>, <feature2>, <feature3>, <entity>, <dataset>, <view>. - They are filled with real names read from the turn's catalog snapshot, at no extra cost. - An example whose name the store cannot supply is dropped, never shown with a placeholder. - No fixed names left. - Store-specific names were also removed from the classifier's field descriptions, the system prompt, the multi-feature definition prompt and one suggestion shown to the user. - A test lists the names of the old demo stores and fails if one reaches the classifier again. - Only real domains scope an answer. - The summary skill and the domain-detection node now keep a domain only when the store has it: a classifier guess, a remembered domain or the session DATA_DOMAIN. - With a single data domain, that domain is used. With several, the agent asks which one. - The "no features" message names only a real domain.

Guarded by tests/test_store_examples.py.

Added - default AI steps of upload_features from the configuration

What changes. - upload_features(document=, explain=, optimize_query=, vector_indexing=, cascade_redocumentation=) now default to None, which reads TDFS4DS_UPLOAD_DOCUMENT, _EXPLAIN, _OPTIMIZE_QUERY, _VECTOR_INDEXING and _CASCADE_REDOCUMENTATION. - TDFS4DS_AGENTIC_FEATURES=false turns all of them off at once. - An explicit argument always wins. - explain is new. It skips the LLM analysis of the EXPLAIN plan while keeping the documentation (document_process_incremental(explain=False)), and the stored analysis is left unchanged.

Compatibility. The defaults are the previous behaviour (documentation and EXPLAIN analysis on, the rest off), so a configuration that sets nothing is unchanged.

Datasets in the lineage graph. This was already the default, and a test now guards it.

Guarded by tests/test_upload_ai_switches.py.

[0.3.1.236] - 2026-10-08

Fixed - the consumer agent took ~30 s to list features

What was slow. Measured on the demo store, a "what are the available features?" turn took about 30 s. The LLM accounted for 2 s of it; the rest was catalog reads. - teradataml objects: a teradataml DataFrame spends about 1 s on metadata calls before reading a row. - Dataset catalog accessors: dataset_catalog() and get_dataset_features() each built a DatasetCatalog, whose constructor listed the database (about 4 s per call). - Repeated reads: the turn read the catalogs 8 to 10 times that way. The same rows come back in 0.1 s with raw SQL.

The fixes. - One request per turn: - the agent reads the catalogs in a single multi-statement raw-SQL request, about 0.3 s, memoized for the turn; - the statements: feature, process and dataset catalogs, dataset features and entity, feature-column descriptions, process documentation; - every catalog read of the agent (about 40 call sites) is served from it; - the run history it needs for freshness is read in raw SQL too. - Valid time: the request states CURRENT VALIDTIME, so the agent no longer depends on the session's temporal default. - Read federation: with read-only stores attached the snapshot is not used, and the federated accessors keep their behaviour. - DatasetCatalog (package-wide): - it checks that its objects exist once per process instead of on every construction; - an explicit create_catalog() still re-checks; - its three DataFrames (catalog, entity, features) are built on first use.

Measured on the demo store.

Turn Before After
listing ~30 s ~4 s (the LLM about 2 s)
freshness 5.1 s 2.6 s
database time per turn, all skills — 0.2–0.7 s

[0.3.1.235] - 2026-10-08

Fixed - the consumer agent invented themes for large feature lists, and could not narrow them

Reported from the field on a 362-feature store. - What happened: "which features are available?" came back with themes, and even feature names ("STD_CORR_NB (si présent)"), that the store does not contain. Asking for a subgroup ("the features of entity X for corrections") repeated the same overview. - Why: above AGENT_LIST_MAX_ITEMS the LLM received counts and three example features per entity, and was asked for "3 to 5 themes".

What large listings show now. The answer is built in Python from what the store records; no LLM text. - Overview: features grouped by the process that computes them. Each group shows: - the first sentence of the process's stored documentation (business logic and entity description), or "(not documented)"; - a few names; - suggestions that name real processes, never the scope just asked. - Subject in the question ("features of entity X about corrections"): - the LLM picks the matching features from the scope's documented descriptions; - every returned name is checked against the scope, and unknown names are dropped and reported in topic_dropped; - the matches are listed in full with their descriptions; - a subject with no match says so. - Process scope: "features of process V_…" lists everything that process computes. Before this release a process view name was not recognised and the whole domain was listed. - Below the cap: listings are unchanged. - Follow-up questions: after a subject turn, follow-ups ("are these up to date?") apply to the selection.

Added - the agent states each dataset's own time

Two times. - The metadata plane (what is registered and documented) is read at the current valid time. - Each dataset has its own data time. Datasets are now: - snapshots at the current valid time (the only kind before 0.3.1.216); - snapshots at a fixed date (as_of=, or legacy feature_store_time=True); - snapshots at a TimeManager's business date (moves with tm.update()); - historical datasets (one row per period).

What the agent says. - Which answers carry the dataset's time: - SUMMARY on a dataset; - DATASET (each dataset is tagged, current snapshots first); - USAGE (for a historical dataset the snippet selects ValidStart / ValidEnd and shows how to read one instant); - FRESHNESS on a dataset (a fixed snapshot is "frozen"; a TimeManager dataset is stale when its processes have not completed a run for the manager's business date). - Where the time is read from: - the build/window rows, read straight from their public views (never through the dataset catalog constructor, which creates missing objects); - otherwise the view's DDL, which is how a dataset registered before the build table existed is read. - Current snapshots get no line, so a store from an earlier version answers exactly as before. - HELP explains the two times.

The metadata stays current. _recognize_object_type no longer calls get_list_entity, the one agent read that followed tdfs4ds.FEATURE_STORE_TIME (an ingestion setting). A source-level test guards the agent modules.

Changed - feature documentation names the entity by its meaning

  • The rule: the documentation prompt says a feature description refers to the entity by its business meaning ("for each study"), never by its key columns ("per STD_CDH_ID").
  • Retry: a description that still names a key column triggers the existing single correction retry, with the entity description in the note. A formula violation and a key leak still share one retry.
  • Existing documentation changes only when the process is re-documented (rebuild_documentation(objects=[...], force_update=True) or the Documentation panel).

Added - Report: each entity column marked as primary index / partitioning / neither

  • The inventory tree tags each entity column PI (primary index) or P (partitioning column) of the entity's storage tables. PI* / P* means only some of the tables agree.
  • The tags come from one batched data-dictionary read.
  • A legend explains them, and when the dictionary cannot be read the tree says so.

Fixed

  • skill_freshness raised UnboundLocalError on every cache hit.
  • skill_usage never put the entity keys in its SELECT snippets: it read ENTITY_ID, but the feature catalog column is ENTITY_NAME.
  • Dataset freshness compared bare and qualified process view names; it now matches on the object name.

Offline: tests/test_large_listing.py (rewritten), tests/test_agent_valid_time.py, tests/test_doc_entity_meaning.py, tests/test_store_consistency.py.

Live, on the demo store: - the overview; - subject selection (EN and FR); - the process scope; - as_of and historical datasets, read from the build row and from the DDL only (throwaway datasets, dropped); - the entity key roles.

Not done: the time line in LINEAGE answers on a dataset node.

[0.3.1.234] - 2026-10-08

Fixed - the admin app's feature counts disagreed, and none said what it counted

Reported from the field: on one store the Report showed 362 features in its KPIs and 337 in its inventory tree, the EFS map 362, and the Documentation coverage panel 346.

  • One counting rule: a feature is (data domain, entity, feature name).
  • The inventory tree badged a domain with the number of distinct names, so a name registered under two entities counted once (the 337).
  • _distinct_feature_count now includes the data domain when the frame has one. On a single-domain frame its result is unchanged. The consumer agent's ungrouped counts on an all-domain frame (search, summary with no domain selected) used to undercount a name repeated in two domains; they no longer do.
  • Removed features were counted as current. FS_V_FEATURE_CATALOG is the only catalog view without CURRENT VALIDTIME. A session whose temporal qualifier is ANSIQUALIFIER (verified live) reads it nonsequenced, so feature_catalog() also returned closed rows: 16 instead of 14 on the test store, the 2 extra being removed features whose table no longer exists.
  • The Report, the EFS map and the coverage note now read the base table CURRENT VALIDTIME. With read-only stores attached they keep the federated accessor.
  • feature_catalog() (list_features) now keeps the rows valid now when the view comes back with ValidStart/ValidEnd (the ANSI case), using a plain predicate. A session whose default is CURRENT already returns current rows only and reads the view exactly as before. Read federation is unchanged.
  • A new feature could reuse a removed feature's ID. GetTheLargestFeatureID read MAX(FEATURE_ID) through the view, which a CURRENT-default session limits to current rows. A removed feature's ID, whose values may still sit in its FS_T_* table, could then be given to a new feature. It now reads every version of the base table (NONSEQUENCED VALIDTIME), which is always at least what it returned before.
  • The view itself is unchanged: registration joins it, and the per-feature value views are built SEQUENCED VALIDTIME over it.
  • The documentation coverage card is "Features (process outputs)". It counts one row per name a current process lists, because documentation lives on the process view's column. A sub-line now reconciles it with the catalog: 362 in the catalog · N without a current producer (not documentable here) · M listed by several processes.
  • Remarkable facts:
  • the richest and leanest datasets were shown by DATASET_ID (the dataset-features view has no name column); they now show db.name, counted by FEATURE_ID;
  • the refresh cadence measured the gaps between the FilterManager segments of one execution ("every 0.0 h"); it now measures the gaps between executions (RUN_ID, VALIDTIME_DATE).

Added - catalog valid time, and consistency & orphans in the Report and the EFS map

  • Catalog valid time.
  • Both panels gain a valid-time input: blank = current, otherwise a date-time read as UTC.
  • Every catalog count states the valid time it was read at.
  • A date reads the primary store's base tables VALIDTIME AS OF. The process-catalog SELECT is now one helper (process_catalog_select_sql) shared with the FS_V_PROCESS_CATALOG view, whose text is byte-identical.
  • Routes /ui/dashboard/report, /ui/dashboard/report/download, /ui/efs and /ui/efs/download take as_of=.
  • New module serve/store_consistency.py. It holds the snapshot, a pure consistency model and the renderer. The model reports:
  • features with no current producer, listed by several processes, retired since registration, or whose storage table is missing;
  • processes whose view is gone, whose features are not registered, that never ran, or whose features no dataset reads;
  • FS_T_* tables no catalog row points to (or only retired ones);
  • datasets reading feature ids the catalog no longer has, or whose view is gone.

Each finding names the command that fixes it; nothing writes. A reconciliation line explains every feature number. - Report. - New Consistency & orphans section. - KPI sub-lines (4 retired · 2 without producer, 3 never run, 1 unreferenced). - A Scan storage for orphans button: SSE /ui/dashboard/orphans, one grouped FEATURE_ID, COUNT(*) read per storage table, all versions and at the valid time. It finds FEATURE_IDs stored without a catalog row (known in the history, or never registered) and registered features with no value. - EFS map. - Orphan-ID badges on the table cards once a scan exists. - An Unreferenced storage tables block: the map only knew the catalog's tables, so these were invisible. - KPI sub-lines and the reconciliation line. - Table and view existence is read from the data dictionary now, whatever the valid time; physical row counts keep their own time base (statistics or exact counts). Offline: tests/test_store_consistency.py.

[0.3.1.233] - 2026-10-08

Fixed - batch lookups now reach the storage tables' primary index and partitions

Reported from the field: the benchmark's multi-key batch phase ran 64 times over its estimate on a 10-column grouped dataset whose storage tables are partitioned on FEATURE_ID and an entity column.

  • Keys pushed into every storage-table read. retrieve_batch and the benchmark used to filter the dataset view (SELECT * FROM view WHERE <keys>). Under FULL OUTER the view's entity is a COALESCE over every sub-query, so nothing can be pushed through it; in a grouped view only some sub-queries received the keys. EXPLAIN on a live 3-group dataset: the view form filtered the first group and aggregated the other two whole (estimated 4,989 and 4,277 rows) before applying the keys, while every pushed-down read filters before GROUP BY (10 rows each). The dataset's query is now regenerated from its own DDL (ddl_spec.render_retrieval_sql, the same renderer as the view) with the key predicate - or the join to the volatile key table - inside each FS_T_* read. New retrieve_batch(pushdown='auto'|True|False); False is the previous SQL byte for byte, and historical or unparseable views fall back to it.
  • Partition elimination on entity columns. When a storage table is partitioned on an entity column, a redundant B1.<col> IN (<distinct values>) is added to its read (implied by the keys, so no result changes); the optimizer derives static partition elimination from it, not from an OR of ANDs or a join. One rule, partitioning_entity_columns, now serves both this and the statistics check.
  • The volatile key table gets the real primary index. The storage tables were looked up in the process view's DDL (the user's source view), so none was found and the key table always fell back to the full entity as its PRIMARY INDEX. They are now read from the dataset DDL.
  • Date and timestamp keys matched nothing. _fmt_val rendered a date as col = 2016-01-01 (integer arithmetic), so every in_list and single-key lookup on a date-keyed dataset returned no row - the benchmark timed empty lookups. Dates, timestamps and times now render as typed literals.
  • Benchmark: the batch, large-batch and batch-strategy phases call the shared builder (build_batch_sql), so they measure what retrieve_batch runs; run_benchmark(batch_pushdown=False) measures the old form for comparison, and the report states the query shape and, per storage table, its primary index and partitioning entity columns.
  • Live-verified: rows identical to the view form for 6 registered datasets and for INNER/FULL OUTER x multi-join/grouped (with and without the partition IN-list, including a key that matches nothing), both strategies. Offline: tests/test_batch_pushdown.py.

[0.3.1.232] - 2026-10-08

Added - tdfs4ds.configure_interactively(): guided, non-destructive .env setup

  • Asks for the database connection, the instruct model, the embedding model and the vector store, and updates the existing .env line by line through patch_dotenv. Comments, ordering, commented alternatives and unrelated variables survive; values already in the file are the defaults (Enter keeps them); a secret is shown as set (hidden, N chars). generate_ai_studio_env is unchanged and still rewrites its file.
  • Models are listed by the provider's own mechanism: HTTP /models for the OpenAI-compatible family, boto3 for Bedrock (foundation models plus inference profiles, since many models can only be invoked through a profile id). Azure, Vertex and LiteLLM have no list call in the package, so the model is typed and the live check decides.
  • An API key is asked for only when needed. The wizard first tries without one (cloud role, profile, keyless gateway) and asks only after an authentication failure, either when listing or at the live model call. A keyless OpenAI-compatible endpoint is recorded as EMPTY, which the SDK accepts.
  • The Teradata Vector Store may be unavailable (service not enabled, teradatagenai missing, old version). The wizard runs the existing auth and service checks and offers local Chroma instead of writing a backend that cannot work.
  • Secrets are written only after an explicit yes (write_secrets='ask'|'never'|'always', default asks with no), in a separate allow_secrets=True call. Nothing secret is printed or returned.
  • Fixed: INSTRUCT_MODEL_REGION, INSTRUCT_MODEL_AZURE_*, INSTRUCT_MODEL_GCP_* and the EMBEDDING_MODEL_* counterparts could not be set from a .env (only from Python), so a Bedrock region could not be persisted. They are now registered config variables.
  • Guarded by tests/test_configure.py and tests/test_configure_bedrock.py (offline).

Added - AWS Bedrock through its OpenAI-compatible endpoint, with an API key or the environment's role

  • INSTRUCT_MODEL_PROVIDER=aws (a spelling of bedrock) or bedrock, with INSTRUCT_MODEL_URL ending in /openai/v1 (for example https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1), now speaks the OpenAI protocol to Bedrock. Authentication is the API key as the bearer token, or - with INSTRUCT_MODEL_CLIENT_MODE=ambient, or no key - SigV4 signing with the ambient AWS credential chain (instance or task role, SSO, profile). New settings INSTRUCT_MODEL_CLIENT_MODE and EMBEDDING_MODEL_CLIENT_MODE. The routing and key rules live in one place, tdfs4ds/utils/llm_provider.py.
  • Backward compatible: a bedrock configuration without such a URL builds the same ChatBedrockConverse client with the same arguments as before (asserted in tests/test_llm_provider.py). One deliberate exception: a bedrock configuration that already carried an .../openai/v1 URL, which used to be ignored, now takes the OpenAI-compatible route.
  • Fixed: a role-based setup (Bedrock, Vertex) could never document anything - document_sql_query_columns and the two other documenters refused to run without INSTRUCT_MODEL_API_KEY for every provider. They now ask for a key only where one is required.
  • check_configuration() names how a Bedrock or Vertex call authenticated (never a value), and an AccessDenied is reported as a permission problem (bedrock:InvokeModel on the model or inference profile) instead of "check the API key". The JSON-constraints sweep signs its requests like the client does. A Bedrock embedding provider left on the OpenAI default model name is called out.
  • The Converse client also resolves its region from AWS_REGION (an explicit setting or AWS_DEFAULT_REGION still wins).
  • The wizard asks Converse or OpenAI-compatible, then role or key; a role that may call a model but not list models is not asked for a key - it types the model id (several candidates are tried in turn) and is offered the eu. / us. / apac. inference-profile form when the bare id is refused.

[0.3.1.231] - 2026-10-07

Changed - Execution plan and Migration marked BETA in the admin app

  • The rail shows a BETA badge on Execution plan and Migration, as on Evaluation.

[0.3.1.230] - 2026-10-06

Fixed - the consumer agent forgot the conversation when the MCP toggle changed

  • Each compiled agent graph (one per skill set, plugin set and MCP setting) had its own conversation memory (MemorySaver). Turning MCP on mid-conversation, for example to sample a dataset after a few turns, switched to a graph with no memory for that thread. The agent lost the earlier turns, and the knowledge graph shrank to the current turn. All graphs now share one checkpointer, so the memory follows the thread, and the knowledge graph only grows within a conversation. reset_consumer_agent() rebuilds the graphs but keeps the memory; Reset Conversation (a new thread id) still starts fresh. Guarded by tests/test_conversation_memory_shared.py.

[0.3.1.229] - 2026-10-06

Added - Documentation section in the admin app; left rail grouped by theme

Documentation → Coverage / Objects (serve/doc_coverage.py, routes /ui/doc/*): what is documented, what is missing, and what is in the vector index.

  • Objects: processes and each feature column, datasets (description + the five sections), intermediate views and source tables of the lineage graph. Statuses: complete / partial / missing. A blank description is missing, and documentation outside the expected slots (a custom section, documented columns) makes an object partial. FS_T_* and *_HIDDEN objects are not counted. Source tables are flagged user-supplied (tdfs4ds never generates them). Documentation of objects the store no longer has is listed as orphans.
  • Lineage nodes absent from DBC.TablesV are left out and listed. The live graph held SQL tokens stored as object names (1, APPLY, POSITION); they had shown up as views missing documentation. Fixing the lineage builder is not part of this change.
  • Vector index status per documented row:
  • Chroma: exact. vector_index.chroma_index_status builds the expected documents with the build's own builders. It compares them through classify_stored, the one comparison now shared with the build's skip of unchanged rows (_unchanged_ids). Rows are indexed, stale or not indexed; stored ids no longer documented are orphans. A collection is opened without creating it and without an embedding client.
  • Teradata Vector Store: inferred, and labelled so. A row is indexed when the collection is ready and the row's ValidStart precedes the collection's last_updated, and pending otherwise. A missing, failed or building collection is shown as such. The backing tables are never read, because TD_ID is a row position.
  • Coverage panel: KPIs per kind, one line per collection, gap lists linked to the objects, and a CSV export of the gaps.
  • Objects panel: filters, row selection, and object detail. The detail shares _object_documentation_md with the Lineage node detail (extracted from it, same output).
  • Actions, streamed: Document missing / selected (process, dataset, intermediate view; one call per process; source tables skipped), then build_vector_index(force_rebuild=False); Update vector index. A long selection is parked through POST /ui/doc/selection, because EventSource can only GET.
  • No feature-store database selected now gives a plain message instead of a teradataml type error (seen when the server started without TDFS4DS_SCHEMA).

Rail grouped by theme: - Sections: Assistant, Explore, Documentation, Operations, Performance, Settings. - Every entry has a one-line description, shown on hover. It is positioned in JS because the rail scrolls and would clip a CSS tooltip; title stays as the fallback. - Sections collapse when their heading is clicked. The collapsed state is remembered in localStorage, which is optional. The section holding the panel in the URL always opens. - A test checks that every rail entry has a registered panel and a description.

Live-checked on a store with real documentation and the Teradata Vector Store. - Figures: 38 objects; 6 documentation gaps; 32 index gaps, because one collection does not exist and another was last updated in July. - Phantom nodes: 3 lineage nodes left out. - Chroma: the exact status, run against a local index, matched the documentation tables' counts exactly. - Browser check (Edge driven by Playwright): tooltip, collapse, reopening on boot, gap link to object detail, filters; no page errors. - Not run live: the Document / Update index actions, since they write.

Also corrected: the rebuild_vector_indexing and build_vector_index docstrings still described the Teradata behaviour from before 0.3.1.167 / 169.

Fixed - raw Markdown in the admin app (lineage node detail, reports)

  • The admin panels render documentation with the markdown package, which was imported but declared nowhere. An install without it fell back to the raw text in a <pre> block, logging only at DEBUG. markdown is now in the serve extra, and the fallback logs a warning and states the reason above the text.
  • td.css is loaded with ?v=<version> like app.js, so a browser does not keep an old stylesheet after an upgrade.

[0.3.1.228] - 2026-10-06

Fixed - Vector Store collections turned update_failed right after a successful build

Cause of the field report left open in 0.3.1.225, now established from a customer's log and collection listing: rebuild_vector_indexing(force_rebuild=True) created the three collections, which reached ready; the MAJOR "finalize" update tdfs4ds sends after every build (collection.update(update_style="MAJOR")) then turned two of them into update_failed within seconds, the third being still update_started. The build logged "MAJOR finalize pass complete", and the next tdfs4ds.serve start found failed collections.

  • The MAJOR finalize pass is removed. It was added in 0.3.1.139 to consolidate after add_datasets(update_style="MINOR"), and no build has called add_datasets since 0.3.1.169: a collection is created from scratch or refreshed (update(data_refresh=True), else destroy + recreate), and neither needs it. The self-test drops its major_finalize step for the same reason (it only checked that the update was accepted, so it passed on the affected service). A source-level test guards that no update_style="MAJOR" call remains.
  • A write is confirmed only by a status that follows it. _verify_write_succeeded(before=...) no longer accepts the ready status still reported for a moment after an update is accepted: ready counts once last_updated has changed or a non-ready status was seen (after a few polls when the service gives no last_updated). This race is why 0.3.1.225's check logged "complete" for a pass that failed. Used on the refresh path; a create needs no baseline. _lookup_store_info now returns last_updated.

Not live-tested (no affected Vector Store service available); offline tests replay the customer's status sequence.

[0.3.1.227] - 2026-10-06

Changed - one valid start per run; run() uses the process's TimeManager

Run processes with a TimeManager or with tdfs4ds.FEATURE_STORE_TIME set: it is the only way several processes share one valid time, and the only way values computed for a business date carry that date. Real time is a fallback. Two gaps made the fallback, and manual stepping, worse than they needed to be:

  • A real-time run stamped each feature separately. Without FEATURE_STORE_TIME, each feature was merged by its own CURRENT VALIDTIME statement with its own clock reading, so features of one run started about a second apart, and a historical dataset showed that second as a period. A real-time run now reads the database time once and every feature and every FilterManager partition of the run uses it. Checked live: one run, 48 rows, 8 features, one valid start. The follow-up row still records VALIDTIME_DATE as NULL, so per-segment freshness is unchanged. Rows written this way end at 9999-01-01 like any FEATURE_STORE_TIME run, rather than at 9999-12-31 23:59:59; both read as open.
  • run() alone ignored the process's TimeManager. After tm.update(time_id=i), run(pid) computed the manager's business date but stamped it with the current time unless FEATURE_STORE_TIME was set by hand. run() now decides the valid time once, in this order: FEATURE_STORE_TIME if set; then a new time_manager= argument (the pairing is recorded, as roll_out records it); then the TimeManager recorded for the process; then real time. With a TimeManager, FEATURE_STORE_TIME is set for the run (the follow-up records the date) and restored afterwards. A recorded manager that cannot be read is a warning, never a failure.
  • roll_out left the last rolled-out date in FEATURE_STORE_TIME. A later run() or build_dataset in the same session silently used it. roll_out now restores the previous value, also when a process fails.

upload_tdstone2_scores also uses one database timestamp when FEATURE_STORE_TIME is not set. Generated parallel-execution scripts are unchanged (they drive processes with a TimeManager through roll_out).

All five cases were checked live on Vantage 20 (demo store from demos/): real time; recorded TimeManager; time_manager=, with the pairing recorded; an explicit FEATURE_STORE_TIME winning over the recorded manager; and roll_out restoring the global.

Found while testing, not changed: the STORE_FEATURE='UPDATE_INSERT' storage path does not work. Its table loop reads the table and database names from a groupby().count() frame that only holds the count, and it fails when DISPLAY_LOGS is on. The default 'MERGE' path is unaffected.

Fixed - setup() on a system without temporal tables: one clear error, no false "created"

Reported from the field (Vantage 20.0.0.69): every catalog CREATE failed with "Temporal operations are not supported on this system", and the log showed each failure twice, each followed by about twenty lines of the driver's Go stack trace, then a secondary index attempted on each missing table (Error 3807), then "Feature catalog table created", "Process catalog table created" and so on for tables that did not exist.

  • setup() first checks that a VALIDTIME table can be created (a session-only volatile table, the probe check_configuration already uses). When the probe fails for a temporal reason, it stops before creating anything, with one error naming the cause and the remedy (ask the DBA to enable the Temporal feature). Any other probe failure only warns, so a transient problem in the probe never blocks a store that would work.
  • describe_db_error(exc) (utils/query_management.py) drops the driver's stack frames (at … .go:NN) and keeps every message line. Messages are still not cut to their first line: the line that names the fault or the fix is often a later one. Used by execute_query, create_object_if_missing and setup().
  • Each failure is logged once: execute_query(log_errors=False) lets a caller that re-raises and logs itself (create_object_if_missing, the follow-up table, the if_exists='replace' re-create) skip the generic "Error executing query" line.
  • The feature and process catalogs skip their secondary index when the table failed, instead of adding an Error 3807 that hides the cause.
  • setup() no longer announces "… table created" after each step. The creation helpers already log what they created, found or failed to create, and setup() now ends with one checked line: "Setup complete: the catalog tables exist in ", or "Setup incomplete: N catalog table(s) missing: …", read from DBC.TablesV. A failing dataset catalog is reported instead of raising out of setup().

Live-checked on Vantage 20 (a store with temporal support): a real driver error went from 22 lines (19 stack frames) to its one message line, and setup() on an existing store logged each object once as "already exists", ended with "Setup complete", and left the catalog tables untouched (same CreateTimeStamp). The no-temporal path is covered offline, replaying the field error (tests/test_setup_error_reporting.py).

[0.3.1.226] - 2026-10-06

Added - historical datasets: build_historical_dataset

Every dataset so far was a snapshot: each entity's feature values at one valid instant. tdfs4ds.build_historical_dataset(...) returns their history over a window instead — one row per entity per period during which its feature values did not change:

<entity columns>, ValidPeriod, ValidStart, ValidEnd, <feature columns>
  • Still a view. Nothing is materialised: new ingestions show up at once, and a relative window moves by itself.
  • The window: the full history by default, or start / end (date or timestamp), or a duration and a delay relative to the current valid time (duration='12 months', delay='1 day' = [now - 1 day - 12 months, now - 1 day)). Durations accept '3 months', '90 days', '1 year 2 months', ISO 8601, (3, 'months') or a timedelta; months use ADD_MONTHS, so 31 May minus 3 months is 29 Feb. now is evaluated when the view is queried, and the end is always defined: a period still open in the store ends at the window's end, not at 9999-12-31.
  • Join types. INNER keeps a period only where every feature has a value; FULL OUTER keeps every period where at least one has, with NULLs elsewhere. At any instant of the window the history equals build_dataset(as_of=<instant>) with the same join type.
  • Consecutive rows always differ. Equal neighbours are merged with NORMALIZE ON MEETS OR OVERLAPS. tdfs4ds' own ingestion does not create them: re-writing an unchanged value leaves the stored period intact (checked live, in real time and at an explicit valid time), and on the demo store the merge found nothing to do. It guarantees the property for histories that do hold equal neighbours, for example rows written outside tdfs4ds.
  • Catalog. Registered like any dataset, with TIME_MODE='HISTORY' in the build table. parse_dataset_ddl reads the view back, so rebuild_dataset (new arguments start/end/duration/delay/history: turn a snapshot into a history, move a window, or go back to a snapshot), restore_dataset_catalog and cross-store migration handle it.
  • Window table FS_<catalog>_WINDOW / FS_V_<catalog>_WINDOW, read by tdfs4ds.get_dataset_window(): one row per historical dataset, joined to the catalog on DATASET_ID (LEFT OUTER JOIN — snapshots have no row). Typed columns: START_KIND (FIXED/END_MINUS_DURATION/EARLIEST), END_KIND (FIXED/START_PLUS_DURATION/NOW), START_TS, END_TS, duration and delay as ISO 8601 and as months + seconds, SLIDING, WINDOW_TEXT. The view adds WINDOW_START / WINDOW_END, the bounds when queried, so a sliding window shows real timestamps. Live-verified on Vantage 20: for all eight window shapes they equal the bounds the dataset view computes, in the same statement. A separate optional table, like _BUILD: no ALTER of an existing catalog, older versions ignore it, a user without CREATE rights gets a warning. restore_dataset_catalog reports a history without its row as REPAIR and writes it; a history rebuilt in place as a snapshot has its row closed. AS_OF_TIMESTAMP keeps meaning an as-of instant only (NULL for a history). Two findings on the way: CURRENT VALIDTIME over a derived table is Error 9330, so the view is one level with the end expression repeated; an interval cannot be built from a column with a DAY TO SECOND literal, so the seconds are split into DAY(4)/HOUR/MINUTE/SECOND casts (checked to 9999 days 23:59:59). Each optional view is now skipped only when its own table failed, not when any optional object did.

How the SQL was chosen, from EXPLAIN plans read on Vantage 20:

  • INNER uses Teradata's native SEQUENCED VALIDTIME join, which intersects the valid periods itself: each storage table is read once from a single partition, the joins are rowkey-based merge joins, and even a sliding window is folded into the join step as constants.
  • A sequenced outer join is refused (Error 9347), so FULL OUTER cuts each entity's timeline at every change point and attaches each feature on containment. It reads each storage table twice, and is about twice as slow on the tested data. The change points come from one read per feature emitting both bounds: against reading starts and ends separately, it gave the same rows with 3 storage reads instead of 8 and 2 redistributions instead of 10.
  • NORMALIZE ON MEETS OR OVERLAPS was preferred to the TD_NORMALIZE_OVERLAP_MEET table function. The function needs exact RETURNS types and only works behind a WITH clause; fed by a derived table, it silently multiplied the rows (3 rows in, 6 out).

Verified live on a feature store built from the demos/ notebooks: for three cases (customer features, category features, and category features under three process versions), both join types were created as views and returned exactly the rows of an independent reference computed from the raw storage rows, and they matched build_dataset(as_of=...) at sampled instants. On the three-version case FULL OUTER returned 112 rows and INNER 75: INNER starts each history when the latest process first wrote a value.

Observed while testing, documented and not changed:

  • In real time (no FEATURE_STORE_TIME), ingestion merged each feature with its own statement and timestamp, so features of one run could start a second apart; a combined history shows that second as a period (one valid start per run from 0.3.1.227).
  • A float aggregate recomputed in a different order can differ in its last bits (16038.699999999999 vs 16038.7), which opens a new stored period and a history row that prints like its neighbour.

[0.3.1.225] - 2026-10-05

Fixed - a failed Vector Store collection blocked the server until a manual reset

Reported from a deployment: after a documentation + vector-index build that completed without error, all three Teradata Vector Store collections ended in update_failed, and the next python -m tdfs4ds.serve start refused them and asked for a manual reset_teradata_store().

  • The finalize pass is verified. After refreshing the collections the build runs one consolidation update per collection. It is asynchronous, and the build logged "complete" as soon as the request was accepted, so a finalize that ended in update_failed went unnoticed. The outcome is now checked, and a collection it left failed is recreated.
  • A failed collection is recreated. It can be neither searched nor updated, so recreating it loses nothing. The build used to stop and ask for a manual reset; it now recreates the collection when the connected user is allowed to (admin), and still asks for the reset otherwise.

The finalize update itself completed normally on the service used for testing, so what made it fail on that deployment is not confirmed.

Fixed - Markdown rendering in the admin app; lineage node documentation

  • LLM-written text (business dictionary, reports) uses GitHub-style Markdown, which the app's renderer reads more strictly: a list right after a line of text showed as one paragraph with literal dashes, sub-items indented by two spaces were flattened, and numbered questions stored on one line ("1. ...
  • ... 3. ...") showed as a single item. These are now adapted before rendering, everywhere the app renders Markdown. Code blocks are not changed.
  • Lineage → Inspect node: the documentation was shown in the grey italic "loading" style, so it read like placeholder text and only the SQL stood out. It is now plain text, sections come in a fixed order with readable titles, the overview is no longer shown twice, column descriptions are no longer cut at 120 characters, and a node with no documentation says so (a failed read is reported instead of being silently skipped). The Query optimization process detail had the same grey italic style.

[0.3.1.224] - 2026-10-05

Fixed - consumer agent ignored a data domain named in the question

Asked "what are the available features contained in the "Governed Demand Forecasting M5" data domain?" on a store with several data domains, the agent answered with a list of datasets presented as features, and its context knowledge graph showed a single domain node, from an earlier turn, with nothing attached. Reproduced on the live store; three faults.

  • The named domain was not resolved. The classifier extracts it into its domain field, but the domain-detection step resolved a domain only from an object name. The domain is now matched to the stored DATA_DOMAIN (ignoring case and quotes, then a close-spelling match; a theme such as "pharmacy" does not match) and becomes the active domain of the conversation. The graph's domain node follows it.
  • The summary scanned another domain. It received the domain as a label but read the catalog under the session's DATA_DOMAIN. In the admin app, which runs with none, the catalog filter compared to the string "None" and kept nothing. The summary now reads under the domain it was asked about, and so does the "only one domain, use it" shortcut, which had the same flaw. With no active domain the catalog helper reads every domain.
  • Zero features were described anyway. With an empty feature list the LLM was still given every dataset's documentation and listed the datasets. A domain without features now gets a fixed answer saying so.

Live, after the fix: 14 features, 35 graph nodes, no unconnected node; a follow-up on a second domain in the same conversation shows both domains.

Fixed - English question answered in French

An English question was answered in French. Every agent LLM call now states the answer language when it is clear from the question (two or more French marker words: French; none: English; otherwise it is left to the model).

[0.3.1.223] - 2026-10-05

Fixed - consumer agent asked "which did you mean?" for plain inventory questions

Seen in the admin app's chatbot: "what are the available features?" got a numbered "which did you mean?" prompt, and the next question, "what are the data domains?", got one too. Reproduced on the configured model (Mistral Small); there were five faults.

  • The classifier prompt asked for low confidence. Its rule said "below 0.5 … when no specific entity is named". An inventory names none by design, so "what are the available features?", "list the features" and "which processes exist?" all came back at 0.3 with SUMMARY/SEARCH interpretations. The rule is gone, and the prompt now says that a question needing no object (an inventory, a conceptual question) is not ambiguous for lacking one. Measured: 0.3 → 0.95. A deterministic backstop (_is_feature_inventory_question, EN/FR) also sends a feature inventory to SUMMARY without a clarification, whatever the model returns. A question with a scope ("…of dataset X", "…for STORE", "…about risk") is not an inventory.
  • A new question was matched against the pending options. Asked while a clarification was pending, "what are the data domains?" matched no option and the same prompt was offered again. A reply that is a new question (more than three words, ending with "?" or opening like a question; _is_new_question) now drops the pending choice and is classified on its own. A short reply ("2", "the summary one") still resolves the choice.
  • The chosen reading was answered as "1". After a choice was resolved, the skill still received the literal reply as its question: a search ran on "1" and answered in French. The resolved turn now carries the chosen option's wording.
  • The clarification text drifted in language. It was an LLM call restating a fixed list, and two English questions got French text. It is now a fixed EN/FR template (like the entity clarification), which also saves a model call.
  • A search with no data domain selected saw no features and answered that the store held "exactly 0". skill_search now reads every domain when none is selected.

Fixed - Query optimization panel: no SQL, no scores for most processes

get_process_info looked a process up in the session's data domain only. The panel lists every domain's processes and runs with no domain, so the lookup failed ("list index out of range") and the detail showed the raw process id, empty scores and "No stored EXPLAIN documentation". This happened although the process was scored (4/5, 3/5) and the chart above showed it. A PROCESS_ID is unique across domains, so the lookup is no longer scoped. This fixes every get_process_info caller, including documentation and the optimizer.

Verified live on the configured model and store: - the reported two-turn conversation is answered, and all four inventory phrasings get 0.95; - "risk" still asks, in English; "1" answers the chosen reading in English, quoting the 16 registered features; - the panel detail shows the view name, its 4/5 score and its SQL.

Offline: 28 new tests; the suite has 1631 passing.

[0.3.1.222] - 2026-10-05

Added - query band per process execution; follow-up ↔ DBQL resource usage

Every query of a process execution now runs with a session query band. This covers the follow-up INSERT, the ingestion, the MERGE and the follow-up UPDATE. The band carries TDFS4DS_RUN_ID, TDFS4DS_PROCESS_ID, TDFS4DS_EXEC_SEQ, TDFS4DS_EXECUTION_ID, TDFS4DS_VALIDTIME and TDFS4DS_VERSION (new module tdfs4ds/utils/query_band.py).

  • The caller's band is kept. The pairs are added with UPDATE FOR SESSION, and the band read beforehand is restored afterwards, also when the execution fails. Setting a pair to NONE would store the literal "NONE" rather than remove it (verified).
  • Never fatal. A refused band means the execution runs untagged, with a warning logged once. tdfs4ds.QUERY_BAND = False (TDFS4DS_QUERY_BAND) turns it off.
  • followup_resource_usage(process_id=, run_id=, days=30, flush_minutes=10, only_logged=False) returns one row per execution: every follow-up column, then the resource usage of its queries:
    • N_QUERIES, N_ERRORS;
    • CPU: AMP_CPU_SEC, MAX_AMP_CPU_SEC, IMPACT_CPU_SEC (busiest AMP × active AMPs), CPU_SKEW_PCT (1 − AMP_CPU / IMPACT_CPU) and PARSER_CPU_SEC;
    • I/O: TOTAL_IO, PHYSICAL_IO, IO_SKEW_PCT;
    • spool: MAX_SPOOL_BYTES;
    • DBQL times: FIRST_QUERY_START, LAST_RESPONSE.

followup_resource_usage_sql() returns the SQL. create_followup_resource_view() creates FS_V_FOLLOW_UP_RESOURCES (the follow-up columns plus these) where the view database may read DBQL. The existing FS_V_FOLLOW_UP is left unchanged, so it keeps working for users without DBQL access. - DBQL is flushed from a cache (every 10 minutes by default), so a recent execution has no DBQL row yet. The join is a left join, and each row carries DBQL_STATUS: LOGGED, PARTIAL (totals may still grow), PENDING (nothing written yet) or NOT LOGGED. Pending is judged on the last activity (end, or start when it never ended), so a row left RUNNING by a crash months ago reads NOT LOGGED rather than pending forever (seen live). A log line says when executions are still pending. - Pairing without a schema change. FS_FOLLOW_UP has no execution id, and adding one is not possible: its INSERT … SELECT lists no columns, so older clients sharing the store would fail with Error 3812 (verified). A row is identified by RUN_ID, PROCESS_ID and its rank by START_DATETIME, which TDFS4DS_EXEC_SEQ carries. A gap in DBQL therefore does not shift the pairing. The limits are documented: a failed follow-up insert, and two executions of the same process starting within the same second.

Added - check_configuration: temporal and query band / DBQL

Two checks run first when a database session is open; without one they are skipped with a hint.

  • Temporal (VALIDTIME): a session-only VALIDTIME table is created, written, read CURRENT and NONSEQUENCED, then dropped.
  • Query band & DBQL: the band is set, read back and restored. The check then reports whether DBQL is readable and logs this user's queries.
  • Only a refused SET QUERY_BAND fails the check.
  • Missing DBQL access or logging returns the statement to ask the DBA for (GRANT SELECT ON DBC.QryLogV TO … / BEGIN QUERY LOGGING WITH SQL ON …).

Added - benchmark: parse-time check first

run_benchmark(max_parse_ms=5000) first times EXPLAIN SELECT * FROM <dataset> three times: the parse and optimise cost, with nothing executed. It records the view's DDL size and plan steps. This runs before the key sampling, which already executes the whole view.

Above the threshold, the run stops and the report explains why, suggesting the grouped strategy; 0 never stops. The admin Benchmark panel has a Stop if parse time exceeds field and a Parse-time check skip box, and the CLI has --max-parse-ms.

Verified live on Vantage 20, 11/11, with no permanent write: - both new checks work: temporal OK, and DBQL readable but not logging for this user, reported with the BEGIN QUERY LOGGING command; - the band is set inside the real ingestion path and restored, even after a failed execution, and the caller's own pairs are kept; - the follow-up ↔ DBQL join query executes: 8 real follow-up rows, all NOT LOGGED, since this user's queries are not logged. With DBQL empty for this user, the skew arithmetic was checked on literal values instead: 50 % skewed, 0 % even, NULL when no CPU was used; - the parse gate stops a run and lets one continue. Measured on the same dataset: multi-join 580 ms (5.9k characters of DDL), grouped 338 ms (3.1k).

Offline: 18 new tests; the suite has 1582 passing.

[0.3.1.221] - 2026-10-02

Added - EFS map panel in the admin app

A new EFS map panel shows every object of the feature store, for architects and DBAs, split into two planes.

  • Metadata plane — the catalogs that describe, govern and drive the data, in six groups:
    • feature catalog;
    • process catalog;
    • run history and run drivers, including the FilterManager / TimeManager objects named by the process catalog;
    • dataset catalog;
    • business dictionary and documentation;
    • lineage graph.

Each hidden table is paired with its public view, with rows and perm size. Unknown FS_ objects go to an Other group rather than being dropped. - Data plane — the FS_T_* storage tables grouped by entity (the key column set, order-insensitive), between the process views that write them and the datasets that read them. Each table card shows: - the number of features and of rows, with a bar relative to the largest table; - the value type, perm size and AMP skew; - the primary index and the partitioning expression; - the companion FS_V_* view.

A table listed in the feature catalog but absent from the database is flagged TABLE MISSING. - Row counts come from the table-level statistics by default. Count rows exactly runs COUNT(*) in CURRENT and NONSEQUENCED VALIDTIME, one UNION ALL per 25 tables, with a per-table fallback so one unreadable table does not hide the others. Current and history rows are then shown separately. - Rows per feature: a bar chart per storage table, one bar per feature and process version, current and history. A FEATURE_ID that is not in the catalog for that table is shown in red. Only storage tables of the store are accepted. - Export HTML writes the map as one standalone file.

The panel is built by a new module, tdfs4ds/serve/efs_map.py: - collect: catalogs read once; the dictionary (TablesV, IndicesV, PartitioningConstraintsV, StatsV, TableSizeV) read in batched IN lists, reusing feature_store.statistics; - model: a pure model builder; - render: an HTML renderer with scoped CSS and inline SVG, no script.

Routes: GET /ui/efs?exact=, GET /ui/efs/table?db=&table=, GET /ui/efs/download. process_storage_tables accepts catalog frames already read.

Found while building it: in a UNION ALL, Teradata types an uncast literal column from the first branch, so the db.table labels of the batched count were truncated and most counts were lost. The labels are now cast to VARCHAR.

Verified live on Vantage, read-only: - the map: 7 storage tables in 4 entity groups and 32 metadata objects with their counts, in 13 s; - exact counts equal to the statistics; - the rows-per-feature chart in 3 s; - through the HTTP routes, 9/9, including refusing a catalog table as a chart target and the standalone export.

Offline: 12 new tests.

[0.3.1.220] - 2026-10-02

Fixed - datasets were missing from the lineage graph

build_dataset wrote a dataset's lineage only with record_lineage=True, which defaulted to False, and rebuild_dataset / build_grouped_dataset never wrote it. A dataset therefore reached FS_LINEAGE_GRAPH only after a full rebuild_lineage_graph(). On the test store none of the 7 datasets was in the graph, although the graph built from each one is correct (14 edges for ds_m5_diagnostic_features_all, down to its source table).

  • record_lineage now defaults to True. No LLM is involved, the write is diff-based (an unchanged dataset writes nothing) and a failure is a warning. The internal upload_features / run() call now records it too.
  • rebuild_dataset and build_grouped_dataset record it.
  • Lineage panel: registered datasets absent from the graph are named, with an Add them to the lineage button (POST /ui/lineage/add-datasets → panels.lineage_add_datasets, which records only those datasets).
  • The persisted-graph page documented an upload_features(record_lineage=) parameter that does not exist; corrected.

Fixed - admin app: Benchmark did not see a newly created dataset

Panels are rendered once and kept, so the Benchmark dataset list was read only the first time the panel opened. A grouped dataset created in Maintenance then stayed invisible until the page was reloaded. The list now reloads each time the panel is shown again, and a ↻ Refresh list button reloads it on demand (the selection and the last report are kept).

Changed - grouped conversion confirms the catalog registration

Maintenance → Datasets — grouped version now reads the dataset catalog after the conversion and says whether the dataset is registered, with its id and strategy, or points at Dataset catalog → Restore if it is not.

Changed - consumer agent: large listings become an overview

A store with 300+ features answered "what features are available?" with every feature. Above the new tdfs4ds.AGENT_LIST_MAX_ITEMS (default 40; 0 restores the former behaviour) the SUMMARY skill (store, data domain, entity or dataset scope) answers with:

  • the count and the breakdown by entity;
  • the main themes;
  • at most a few examples per group;
  • a fixed EN/FR footer with questions that narrow the request, naming the largest real entities.

Only the examples reach the model, so it cannot enumerate everything; the full list stays in the skill result for the context graph and the follow-up cache. The data-domain list shows the largest domains and says how many more exist.

Verified live on Vantage, 9/9 with scratch objects: the Lineage panel named ds_m5_diagnostic_features_all as missing; a grouped conversion through the app reported its registration (strategy GROUPED), recorded its lineage and appeared in the Benchmark list; Add them to the lineage recorded a dataset; the real lineage edges and catalog rows were unchanged. Offline: 15 new tests; full suite 1570 passed, the 6 known vector-store baseline failures apart.

[0.3.1.219] - 2026-10-02

Added - restore the dataset catalog from the admin app

Maintenance → Dataset catalog runs restore_dataset_catalog through the Maintenance live-log route (op=dataset_catalog, restore_mode=check|apply, drop_orphans=, datasets=):

  • Check is the dry run; Restore applies after a confirmation.
  • The result table shows, per dataset, the status, how the view is built, what disagrees and the action.
  • restore_dataset_catalog now logs a dataset catalog PROGRESS k/N line per dataset, which drives the progress bar.

Verified live through panels.rebuild_run, 9/9: a store-wide check found the 6 real datasets needing repair and wrote nothing; a scoped restore of a corrupted scratch dataset fixed its valid time, ids and build row with the same DATASET_ID; the real datasets' rows were untouched.

Fixed - cross-store migration dropped a dataset's build options

migrate_objects rebuilds each migrated dataset with build_dataset, without its options, so a grouped, fixed-date or TimeManager dataset arrived in the target store as a current multi-join one. The strategy, join type and point in time are now read from the source view's DDL (_dataset_build_options) and passed through.

A TimeManager dataset is re-pointed to the manager's migrated copy when the plan placed one. Otherwise it keeps referencing the source store's manager, and the migration report says so (note), so it can be re-pointed with rebuild_dataset(..., time_manager=...).

Docs - the dataset catalog schema

concepts/data-model.md named a dataset catalog table that does not exist (FS_DATASET). It now lists the four tables and views (FS_FS_DATASET_CATALOG, _ENTITY, _FEATURES, _BUILD) with their columns, checked against DBC.ColumnsV on a live store.

[0.3.1.218] - 2026-10-02

Changed - TimeManager datasets keep partition elimination

A dataset built with time_manager= used VALIDTIME AS OF (SELECT BUSINESS_DATE FROM <tm view>) in each feature sub-query. Reading the EXPLAIN steps (not just their count) showed that, in the grouped strategy, it makes Teradata read every storage table "by way of an all-rows scan", where a literal date reads "from 3 partitions of…". That is a full scan of each storage table on every query.

Each sub-query is now NONSEQUENCED VALIDTIME with the period filtered by the manager's business date as a scalar: B1.ValidStart <= (SELECT CAST(BUSINESS_DATE AS TIMESTAMP(0) WITH TIME ZONE) FROM <tm view>) AND B1.ValidEnd > (…). This is the predicate Teradata itself generates when it rewrites an AS OF, so the instant selected is the same.

Verified live on a 10-feature dataset, both strategies: - rows identical to a literal date, for a full read and a single-entity lookup; - partition elimination in every storage read (primary-index access for the single entity), no all-rows scan of a storage table; - the business date read once per query, through the manager's primary index; - the dataset follows update().

The business date is deliberately not wrapped in MAX(...). That would raise the optimizer's estimate of the manager view from "2 rows, low confidence" to "1 row, high confidence", but it changes nothing in the feature-table plan, because the scalar is substituted as a parameter first. And a manager view that wrongly returned two rows would then pick one date silently instead of failing with Error 3669.

DATASET_VALIDTIME now records TIME_MANAGER <db>.<view> for such a dataset. Views written with the earlier shape still parse as TIME_MANAGER; rebuild them (rebuild_dataset(view, replace=True)) to get the new filter. CURRENT and AS_OF datasets are byte-identical.

Added - convert a dataset to its grouped version from the admin app

Maintenance → Datasets — grouped version lists the registered datasets with their strategy, join type and time mode (GET /ui/datasets). It converts one to the grouped strategy, either in place (same name, same DATASET_ID) or as a new view (default <name>_GROUPED, optional overwrite), and can preview the SQL first (POST /ui/datasets/grouped, backed by panels.dataset_grouped → build_grouped_dataset).

  • An in-place request on an already-grouped dataset does nothing and says so.
  • A refusal (target exists, not a dataset view) is shown with its reason. The panel forces DISPLAY_LOGS on for the call because logger_safe drops every message, errors included, while it is off.

Verified live, through the panel functions (7/7) and the HTTP routes (listing, preview, convert, refusal, unknown view).

Fixed - connect() re-created the build table on every call

connect() checks a fixed list of catalog object names, and the build table/view were missing from it. The catalog therefore saw them as absent, retried their CREATE, hit Error 3803 and logged a misleading "could not create the optional object" warning. The pair is now on the list, and an "already exists" during an optional creation is treated as present.

Offline: 1547 passed (7 new in tests/test_datasets_panel.py); the 6 failures are the known vector-store baseline.

[0.3.1.217] - 2026-10-02

Added - review, complete and refresh the storage statistics (check_storage_statistics, admin app)

Ingestion's normal path refreshes only the statistics a storage table already has, so the (entity columns, FEATURE_ID) statistic added in 0.3.1.216 never reaches an existing table on its own. tdfs4ds.check_storage_statistics(domains=, processes=, tables=, add_missing=False, refresh=False) (new module tdfs4ds.feature_store.statistics) compares each FS_T_* table's statistics (DBC.StatsV) with the expected set. It reports OK / MISSING / NO_STATS / TABLE_MISSING with the missing column sets, statistics defined outside tdfs4ds (EXTRA, never dropped) and the oldest/newest collection date.

  • Default is read-only. add_missing=True defines what is absent with ingestion's own options, so later merges keep it current. refresh=True runs the same COLLECT STATS ON <table> ingestion runs.
  • Batched reads. The data dictionary is read in four batched queries (TablesV, IndicesV, PartitioningConstraintsV, StatsV), not one per table.
  • One rule. The expected set comes from the new expected_statistics(), which generate_collect_stats now renders from (output unchanged). The check therefore cannot disagree with ingestion.
  • TABLE_MISSING was found live: a test cleanup had dropped a storage table but left its feature-catalog rows. The first version reported it as NO_STATS and tried to collect on it (Error 3807). It is now detected via DBC.TablesV and never acted on.
  • Admin app. A Storage statistics card in Maintenance (Check, Add missing, Refresh, Add missing + refresh, scoped by data domain and process, with progress streamed to the live log). A read-only Storage statistics section in the Report, included in the exported HTML. The Report's process → storage-table map now delegates to the same package function.

Verified live through the real HTTP routes, 11/11. Covered: store-wide check (6 MISSING, 1 TABLE_MISSING), add + refresh scoped to one process's two tables, re-check OK, refresh alone, and the Report section. The added statistics were dropped afterwards, restoring the tables' original state.

Offline: 1534 passed (12 new in tests/test_storage_statistics.py); the 6 failures are the known vector-store baseline.

[0.3.1.216] - 2026-10-02

Added - datasets at a fixed date or following a TimeManager

build_dataset reads CURRENT VALIDTIME by default (unchanged). Two new arguments choose another point in time:

  • as_of=: a fixed date. ISO strings ('2024-03-31', '2024-03-31 23:59', '2024-03-31T23:59:59Z', '... +0200'), datetime, date, pandas.Timestamp and numpy.datetime64 are normalised by tdfs4ds.format_timestamp_literal. A bare date becomes midnight: Teradata rejects a date-only TIMESTAMP literal (Error 3706, verified live). Ambiguous input such as '03/31/2024' is refused, not guessed.
  • time_manager=: the dataset follows a TimeManager (or a 'db.view' name). Each feature sub-query reads VALIDTIME AS OF (SELECT BUSINESS_DATE FROM <manager view>), so tm.update(i) moves the dataset without rebuilding it. This shape was chosen by comparing EXPLAINs against two join-based alternatives (NONSEQUENCED VALIDTIME + CROSS JOIN with a range predicate, and a ValidPeriod CONTAINS join). Its plan matches a literal date (2 extra steps, no product join), while the alternatives doubled the steps and added one product join per sub-query. Verified live for DATE, TIMESTAMP(0) and TIMESTAMP(0) WITH TIME ZONE business dates, including the exact day boundary. Superseded in 0.3.1.218: in the grouped strategy this shape loses partition elimination; the period is now filtered with a scalar WHERE instead.

The three modes (including the historical feature_store_time=True) are mutually exclusive. feature_store_time=True now normalises FEATURE_STORE_TIME the same way, which leaves a well-formed value byte-identical.

Added - rebuild a dataset from its own DDL: build_grouped_dataset, rebuild_dataset

New module tdfs4ds.dataset.ddl_spec reads a dataset view's DDL back into its specification (parse_dataset_ddl): entity, and per feature the storage table, FEATURE_ID, process version and column name, plus join type, strategy and temporal mode. render_dataset_ddl writes it back. Both strategies now render through this one function: the two builders each carried their own copy of the template, and they are now thin wrappers. On 24 configurations the output is byte-identical to before, apart from the HAVING change below.

  • build_grouped_dataset(view) writes the grouped version of an existing dataset (default target <view>_GROUPED, or in place with replace=True).
  • rebuild_dataset(view, grouped=, join_type=, as_of=, time_manager=, current=) is the general form. It can also re-time an existing dataset.
  • read_dataset_spec(view) returns the parsed spec.

Features are not re-resolved against the catalog, so the result does not depend on DATA_DOMAIN or on the versions registered today. An existing target is refused unless replace=True.

Why grouped matters, measured with EXPLAIN on Vantage 20 (10 features, 3 table/version groups): the view SQL drops from 5,842 to 2,972 characters, JOINs in the plan from 9 to 2, EXPLAIN (parse + optimise) from 627 to 190 ms, and a single-entity lookup from ~455 to ~248 ms. The rows are identical (bidirectional EXCEPT ALL = 0).

Fixed - the grouped strategy dropped entities whose feature value is NULL

The grouped INNER form filtered HAVING MAX(CASE ...) IS NOT NULL. The multi-join keeps an entity whose stored value is NULL (the row exists), so the two strategies returned different rows despite the docstring promising the same semantics. It now tests row existence: HAVING COUNT(CASE WHEN FEATURE_ID=n THEN 1 END) > 0. Verified live on a scratch temporal table: multi-join {1,2}, old grouped {1}, new grouped {1,2}.

Added - restore_dataset_catalog(): list every dataset and repair the catalog

Earlier versions wrote parts of the dataset catalog wrongly, and nothing read those parts back. Observed on a live store, where every row was affected:

  • DATASET_VALIDTIME = 'SELECT' (pre-0.3.1.214 registrations);
  • FEATURE_ID stored space-padded (' 6'). The column is VARCHAR, the MERGE passed a bare integer and Teradata padded it on the implicit cast. Now written as a string literal;
  • (pre-0.3.1.79) a view registered with zero features;
  • a view registered twice made every later add_dataset fail. That path raised a bare raise (a RuntimeError: No active exception); it now raises a RuntimeError naming the fix.

restore_dataset_catalog(databases=None, discover=True, dry_run=True, drop_orphans=False, datasets=None) parses each dataset view and compares it with the catalog. It returns one row per dataset with STATUS (OK/REPAIR/REGISTER/DUPLICATE/ORPHAN/UNPARSEABLE), ISSUES and ACTION. Dry run by default. Repairs keep the DATASET_ID (the MERGE is keyed on it); a duplicate keeps one id and closes the others; orphans are closed only with drop_orphans=True. Unregistered dataset views are found through a DBC.TablesV scan (discover=False disables it).

Added - the build table: how each dataset was built

The catalog gains FS_<name>_BUILD / FS_V_<name>_BUILD, read by tdfs4ds.get_dataset_build(): BUILD_STRATEGY, JOIN_TYPE, TIME_MODE, AS_OF_TIMESTAMP, TIME_MANAGER_DATABASE/_VIEW/_TABLE, and SOURCE_DATASET_ID for a rebuilt dataset. It is a separate table, so the existing catalog tables need no ALTER and older versions read them unchanged. DATASET_VALIDTIME also records the governing qualifier for every mode.

It is created by the catalog's self-heal (setup(), connect(), any dataset call). It is optional: a user without CREATE rights gets one warning per process, and connect() keeps succeeding as before; the creation is not retried on every read. create_catalog() now reuses the object sets connect() already holds instead of listing the database again. drop_dataset also closes the build row.

Added - statistic on (entity columns, FEATURE_ID)

DIAGNOSTIC HELPSTATS on dataset queries asked for it. Without it the optimizer estimated a single-entity lookup on a multi-join dataset at 7.79 s instead of 0.02 s (measured live, with planned product joins around the misestimate). generate_collect_stats now includes it in its column set. That set runs when a storage table has no statistics yet; afterwards ingestion refreshes the existing set with COLLECT STATS ON <table>. So a table created from now on gets it, and an existing one needs it defined once: tdfs4ds.check_storage_statistics(add_missing=True) (0.3.1.217) or Maintenance → Storage statistics → Add missing.

Verified

Offline: 1522 passed. The 6 failures are the known vector-store baseline. The new tests are 82 in tests/test_dataset_temporal.py (including a DDL captured from a live store) plus updated catalog/stats tests. Live on Vantage 20: 31/31 end-to-end checks. Covered: CURRENT, grouped, build_grouped_dataset, as_of before/after the data, a TimeManager dataset moving with update(), rebuild_dataset grouped + TimeManager, catalog and build rows for every mode, unpadded ids, and restore of REPAIR/REGISTER/DUPLICATE/ORPHAN with ids preserved. A scoped apply left the real datasets' rows untouched. All scratch objects were removed.

[0.3.1.215] - 2026-10-01

Added - renew an expired LLM token mid-run instead of losing the run

With a temporary credential -- typically a short-lived Bedrock bearer token in INSTRUCT_MODEL_API_KEY -- a long rebuild_documentation() or vector rebuild could outlive the token. The column-documentation call has no try/except, so the first call after expiry raised out of the documentation walk and the whole rebuild stopped. The EXPLAIN and dataset-description calls, which do catch their errors, quietly produced empty documentation for every object after that point.

Every LLM call site already rebuilds its client from tdfs4ds.INSTRUCT_MODEL_API_KEY on each call, so once the session holds a new token, retrying the failed call is enough -- nothing has to be resumed. New module tdfs4ds.utils.credential_renewal:

  • is_credential_expired_error(exc) matches AWS's own wording (ExpiredTokenException, UnrecognizedClientException, "the security token included in the request is expired/invalid", the botocore error code, ...), walking __cause__ and never __context__. No bare word or number -- the same rule as the 0.3.1.212 reconnect classifier, with negative tests against 400/422/503, context-window and structured-output errors and the teradatasql boilerplate header.
  • call_with_credential_renewal(fn, what=, on_renewal=) runs fn(). On an expiry error it asks for a renewed token (hidden input; notebook kernels supported through the existing ai_studio prompt helpers), installs it in INSTRUCT_MODEL_API_KEY, in EMBEDDING_MODEL_API_KEY (only when that is set explicitly) and in AWS_BEARER_TOKEN_BEDROCK, then retries. It allows tdfs4ds.CREDENTIAL_RENEWAL_ATTEMPTS attempts (default 3; 0 = never ask). fn builds its client inside itself, because a client built before the renewal still carries the old key.
  • A blank answer means stop. The declined key is remembered, so a 300-process rebuild does not ask 300 more times; setting a different key, or starting a new run, re-enables prompting.
  • When there is nobody to ask (a server, papermill, a script), nothing is prompted and the original error propagates, as before.

Wired into the four documentation LLM sites (column documentation and its formula-consistency retry, EXPLAIN analysis, [You]-item classification and the dataset description), inside their existing try blocks so that renewal happens before a non-fatal handler can swallow the error. If renewal is declined, _document_from_graph stops the walk cleanly and returns the partial summary with stopped_on_credentials=True and a warning that says how to resume: re-run with force_update=False, which skips everything already written. rebuild_documentation then also skips the vector-index update.

Changed - vector indexing (Chroma) only embeds rows that changed

build_vector_index(force_rebuild=False) used to re-embed every business-dictionary row on every call. Rows already stored with the same text, metadata and embedding model are now skipped. Each row records the model that embedded it in a new embedding_model metadata key, so switching EMBEDDING_MODEL_MODEL re-embeds everything. The lookup is fail-open: a Chroma read error re-embeds everything, as before. The first build after upgrading re-embeds once, because existing rows have no embedding_model key yet. The log now reports N embedded (new or changed), M skipped (unchanged).

The batch size is configurable through tdfs4ds.EMBEDDING_BATCH_SIZE (default 8, unchanged), and an expired embedding token is renewed the same way (the embedding client is rebuilt and the current batch retried). The Teradata Enterprise Vector Store backend embeds inside the database with its own credential and is unaffected by either change.

[0.3.1.214] - 2026-09-24

Fixed - DATASET_VALIDTIME recorded the 5th line of the DDL

Spotted in tdfs4ds.dataset_catalog() on a live store: the column read SELECT A1.CUSTOMERID.

Dataset._get_validtime was self._get_ddl().split(' ')[4].strip() -- the 5th line of the SHOW VIEW text. The generated DDL begins with a newline (the builder's f-string opens with one), so that index lands on the SELECT, and every registered dataset recorded a fragment of its own select list as its valid time. Nothing in the package reads the column back, which is why it survived: it is write-only metadata, displayed to the user and to nobody else.

_extract_validtime(ddl) now scans for the qualifier instead of indexing a line, and handles both shapes dataset/builder.py emits: normally an outer CURRENT VALIDTIME, and for a time-travel build an empty outer qualifier with VALIDTIME AS OF TIMESTAMP '...' inside each feature sub-query. The outer one wins, because it governs the view -- the scan stops at the first SELECT, so the SEQUENCED VALIDTIME inside the sub-queries is never mistaken for it. Neither present yields '': an empty valid time is honest, a wrong one is not.

Fixing it exposed a second bug, so both ship together. The catalog MERGE interpolated the value raw, as '{dataset.valid_time}'. That was harmless only while the recorded value was a SELECT fragment, which happens to carry no quotes -- the correct time-travel qualifier contains apostrophes, and would have closed the literal early and broken the MERGE that registers the dataset. The value is now escaped with sql_literal, which escapes without wrapping (the caller supplies the quotes, per its own docstring); a test asserts the rendered literal whole, having caught this being got backwards during the fix.

Pre-existing rows keep their wrong value until the dataset is registered again (DatasetCatalog.add_dataset, i.e. any re-run of build_dataset); the MERGE is keyed on DATASET_ID, so re-registering repairs it in place.

Guarded by tests/test_dataset_validtime.py (16 offline tests, no DB), including a premise test that line 5 of the generated DDL really is the SELECT (without it the rest could pass vacuously) and a source-level guard that both builders keep {outer_validtime} immediately before SELECT {select_query} -- the relationship the scan depends on.

[0.3.1.213] - 2026-09-24

Fixed - the classifier was taught an intent its own schema forbids

Reported from the field, on a deployed AgentOps agent and reproduced in a plain Python session: "Which features tell me how much a customer spends?" answered fine, and the follow-up "Could you show me a sample?" died with

RuntimeError: Intent classification failed: 14 validation errors for IntentResult
intent.literal['BUILD_DATASET'] Input should be 'BUILD_DATASET'
  [type=literal_error, input_value='DATA_QUERY', input_type=str]
... one per available intent

Two defects, and the first one caused the second.

(1) The prompt advertised what the schema refuses. build_graph drops an unavailable intent from the descriptions AND from the dynamic Pydantic Literal, so with MCP off DATA_QUERY and ALGORITHM are rejected values. But _build_classifier_examples walked the skill catalog unfiltered, so the prompt still carried all five of ca-data-query's few-shot examples -- including, literally, "Montre-moi un echantillon de DS_SCORING". The model was shown "a sample -> DATA_QUERY", answered exactly that, and was rejected for it. The same leak existed for ALGORITHM, masked only by the deterministic _looks_like_algorithm_question pre-guard that fires before the classifier. Examples are now filtered by the intents the classifier may actually return, read from the model itself -- the authority that does the rejecting -- and each built-in anchor example carries its intent as data rather than being parsed back out of its rendered text. available_intents=None keeps every example, so any existing caller is unchanged. A comment in build_graph claiming the classifier "physically cannot emit" a gated intent is corrected: it was wrong twice over.

(2) An unusable classification ended the turn. intent_classifier re-raises as RuntimeError, and nothing caught it, so the user got no answer at all. But whether the schema is enforced is the endpoint's decision, not ours -- the same gateway had already logged structured_output: this endpoint does not enforce the requested response format and fallen back to function_calling, and the traceback confirms it: the failure surfaces in PydanticToolsParser, i.e. the enum travelled in a tool definition the gateway relayed without constraining. On such an endpoint an out-of-vocabulary intent is an expected outcome. _node_classify_impl now recovers: when the rejected value is an MCP-gated intent and MCP is off, the turn answers with a fixed EN/FR message saying that reading the data itself needs MCP and naming what the agent can do (the existing ALGORITHM_UNAVAILABLE precedent, reached reactively rather than from a question-shape regex); otherwise it degrades to SEARCH. It re-raises only when nothing is known well enough to route -- inventing a route would be worse than the error. An MCP-enabled rejection is deliberately NOT reported as "enable MCP", since there the intent is available and the fault is real.

Guarded by tests/test_intent_vocabulary.py (23 offline tests, no DB/LLM/network), 21 of which fail against the pre-fix code, built on the verbatim error text from the field, plus a cross-check that every intent named in the rendered prompt is one the model accepts, and a premise test asserting the shipped skill still carries the examples that would leak (without it, the filter tests could pass vacuously).

[0.3.1.212] - 2026-09-24

Fixed - every SQL error was treated as a lost connection, and the "recovery" ended the session

Reported from the field: connect() died on a store where nothing was wrong, and every statement afterwards failed with [Error 544] TD2 logon mechanism requires username and password -- a notebook whose own create_context had just succeeded. The connection was never lost. tdfs4ds destroyed it, in response to a nominal [Error 3803] Table 'FS_TIME_MANAGER' already exists.

Three independent defects, each needing its own fix.

(1) The classifier said yes to everything. is_connection_error's marker list contained "session", and EVERY teradatasql message opens with [Session <n>] -- so every SQL failure in the package, down to a division by zero, classified as a dropped connection. A bare "803" compounded it by matching the error codes 3803 / 3807 / 2803 as a substring. Live since v0.3.1.78 (e641041), ~130 patch versions, with no test covering it. The new rule is a veto: a message carrying [Teradata Database] was formatted from a database response, so the server received the request and replied -- whatever went wrong, the connection was alive, and no positive marker is even considered. Only then is the text matched against transport phrases, none of which is a bare number any more (socket, connection reset, broken pipe, failure sending request, SQLState 08S01/08003...). SQLState 08000 is deliberately absent: the driver reports its own configuration refusals with it, and Error 544 is 08000. The asymmetry is intentional -- a missed reconnect surfaces the real error, which is recoverable; a spurious one is not. Classification walks __cause__ only, never __context__ (an unrelated exception in flight would otherwise contaminate the verdict), which is the one place it differs from _describe_exception.

(2) reconnect_teradata destroyed a working context on its way to a broken one. It called tdml.remove_context() first, and create_context assigns teradataml's global engine before testing the connection -- so a failed reconnect left get_context() returning a poisoned engine, and every later statement in the process failed with that logon error instead of its own. The session could not self-heal; that is how one "already exists" ended a notebook. Credentials are now probed with the raw driver first (_credentials_probe, the jwt_logon_verdict constraint: teradataml holds ONE process-wide context, so a check must not be able to end a session), and a refusal leaves the existing context untouched. Only a refusal blocks -- an unreachable host is inconclusive and still reconnects, since that is the case the feature exists for. No extra failed logon: on a refusal the probe replaces the create_context that used to fail. If create_context does fail, the broken engine is now cleared.

(3) A token and a password mechanism were combined into a credential set that cannot work. _connect_kwargs preferred TDFS4DS_TD_LOGDATA unconditionally, so a .env holding LOGMECH=TD2 and a leftover LOGDATA sent host+username+TD2 and no password -- refused by the driver before it reached the network, while the same file's USERNAME+PASSWORD would have worked. When LOGMECH names a password mechanism (TD2/LDAP) and LOGDATA demonstrably carries no password=, the working pair now wins, with a warning naming both variables. TDNEGO/KRB5 are excluded (they can complete without a password) and a LOGDATA that carries its own credentials is left alone, so every existing token and password deployment produces byte-identical kwargs.

Not fixed by 0.3.1.211. That release quieted the log line, but create_object_if_missing still calls execute_query(..., raise_error=True) and the exception still passed through the misclassifier before the caller's is_already_exists_error ever saw it.

Guarded by tests/test_connection_error_classification.py (48 offline tests, no DB/LLM/network), 31 of which fail against the pre-fix code, including the incident end to end through execute_query / create_object_if_missing and source-level guards (no marker may be a bare number; the boilerplate header every Teradata message carries must match nothing; the probe must use raw teradatasql, asserted on the parsed code since the docstring names create_context to explain why it is not used). Offline evidence only -- not yet re-confirmed live.

[0.3.1.211] - 2026-09-23

Fixed - a fully provisioned store reported itself as broken

Reported from the field: dozens of ERROR lines per run on a store where nothing was wrong, making the logs useless for supervision -- one cannot watch for ERROR to detect a real problem when the nominal case emits a dozen of them. Measured on Vantage 20 before the fix: 13 ERROR lines for a setup() on an already-provisioned store, 1 for every connect(). After: 0 and 0.

The mechanism was not where it looked. execute_query defaults to raise_error=False: on a failure it logs the error itself and returns normally. So the except wrapped around every catalog CREATE never ran -- the endswith('already exists.') test, the if_exists == 'replace' branch and time_manager_catalog_creation's careful logger_safe('debug', ...) were all dead code. Execution fell through to logger_safe('info', '... has been created'), so each table also produced a second, contradictory log line claiming a creation that never happened. A fix applied to those except blocks would have changed nothing.

A second site printed the whole driver message. documentation_tables_creation interpolated the entire multi-line exception into a logger_safe('error', ...) placed before its already-exists test -- six full Teradata error blocks per setup(). The fix is not to truncate to the first line (the half that names the fault is often the later half): the benign case simply stops being logged as an error, and a genuine failure keeps its full message.

The dangerous part was not the noise. setup(if_exists='fail') -- the default -- calls process_store_catalog_creation() and feature_store_catalog_creation() with no argument, hence with their default if_exists='replace'. Their DROP TABLE branches were unreachable only because the exception was being swallowed. Making the failure visible without touching that default would have made every setup() on an existing store drop FS_PROCESS_CATALOG and FS_FEATURE_CATALOG. Both defaults are therefore now 'fail', which preserves the observable behaviour exactly (setup() still does its own explicit dropping when asked). An offline test asserts no DROP is ever emitted, and the live run verifies the catalogs' DBC.TablesV.CreateTimeStamp is unchanged across a second setup().

The shared mechanism. utils/query_management.py gains is_already_exists_error(exc) -- the single authority for "the object is already there", which for an idempotent CREATE is the nominal outcome, not an error -- and create_object_if_missing(create_query, label, follow_up=()), which reports honestly what happened and raises (ERROR, full message) only on a real failure. execute_query gains quiet_errors=(); the empty default leaves all 58 existing call sites byte-identical, per the package's opt-in convention. Routed through it: the four process-store catalog tables and their secondary index, FS_TIME_MANAGER, the feature catalog and its index, the follow-up table (which had been printing the first error line and the entire CREATE statement to stdout via @execute_query_wrapper), the six documentation tables, and lineage_graph_catalog_creation -- already correct, and the model the rest now follows.

Live-verified on Vantage 20 against a throwaway database, one session (a freshly created database can transiently read as absent in the next one). The run also corrected an assumption: a secondary index reports this condition as Error 3534, "Another index already exists, using the same columns and the same ordering" -- a different code and different wording from a table's 3803, which is why the predicate matches the phrase rather than the code. Guarded by tests/test_create_if_missing.py (7 offline tests, no DB), four of which fail against the pre-fix code.

[0.3.1.210] - 2026-09-22

Fixed - a failed ingestion looked exactly like a clean run

Reported from the field: a process failed with [Error 2646] No more spool space, and the orchestrator driving the waves saw 5 OK, 0 failure(s). The follow-up row stayed RUNNING, later levels computed on features that were never written, and those levels recorded COMPLETED -- so the wrong values were then skipped on every subsequent run. Three independent defects, each of which had to be fixed separately.

1. The per-filter loop had no failure channel. When a FilterManager segments a process, each filter ran in a protected block that logged the error and moved on. No counter, no exception, nothing in the return value; something_computed flipped to True on the first success, so a process with nine failed filters out of ten was indistinguishable from a clean one.

Two things made this worse than it looks. An unsegmented process (no FilterManager) propagates its error normally, so the same failure was loud or silent depending only on whether a FilterManager was attached. And the parallel execution script that tdfs4ds generates itself detects failures by catching an exception from run() -- so the package's own scheduler was exactly as blind as any user's orchestrator.

2. FAILED could never be written -- and this was never a spool problem. followup_close built the UPDATE and then executed a bare raise before returning it. Its decorator calls the function to obtain the query and only then executes it, so the exception escaped first and the statement was discarded. Callers wrapped the call in except Exception: pass to absorb that raise, which hid it completely. The effect: no installation has been able to write a FAILED status since commit bbc3746 (2024-12-04). Anyone observing that STATUS only ever held RUNNING and COMPLETED was seeing a product defect, not something in their environment.

This also silently disabled every feature built on it -- _freshness_segments and the consumer agent's has_recent_failure both look for a FAIL prefix that could not occur.

followup_close no longer raises, no longer uses @execute_query_wrapper (that decorator reports an execution failure with a bare print, and this particular write happens precisely when something has already gone wrong), escapes the status through sql_literal -- a Teradata message quoting an object name would otherwise produce malformed SQL -- and truncates it to the column width. Its WHERE now also pins VALIDTIME_DATE: because RUN_ID is generated once per import rather than once per run, a success at one time step could otherwise close the stale RUNNING row left by a failed earlier step, marking it COMPLETED.

3. An empty ingestion was recorded as a permanent success. A process reading a failed upstream's view does not fail: it ingests zero rows, succeeds, and writes COMPLETED. Since the skip test is an exact match on that value, the empty result was then treated as final. An ingestion that stages no rows is now recorded as COMPLETED,0 rows, which is neither a success the skip test recognises nor a failure (an empty partition is legitimate) -- so it is simply recomputed next time, with no manual cleanup. The cost, accepted deliberately: a partition that stays empty is recalculated on every pass.

Added - on_failure, and an optional upstream guard

run, upload_features, roll_out and _upload_features take on_failure ('continue' | 'stop' | 'raise'); raise_on_failure=True is accepted as a shorthand, and if both are given the explicit on_failure wins with a warning. The default stays 'continue', so control flow is unchanged for existing callers -- what changes by default is that failures are now recorded. 'raise' attempts every filter first, so the trace in the follow-up table is complete, then raises FeatureIngestionError, which carries process_id, n_failed, n_total and failures so an orchestrator never has to parse a message.

check_upstream=True on run()/roll_out refuses to compute when an upstream process has failed at this valid time. The test is "no upstream failed", not "every upstream completed": upstreams legitimately run on different cadences, and a guard that cries wolf gets switched off. When it blocks, nothing is computed, no follow-up row is written, and the message names the failing processes and both ways forward.

Because the guard reads the persisted lineage graph, it distinguishes three situations instead of conflating them: the graph is unavailable, the process is absent from it (typically registered after the last refresh), or the process is known -- in which case having no upstream is a real answer. The first two do not block, but they report verified=False and print the rebuild_lineage_graph(...) command that fixes them. Reporting "nothing to check" would read as an all-clear from a check that never ran.

generate_execution_script gained the wave-level wave_on_failure ('skip_descendants' by default, plus 'continue', 'stop_after_wave', 'raise'). 'skip_descendants' skips only the failed process's downstream closure and lets unrelated branches finish. The generated worker now runs each process with on_failure='raise', since a wave policy can only react to failures it is told about.

The wave setting is named apart from the per-process on_failure on purpose. The two value sets overlap without agreeing -- 'raise' is valid at both levels and means something different, 'stop' exists only per process, and 'skip_descendants'/'stop_after_wave' only per wave -- so sharing one name would make a value that is invalid at one level read as a documentation error.

Known limitation

Re-running a fixed process does not repair downstream processes that already recorded a genuine COMPLETED while computing on empty inputs. Those are skipped on the next pass and need force_compute=True, or their follow-up rows removed.

47 offline tests cover this (tests/test_failure_reporting.py, tests/test_upstream_check.py); no database, LLM or network required.

Live-verified on Vantage 20. A process segmented into three partitions -- one raising a Teradata error, one with no rows, one nominal -- recorded exactly FAILED,[...] [Error 2619] Division by zero ..., COMPLETED,0 rows and COMPLETED. With the default policy run() returned normally while leaving all three rows behind; with on_failure='raise' all three partitions were still attempted (so the trace stays complete) before FeatureIngestionError reported "1 of 3 attempted". A status carrying apostrophes round-tripped intact. A second process reading the first was then blocked by check_upstream=True, which returned without computing and without writing any follow-up row, while check_upstream=False ran it to COMPLETED.

[0.3.1.209] - 2026-09-22

Fixed - a database name with a special character produced invalid SQL everywhere

A Teradata object name is only legal unquoted when it matches [A-Za-z][A-Za-z0-9_$#]*. Anything else -- an @, a hyphen, a space, a dot, an accent -- has to be written as a double-quoted identifier. This matters because tdfs4ds.SCHEMA is routinely the logon user's own database (__init__.py defaults it to _get_current_databasename()), and an LDAP/AD login carries any of those. For such a user setup() failed on its first DDL statement and nothing downstream worked: roughly 210 sites interpolated the name raw, as in f"CREATE MULTISET TABLE {tdfs4ds.SCHEMA}.{...}".

Quoting is conditional, so an ordinary name is untouched. quote_ident('TDB') returns TDB. Verified by capturing every statement the store-creation path emits and diffing it against the same capture taken before the change: byte-identical for a plain name, fully quoted for SVC@CORP. Both are pinned by tests/test_quoted_identifiers.py.

Quoting applies to identifier slots only -- never to a credential, a SQL string literal (WHERE DatabaseName = '...' against DBC compares a string; a quoted name there reads as a column reference), a teradataml schema_name= kwarg (those quote themselves, or need a literal), or a value persisted into a catalog column. tdfs4ds.SCHEMA and view_schema() therefore keep returning the RAW name; the new tdfs4ds.schema_sql() / view_schema_sql() return the identifier form.

New tdfs4ds/utils/identifiers.py is the single authority: needs_quoting, quote_ident(name, force=False), unquote_ident, qualified(db, obj), split_qualified, sql_literal. It replaces five partial implementations (lineage/network._quote_ident, agent/query_optimizer._q, and three copies of _get_database_username) and three competing name splitters.

The existing dot-only workaround was half a fix. _get_database_username quoted a logon name when it contained a dot -- necessary, but the package then re-split the result naively: for "DEINF.CARLOSN"."temp_x", _table_name.split('.')[1] yields CARLOSN, not temp_x. split_qualified is quote-aware, so a dot inside a quoted part no longer splits the name. The DDL scanners in lineage/lineage.py and utils/lineage.py had identifier classes of [\w".], which cannot match an @; those now accept @$#. A hyphen is deliberately not added there -- it is the SQL minus operator and starts a -- comment, so it cannot be matched unambiguously; setup() logs this gap when it sees a name that needs quoting.

Live-verified on Vantage 20 against a real "FS@TEST" database: setup(), upload_features, run(), build_dataset (values correct), TimeManager, FilterManager, segmented ingestion, rebuild_lineage_graph (edges resolved through the quoted database) and the catalog reads. The identical scenario on a plain-named database behaves identically, including a pre-existing Error 3806 in the filtered follow-up close that is unrelated to quoting.

[0.3.1.208] - 2026-09-11

Fixed - 34 of 35 SKILL.md files were not valid YAML

Reported from an editor's Markdown preview:

Failed to parse frontmatter
Nested mappings are not allowed in compact mappings at line 1, column 7:
name: Feature Store: Deploy to AgentOps

Three faults, all present since the files were written: an unquoted : inside a scalar (name, description), an argument-hint beginning with [ — which YAML reads as a flow sequence, so a consumer expecting text silently got a list or an error — and values carrying double quotes.

Nothing noticed because tdfs4ds parsed its own files with a parser nobody else uses. cli._read_frontmatter split on the first colon and stripped surrounding quotes, so it read every one of these happily. Anything loading them the way they are meant to be loaded — an editor preview, an agent framework, any YAML library — failed. These files exist to be read by other tools; agreeing with those tools is the point.

  • Every frontmatter value is now a quoted YAML scalar. The rewrite asserted, per file, that no value changed meaning.
  • _read_frontmatter uses YAML first, keeping the permissive split as a fallback for an environment without PyYAML (a transitive dependency, not a declared one) and for a file that is malformed anyway — where returning what can be read beats returning nothing, since the description decides whether a skill is offered at all.
  • A parametrised test now asserts, for every shipped skill, that the frontmatter is valid YAML, that every value is a string, and that _read_frontmatter agrees with yaml.safe_load exactly.

[0.3.1.207] - 2026-09-11

Fixed - skills carrying claims this session disproved

A review of the skills touched by the AI Studio work found one wrong statement and two gaps.

fs-deploy-agentops named the wrong token. It described auth='jwt' as "the platform authenticates with a token (AI Studio %gen_token)" — conflating the two tokens whose difference cost most of a debugging session. The pod never sees %gen_token's token and cannot call it; what reaches it is the session/SSO token, which on at least one site is refused by the database. The table row no longer names %gen_token, and a note states the distinction and its consequence. The pre-deploy verification snippet also moved from tdml.create_context to raw teradatasql — teradataml holds one process-wide context, so the recommended test would have replaced the working connection in the reader's own notebook.

fs-setup never mentioned the configuration surface. It is the first call of every session and sent readers to hand-write a .env. It now covers generate_ai_studio_env() (derive, ask, check, then write — and pass %gen_token's token where available), export_config for everywhere else, and check_configuration() as the step that runs before anything is built, since it needs no feature store.

fs-vector-store documented backends but not authentication — the failure mode that dominated this session. It now carries the four mechanisms and their order, plus the four traps: the JWT path needs USERNAME when there is no active session; a ready-made JWT is stored unvalidated, so an expired one passes setup and fails later; SQLState 28000 is a refused credential, never "service down"; and the service version floor is invisible to any read-only check because an old service only misbehaves on write.

[0.3.1.206] - 2026-09-11

Fixed - regenerating the .env silently dropped what a probe had discovered

Observed live: a run recorded STRUCTURED_OUTPUT_METHOD='function_calling' — the value check_configuration(pin=True) had just established by probing what the endpoint really enforces — and a later generate_ai_studio_env() rewrote the file with the commented placeholder again. The next check reported the documentation path unconstrained, with nothing saying the setting had been dropped, and every new process went back to paying a failed request per call.

The generator derives what the platform publishes and asks for the rest. STRUCTURED_OUTPUT_METHOD and INSTRUCT_MODEL_CA_BUNDLE are neither: they are discovered — one by probing the endpoint, the other by somebody working out which CA signs it. Regenerating now carries them forward from the file being replaced, marked as such in a comment.

Order is override → previous file → commented slot. A fresh file is unchanged (the slot documents the option without turning it on), and a placeholder left in an old file is not a value.

[0.3.1.205] - 2026-09-11

Added - skill: deploy a CUSTOM LangChain agent to AgentOps

td-agentops-custom-agent covers hosting an agent of your own on AI Studio, wired to a Vantage connection, the Enterprise Vector Store and the MCP server — where fs-deploy-agentops covers the tdfs4ds consumer agent, which tdfs4ds generates and deploys in one call.

It is written around the failures that do not say what is wrong, each one established against a live deployment:

  • the entrypoint must be a bare coroutine — is_async is iscoroutinefunction(_entrypoint) and decorators apply bottom-up, so a tracing decorator between @app.entrypoint and async def makes the platform drive an async agent synchronously;
  • the pod has no DNS for the database — TD_HOST must be the IP, and a TDML_2000 in ~90 ms is a name that did not resolve, not a timeout;
  • two AI Studio tokens, only one of them a database credential, which is why the MCP gateway's refusal is the database's Error 8017 and not an HTTP 401 — and why auth='jwt' must be settled from a notebook before deploying, since a pod cannot call %gen_token;
  • 132 MCP tools overflow the context window before the question is read;
  • runtime_config is not a vault — it renders in clear in the deployment UI; its guarantee is "not committed to source control";
  • the Vector Store JWT path needs a username without an active context, and SQLState 28000 means a refused credential, never "service down";
  • thread_id must never fall back to a constant, or every caller of the service shares one conversation.

Plus a deploy checklist and a symptom table, both drawn from real failures. Registered in the skills catalogue, which was also missing fs-deploy-agentops.

[0.3.1.204] - 2026-09-11

Fixed - five consumer-agent calls asked for JSON on the unconstrained path

An audit of every JSON-expecting LLM call in the three subsystems that need typed replies:

subsystem seam covered by the probes
query optimizer with_structured_output at 7 sites; its 3 raw invokes return prose or SQL yes
documentation _build_provider_llm_caller at 3 sites yes
consumer agent with_structured_output for intent classification and EXPLAIN… yes
consumer agent …and _llm_call — an unconstrained prose call — at 5 more no

Those five asked for JSON, then hand-stripped markdown fences and ran json.loads. Two were written as:

matches = json.loads(cleaned) if cleaned.startswith("[") else []

so a model answering in prose produced an empty result with no error and no warning — one of them on the SEARCH path, the main search skill. Measured on a live LiteLLM gateway, eight of nine constraint syntaxes were accepted and ignored, and the model answered **Classification:** ... to a request for JSON. These calls also prepend the grounding and style prefixes, which push towards prose. Neither STRUCTURED_OUTPUT_METHOD nor check_configuration reached them.

All five now go through _llm_call_json(...), which:

  • sends the prompt through the provider dispatcher — so it honours STRUCTURED_OUTPUT_METHOD, tool calling included (0.3.1.203), and keeps the one-shot unconstrained fallback;
  • declares the reply schema at each site, so on an endpoint that enforces anything, this is what gets enforced;
  • parses with _robust_json_parser rather than a local regex;
  • returns the caller's own default on failure and logs what came back, naming check_configuration() and STRUCTURED_OUTPUT_METHOD='function_calling'. The silent empty is the defect; it must not be reintroduced.

A source-level guard test fails if any call site goes back to parsing JSON out of _llm_call. The failure is structural — each site looked plausible on its own, and only the pattern across five of them was wrong.

Fixed - capability checks no longer act as though a feature store exists

Proving that an endpoint does tool calling has nothing to do with a feature store, and settling it before creating one is the sensible order. But the MCP check's tool guard called _get_forbidden_objects, which read three catalogs regardless — so with SCHEMA unset it logged TDML_2008 Invalid type(s) ... 'schema_name' twice and the schema name is not defined at ERROR, then passed anyway. A report that succeeded looked broken, and implied a prerequisite that was not real.

With no SCHEMA there are no process views, feature tables or datasets to protect, so "nothing is forbidden" is the correct answer, not a degraded one. It is now returned directly. With a store configured the guard reads the catalogs exactly as before — both halves tested.

[0.3.1.203] - 2026-09-11

Fixed - documentation could not use the one syntax its gateway enforces

The nine-variant sweep, run live against a LiteLLM gateway, returned a result worth stating plainly:

syntax verdict
response_format json_schema (and strict) accepted, ignored
response_format json_object accepted, ignored
guided_json, nvext.guided_json, grammar, format, extra_body.response_format accepted, ignored
tool calling ENFORCED

One of nine. And the two structured-output systems were not equally placed to use it: the agent could switch (probe_structured_output finds function_calling, STRUCTURED_OUTPUT_METHOD pins it), while the documentation dispatcher had only four mechanisms — guided_json, response_format, format, none — none of them tool calling. So on the one endpoint offering real enforcement, documentation ran prompt-guided, and the .env recorded a setting that fixed half the package.

STRUCTURED_OUTPUT_METHOD now reaches the documentation path too:

  • 'function_calling' → the schema travels as a tool definition with tool_choice, for every provider. The provider names a vendor; this names the mechanism the endpoint in front of it honours, and those are different questions.
  • 'json_mode' → response_format={"type":"json_object"}.
  • unset, or anything unrecognised → the provider mapping, unchanged byte for byte. That is the overwhelming majority of installs, and a typo must not silently drop the constraint.

Tool calling returns the structure as arguments, not as content, and every caller of the dispatcher reads .content. _content_from_tool_call moves them across so the contract — "an AIMessage whose content is the JSON" — is identical for all mechanisms and no call site needs to know which ran. A reply carrying no tool call is returned unchanged: an endpoint that ignored tool_choice and answered in prose is the degradation the parser and check_configuration already handle, and inventing content would hide it.

A side effect worth having: on such a gateway the documentation probe stops reporting trouble, so the nine-call sweep no longer fires on every check_configuration().

Added - tool calling gets its own line in the report

Two unrelated features rest on tool calling and neither announces it: STRUCTURED_OUTPUT_METHOD='function_calling' carries the structure as tool-call arguments, and every MCP skill binds real tools. Where a gateway relays response formats without enforcing them, tool calling is often the only syntax honoured — so it stops being a detail and becomes the question the configuration turns on.

The JSON constraints row now states it outright: supported and enforced, rejected (in which case function_calling and every MCP skill are unavailable), accepted-but-not-a-tool-call, or nothing at all when nothing measured it — an unprobed capability is never reported as working or broken.

[0.3.1.202] - 2026-09-11

Added - a token's expiry is read and reported, before anything spends it

A configuration that had passed every check two and a half hours earlier came back with two failures:

Vector Store service  -> [SQLState 28000] The UserId, Password or Account is invalid.
MCP server            -> [Error 8017]     The UserId, Password or Account is invalid.

Both messages name a password. Neither configuration sends one. The token had simply expired — a ~2h lifetime, used at 2h36m — and every component that still worked was one that authenticates with an API key, which does not expire, so the report looked like two unrelated credential faults rather than one clock.

The Vector Store auth row was the worst of it: still green, because teradataml stores a ready-made JWT without validating it (its validation call is commented out upstream), so "configured" never meant "valid".

The expiry is a claim inside the token. Reading it costs nothing — no network, no service — so the new tdfs4ds/utils/tokens.py does:

  • token_expiry(token) → ('expired'|'expiring'|'valid'|'unknown', detail) with a human interval ("the token EXPIRED 37 minutes ago").
  • jwt_claims(token) — the decoder, now shared with ai_studio, which had its own. Two decoders would eventually disagree about what counts as readable.
  • bearer_from_headers(headers) — recovers the bearer from a resolved header mapping so the MCP check can report on what it is about to send.

'unknown' is not a verdict: an opaque token can be perfectly good, and a malformed one is a different fault. It is never rendered as "expired".

Wired into two checks:

  • Vector Store auth now reports the remaining lifetime, and fails on an expired token instead of passing it to the next check to fail obscurely.
  • MCP server reports it in the detail line, and when stage 1 fails with an expired bearer the hint leads with that instead of opening on the username and password that were never sent.

The MCP check deliberately still attempts the connection with a token that looks expired. exp is absolute time, so a skewed clock would otherwise make it refuse to try a token that works — and if an "expired" token does answer, that is worth seeing rather than having suppressed.

The signature is never verified. This is a caller reading a token it already holds in order to report on it, not authentication; the issuer validates it for real at the other end. Neither function raises, and no detail string ever contains any part of the token — both asserted by tests.

[0.3.1.201] - 2026-09-11

Confirmed - one token serves all three components

The first full green run on AI Studio settles what 0.3.1.199 could only hypothesise. With generate_ai_studio_env('.env', token=user_token):

use OAUTH_ACCESS_TOKEN %gen_token → user_token
database logon (TD_LOGDATA) refused — Error 8017 / SQLState 28000 accepted
Vector Store — accepted (132 collections listed)
MCP gateway bearer 8017 from the gateway's own logon accepted — 132 tools, and a tool call answered

The two are different strings. So token= feeding all three is not a convenience, it is the correct shape: the gateway opens a Teradata session with the bearer, which makes "is this a database credential?" the single question behind all three.

Fixed - the run said using OAUTH_ACCESS_TOKEN while using something else

A hardcoded message survived the addition of token= and named a source that was no longer being used. A report that is confidently wrong about where a credential came from is worse than one that says nothing — this one would have sent the next reader hunting the wrong token. It now names the real source.

Fixed - choosing the TLS route and then not checking the certificate

llm='ingress' selects https, and TD_SSL_VERIFY=false was inherited onto it, producing INSTRUCT_MODEL_VERIFY_SSL=false — encryption without authentication, which is the half that does not protect an API key: anyone able to intercept can present a certificate of their own. It also logged a warning on every single call.

TD_SSL_VERIFY is a platform-wide hint, in practice about the in-cluster services and their own certificates. The ingress was observed serving a perfectly ordinary certificate — Teradata Corporation, issued by Entrust Limited, 56 days to expiry — with verification switched off by inheritance.

So the generator now asks the endpoint instead of copying the hint. _tls_verifies(url) performs a handshake against the system trust store — sending no credential, asserted by a test — and returns True / False / None for could not tell:

  • verifies → INSTRUCT_MODEL_VERIFY_SSL is left commented out, verification stays ON, and a line says the platform hint was not copied onto this route.
  • does not verify → written as false so the endpoint stays usable, with INSTRUCT_MODEL_CA_BUNDLE named as the better fix.
  • could not tell → the platform hint, exactly as before. "Could not tell" is never read as "it is fine".

Only on https://; a plaintext URL has no certificate and is not probed.

Known - configuration-time noise from the MCP check

With no feature store configured yet (SCHEMA unset, as it is while generating a .env), the MCP check's tool guard logs _all_processes_cached failed, the schema name is not defined and a bare TDML_2008. Harmless — the check passes and the guard is fail-open — but it reads like a fault in a green report. Not addressed here.

[0.3.1.200] - 2026-09-11

Fixed - three faults one live AI Studio run showed at once

A single generate_ai_studio_env() run on AI Studio, with auth='jwt', produced a report with three failed checks and one silently wrong setting. They had three different causes.

The Vector Store token path could not supply a username

teradataml.set_auth_token() ... username should be passed in the parameters
or call create_context() before using set_auth_token()

teradataml derives the username from the active database session when it is omitted, and raises when there is no session yet. resolve_auth_mechanism's PAT branch already passed username when it was set; the JWT branch never did — so the one mechanism with no way to supply a username was the one most likely to be asked without a context, since check_configuration() is designed to run before connect().

Two halves, both needed: the JWT branch now forwards VECTOR_STORE_AUTH_USERNAME when it is set (optional, so an install relying on the active session is unchanged), and generate_ai_studio_env(auth='jwt') now writes that variable instead of emitting it commented out. The Vector Store service check failed only because the auth check did, so both rows go green together.

The LLM route was chosen silently, and the platform's only answer is plaintext

The generated file pointed at http://litellm...:4000 — the API key and every prompt in clear — because llm defaulted to 'incluster' and was never asked. The TLS route (https://{JH_FQDN}/one-td/litellm, the same one-td pattern as the MCP gateway) was written commented out, where nobody reads it.

llm=None is now the default and asks, exactly like auth: which route the traffic takes is not derivable — the platform publishes exactly one address and it is the plaintext one, so deriving means always choosing http:// and never saying so. The cost of that choice is invisible in the generated file, which shows only a URL. Without a terminal it still resolves to 'incluster', so scripts are unchanged.

A token the database refuses was written, and nothing checked it

auth='jwt' fans one token into three credentials: the database logon (TD_LOGDATA), the Vector Store, and the MCP bearer. On a site where the platform token is not database-capable — confirmed live — all three fail, and check_configuration cannot see it: the database connection is deliberately outside its remit.

So the MCP row's Error 8017 was the second symptom of a file whose database credential was already known not to work.

  • New token= parameter — the database-capable token, feeding all three. mcp_token= stays as the narrower override for a site where the gateway and the database want different tokens.
  • Under auth='jwt' the logon is attempted before the file is written, with the same raw-teradatasql probe the AgentOps preflight uses. A refusal blocks the write and prints the fix; an inconclusive result (unreachable host, no driver) is never rendered as a refusal.
%gen_token
tdfs4ds.generate_ai_studio_env('.env', token=user_token)

Changed - one definition of "is this token a database credential?"

_jwt_logon_verdict moved from agentops/preflight.py to tdfs4ds.utils.reconnect.jwt_logon_verdict(), the module that already owns the TDFS4DS_TD_* → connection mapping. preflight now delegates. Two callers now ask the question and two copies would drift — one of them silently, which is the exact failure mode the probe exists to catch. Its two source-inspecting guards follow the implementation rather than the file it used to live in, and compare parsed code rather than raw source, since the docstrings legitimately name create_context and OAUTH_ACCESS_TOKEN to explain why neither is used.

Note

A local named token already held OAUTH_ACCESS_TOKEN inside the generator, so adding a token= parameter would have let the local discard it — accepted and ignored, precisely how mcp_token= broke in 0.3.1.190. It is renamed session_token, with a test that compares the two.

[0.3.1.199] - 2026-09-11

Added - mcp_token=: the gateway needs a DATABASE credential, not just a valid bearer

AI Studio hands out two different tokens, and only one of them logs on to Teradata:

token gateway accepts it database accepts it
OAUTH_ACCESS_TOKEN (session SSO) yes no — Error 8017, proven directly
%gen_token → user_token not yet measured yes — confirmed live 2026-09-11, and it is a different string from the session token

The Enterprise gateway opens a Teradata session with the bearer it is given, which is why a failure there surfaces as the database's Error 8017 rather than an HTTP 401. So the bearer has to be a database credential — a stricter requirement than being a valid token, and one the session token does not meet on every site.

%gen_token
tdfs4ds.generate_ai_studio_env('.env', mcp_token=user_token)

OAUTH_ACCESS_TOKEN remains the fallback — it is the right answer where the platform token is a database credential — followed by MCP_TOKEN from the environment and then the device flow. The generated file now explains which token the gateway needs and what a refusal looks like, so a reader hitting 8017 can tell from the file alone that it is a database refusal and not a bad bearer.

Fixed - mcp_token= was silently discarded

An mcp_token = None initialiser predating the parameter reset it before the resolution order ran, so the argument was accepted and ignored. Caught by its own test.

[0.3.1.198] - 2026-09-11

Added - the preflight verifies the condition auth='jwt' rests on

Confirmed live, and it settles two open questions at once: the platform's session token is refused by the database as a JWT logon — Error 8017, SQLState 28000 from create_context(logmech='JWT', logdata='token=...').

So in this environment auth='jwt' is not deployable, exactly as log_insights warned it might not be, and it also explains the MCP failure that had been open for several rounds: the gateway accepts the bearer, then attempts its own database logon on the caller's behalf and meets the same refusal. That is a Vantage-side trust setting for the token's issuer, not anything tdfs4ds configures.

The preflight checked only that TDFS4DS_TD_HOST was set under jwt — so it would happily ship a pod in which every request fails this way. It now attempts the logon when it can:

  • refused → the check FAILS and says no deployment setting will rescue it;
  • accepted → reported as verified;
  • cannot tell → ok, with the exact command to settle it — never silently a pass.

Two constraints, each with a test: it uses raw teradatasql, never tdml.create_context, because teradataml holds one process-wide context and a preflight must not replace the caller's working connection; and it reads the token from the named TDFS4DS_TD_LOGDATA, never the platform's unprefixed OAUTH_ACCESS_TOKEN — a preflight forwarding some other component's credential to the database is precisely what that rule exists to prevent.

Only a credential refusal (8017 / 28000) counts as a "no"; an unreachable host is inconclusive, not a verdict on the token.

[0.3.1.197] - 2026-09-11

Documented - which AgentOps auth mode to use, and the condition on it

auth's entire documentation was "Credential model baked into the generated agent." — a capability statement with no recommendation, and the docs table framed the choice as "when a password exists" rather than which is better.

Prefer jwt for a deployed pod, and note the reasoning is the opposite of the notebook case, where a token expiring in two hours is precisely what makes a .env stop working:

  • the token arrives per request from the caller, so expiry self-heals;
  • the pod connects as the caller, so permissions and audit reflect the real user rather than one shared account;
  • no durable database password sits in runtime_config, which is rendered in clear in the deployment UI.

service stays the default, because jwt is conditional on something outside tdfs4ds: the platform's auth_context.token must be accepted by Vantage as a JWT logon. A token minted for a platform API is not automatically a database credential — this session watched the MCP gateway accept a platform bearer and the database then refuse the logon it attempted with it (Error 8017). A pod cannot mint its own, so where the platform token is not a valid Vantage JWT, jwt is not deployable at all.

The docs, the docstring and the fs-deploy-agentops skill now carry the one-line test that settles it before deploying:

tdml.create_context(host=..., logmech='JWT',
                    logdata='token=' + os.environ['OAUTH_ACCESS_TOKEN'])

[0.3.1.196] - 2026-09-11

Changed - the auth mode is asked, not defaulted silently

auth now defaults to None, which asks when there is a terminal and the platform is AI Studio, and falls back to 'password' when there is not. An explicit auth='password' or auth='jwt' is never second-guessed.

It is the one choice in the generated file that cannot be derived and that changes how every component connects, so it belongs to the user. The prompt states the trade-off rather than just listing options:

  • password — asked once and reused for the Vector Store; works indefinitely, and is the only option that still works after the notebook is closed. A real password sits in the file.
  • jwt — your SSO identity, no password on disk, but the token expires about two hours after issue and the file then fails with errors that never mention expiry.

And the part that is easy to get wrong: neither is more secure on disk. A live bearer token is as sensitive as a password while it lasts. The real difference is how long the file stays valid.

[0.3.1.195] - 2026-09-11

Fixed - the TLS warning fired when nobody had disabled TLS

describe_config's footer read not getattr(tdfs4ds, 'INSTRUCT_MODEL_VERIFY_SSL', True). UNSET is None, and not None is True — so once v0.3.1.194 started clearing the variable for the pre-write check, the footer announced "INSTRUCT_MODEL_VERIFY_SSL is False - the API key is sent over a connection whose peer is not authenticated" about a variable nobody had set.

It now tests is False, the idiom diagnostics.py already used at both of its own sites. This is the same class of bug v0.3.1.159 fixed in _env_attr_status — truthiness is not "unset" — recurring in a different place, so it is worth saying once more: a tri-state setting (on / off / never chosen) cannot be read with a boolean operator.

[0.3.1.194] - 2026-09-11

Fixed - the pre-write check described the session, not the file

Staging set what the generated file contains and left everything else alone. So a variable the file deliberately omits kept whatever an earlier run had put in the session, and was then checked as though the file had set it.

That is how a run whose file correctly omitted INSTRUCT_MODEL_VERIFY_SSL still reported TLS verification DISABLED, warned on every call, and printed INSTRUCT_MODEL_VERIFY_SSL=False has NO EFFECT here — the value came from the previous run, not from the file under test. The same applied to a leftover MCP_PASSWORD holding <set-this>.

Staging now clears any variable the generator manages but did not set, so the check describes the file as a fresh process would load it. Clearing is scoped to the families this generator owns (TD_*, VECTOR_STORE_*, TERADATA_AI_*, INSTRUCT_MODEL_*, EMBEDDING_MODEL_*, MCP_*, STRUCTURED_OUTPUT_METHOD) — SCHEMA, DATA_DOMAIN and the rest are the user's and survive untouched — and it is undone on exit like every other staged value.

Added - the stale session is named

Clearing them for the check is not enough: they are still live in the running kernel, so the session and the file disagree until it restarts. The generator now lists them before checking, which is the difference between "why does it still say TLS is disabled?" and a one-line answer.

[0.3.1.193] - 2026-09-11

Added - llm='ingress': reach LiteLLM over TLS

Probing the platform settled what its variables do not say. The in-cluster service is plaintext-only — a TLS handshake on port 4000 fails with a read error, and there is no 443 — while https://{JH_FQDN}/one-td/litellm answers 200, the same one-td ingress pattern as the MCP gateway.

So there are two routes to the same gateway, and the choice is real:

route trade-off
llm='incluster' (default) TD_LITELLM_BASE_URL shortest path, and what an AgentOps pod takes — but http://, so the API key and every prompt travel in clear
llm='ingress' https://{JH_FQDN}/one-td/litellm TLS, at the cost of a hop

The in-cluster URL stays the default because it is what the platform publishes. Whichever is not chosen is written commented out, so switching is an edit. With ingress, INSTRUCT_MODEL_VERIFY_SSL becomes meaningful again and is emitted; with incluster it is omitted, because on http:// it only builds a custom HTTP client and warns on every call.

[0.3.1.192] - 2026-09-11

Fixed - a <set-this> placeholder read as a configured credential

describe_config reported TDFS4DS_MCP_PASSWORD as set (hidden, 10 chars). Ten characters is exactly <set-this> — a placeholder left in the session by an earlier run, hidden by the very redaction meant to protect it, and then sent to a server as if it were a password.

That cost a wrong diagnosis. v0.3.1.190 concluded from the resulting Error 8017 that the Enterprise MCP gateway rejects Basic auth outright. The password in that run was the placeholder, so the failure is explained without any claim about Basic — and whether Basic works with a correct password is still untested.

PLACEHOLDER now lives in config.py, beside the code that has to recognise it: a variable holding it reports as PLACEHOLDER - never filled in and is listed under MISSING, never as set.

Fixed - the 8017 hint blamed credentials that were never sent

With a Bearer token, MCP_USERNAME/MCP_PASSWORD are not on the wire at all — MCP_SERVER_HEADERS takes precedence — yet the hint sent the reader to check them. It now branches on what was actually sent:

  • Basic — those two variables were forwarded; compare them against a credential known to work, and watch for a leftover placeholder.
  • Custom headers — the token was accepted (an invalid one is refused earlier and differently, as Invalid JWT token), so the 8017 is the server's own database logon on your behalf. tdfs4ds sent no password, so no tdfs4ds setting will fix it.

Fixed - INSTRUCT_MODEL_VERIFY_SSL emitted for a plaintext endpoint

AI Studio's LiteLLM service is http://, so there is no TLS to verify. Mapping TD_SSL_VERIFY=false onto it disabled verification that was never happening, built a custom HTTP client for nothing, and logged "the API key is sent over a connection whose peer is not authenticated" on every single call — eight times in one check_configuration() run.

The generator now omits it for an http:// URL and offers it as a commented slot instead; it is still emitted for https://. The transport check reports INSTRUCT_MODEL_VERIFY_SSL=False has NO EFFECT here when the URL is plaintext.

[0.3.1.191] - 2026-09-11

Fixed - the MCP bearer was already in the environment

0.3.1.190 added a device flow for the Enterprise gateway's token and stated that OAUTH_ACCESS_TOKEN was "issued for a different audience". That was wrong. Decoded from a live notebook, the session token carries the same issuer, the same client_id (onetd-<SITE>) and the same scope (openid profile email) that the device flow asks for. It is very likely the right token, and it needs no browser.

mcp='gateway' now uses it — after an explicit override and after MCP_TOKEN, before offering the device flow. It expires about two hours after issue, which is what generate_mcp_token() remains for.

Fixed - the device flow reported its own parsing, not the IdP's refusal

A refusal surfaced as a bare 'device_code' KeyError, because the payload was indexed directly. It now reports the HTTP status and the IdP's own error_description, and on a 400/401 points at IDP_CLIENT_ID and the device-code grant.

Changed - the IdP is read from the platform, not guessed

AI Studio publishes IDP_BASE_URL, IDP_CLIENT_ID, IDP_TOKEN_URL and TD_KEYCLOAK_BASE_URL (the device-authorization endpoint itself). All four are used, so discovery is a fallback rather than the first move.

This matters more than it sounds: which IdP a site uses is not derivable from its hostname. A site whose FQDN says preprod was observed pointing at the staging provider, while every hand-written snippet hardcodes production — so the device flow would have gone to an identity provider that has never heard of the client.

Fixed - describe_config reported a variable that had just been set

The pre-write check stages values, but non-pass-through variables are staged on the module while describe_config defaults to source='env'. A model chosen seconds earlier was reported NOT SET (required). It now reads source='module', where pass-through variables still come from os.environ — which staging also sets.

[0.3.1.190] - 2026-09-11

Fixed - the generator never actually asked anything in a notebook

_interactive() gated on sys.stdin.isatty(). A Jupyter kernel's stdin is not a terminal, yet input() and getpass() work there perfectly — they prompt in the notebook over the kernel's stdin channel. So on AI Studio, the one environment this generator exists for, every question was silently skipped and the file came out full of <set-this> with no prompt ever shown.

It now recognises a ZMQInteractiveShell as interactive, and _ask treats a prompt that cannot be displayed (ipykernel's StdinNotImplementedError, under nbconvert or papermill) as a missing answer rather than a crash.

Fixed - "no API key" was written as None, which cannot build a client

None, 'none' and '' all load as Python None, and the OpenAI SDK then refuses to construct a client at all — "The api_key client option must be set" — unless OPENAI_API_KEY happens to be exported. The instruct model failed before reaching the gateway, with an error naming a variable nobody set, and the report's hint pointed at Bedrock and GCP credentials.

The sentinel is now EMPTY, vLLM's own convention for a server that does not check keys, which survives _coerce as a real string. The preview shows it verbatim instead of set (hidden, 4 chars), which had concealed that no key was given. A new hint matches the missing-api-key error specifically, before the cloud-credentials one.

Fixed - the Enterprise MCP gateway wants a bearer token, not a password

0.3.1.188 wrote MCP_USERNAME / MCP_PASSWORD for the one-td gateway and called it "the verified combination". It is not: the gateway authenticates against the platform's identity provider, and database credentials come back as the database's own Error 8017, The UserId, Password or Account is invalid — which reads like a wrong password and is really the wrong kind of credential.

tdfs4ds.generate_mcp_token() performs the OAuth 2.0 Device Authorization Grant: it prints a URL and a short code, opens a browser, and polls until the token is issued. The client id is onetd-<SITE>, where <SITE> is JH_FQDN's first label upper-cased (tdfs4ds.mcp_site_id()); MCP_IDP_URLS carries the prod, staging and dev identity providers. It never raises — a failure leaves you free to paste a token instead.

mcp='gateway' now emits MCP_SERVER_HEADERS with that bearer, taking an existing MCP_TOKEN from the environment, else offering to fetch one, else reporting it as a gap. This token is not OAUTH_ACCESS_TOKEN: that one is the notebook session's, issued for a different audience.

Enterprise remains the default for AI Studio. The commented community line is the in-cluster alternative; the active block is now labelled ACTIVE and the alternative ALTERNATIVE (commented out), since that was read as the gateway itself being disabled.

Fixed - a LiteLLM web console reported as an intercepting page

The transport check probes /chat/completions then the base URL. LiteLLM's root serves its web console, so a healthy gateway was flagged INTERCEPTED: 200 text/html. An API answering a bare GET with 405/401/404 is the API, so the probe now stops there and never reaches the base URL.

[0.3.1.189] - 2026-09-11

Added - probe what the endpoint actually accepts: JSON constraints and TLS

Two configuration failures kept surfacing late, and both were invisible to check_configuration().

There are TWO structured-output systems, and only one was ever probed

Path Used by Mechanism Setting
langchain with_structured_output consumer agent, query optimizer json_schema → function_calling → json_mode STRUCTURED_OUTPUT_METHOD
provider-branched wire parameter documentation generator guided_json / response_format / format= INSTRUCT_MODEL_PROVIDER

The second was never probed, and it retries once unconstrained when the backend rejects its parameter. So a provider whose syntax the endpoint refuses left documentation and the business dictionary running on prompt-guided JSON — plausible output at strictly lower reliability — while every check stayed green.

Found while fixing it: the guided_json 422 hint could not fire for its own trigger. It was reachable only from a stage-1 instruct failure, and stage 1 is llm.invoke("Reply with exactly: …"), which never sends guided_json.

A new JSON constraints check probes both paths. The instruct check's second stage moves into it, so a row moves in the report and instruct_model goes back to meaning what its label says: can the model answer at all.

tdfs4ds.probe_json_constraints() (agent/json_constraints.py, also python -m tdfs4ds.agent.json_constraints) is a nine-request wire sweep: response_format json_schema with and without strict, tool calling, json_object, and the four vendor extensions (guided_json, nvext.guided_json, grammar, format). There is no single industry parameter for constrained decoding, so the accept/reject pattern identifies the serving stack behind the gateway — which no amount of configuration inspection can. It runs only once a cheaper probe has already reported trouble; nine billable completions on every healthy check would be absurd, and a test enforces it. Engine inference is fail-open: a gateway accepting several mutually exclusive vendor extensions is ignoring unknown fields, so acceptance proves nothing and it says "not identified" rather than guessing.

Live-verified against api.mistral.ai — which accepts strict, contradicting the assumption in _STRICT_JSON_SCHEMA_PROVIDERS. The engine label therefore describes the capability, never a vendor name.

A silent unconstrained fallback now fails the check. That is a deliberate behaviour change: a configuration that was quietly wrong reports as wrong, and the hint names INSTRUCT_MODEL_PROVIDER — not STRUCTURED_OUTPUT_METHOD, which that path does not use.

probe_structured_output also records what it learns (new remember_method()). It called with_structured_output directly, bypassing the seam, so running the diagnostic discovered the working method and the next real call still paid to rediscover it.

TLS is now reported, not merely configured

Nothing anywhere inspected a peer certificate. TLS appeared only as an echo of your own settings or as a substring match on an already-failed call — so an expiring certificate was invisible until the day every LLM call failed at once.

A new LLM transport (TLS) check runs before the instruct model and sends no credential, so its answer frames everything after it. It handshakes with INSTRUCT_MODEL_URL using the configured trust and reports issuer, subject, TLS version and days to expiry, separating three failures that look identical in a traceback and need three different fixes:

Reported Fix
untrusted issuer INSTRUCT_MODEL_CA_BUNDLE — and if the system store accepts the certificate, it says so, because then the bundle is simply the wrong one
hostname mismatch the host in INSTRUCT_MODEL_URL; the certificate's real name is reported
expired renewal on the server
under 30 days left nothing yet, which is the point of saying so now

With verification off it still reads the certificate (unvalidated) so you can see which CA to install and turn verification back on. Naming an untrusted issuer uses ssl._ssl._test_decode_cert — the only certificate parser in the standard library, private, wrapped, and never load-bearing. Verified against real valid, expired, hostname-mismatched, self-signed and DNS-failure endpoints.

It skips when a proxy is configured (a direct socket proves nothing about the proxied path and would just fail on blocked egress) and when no URL is set (Bedrock and Vertex secure their own transport).

It absorbs _redirect_probe, which previously ran only when an error had already matched the redirect hint. Two false positives had to be fixed once it ran always: a non-JSON body is interception only on a 200 (a 4xx HTML page is the server's own error page), and a redirect that merely normalises the URL (/v1 → /v1/, http → https) is not interception at all.

Instruct-only by design. _EMBEDDING_HINTS gained a certificate entry saying so and naming the real fixes, because a proxy fallback cannot weaken security while a verification fallback could.

Fixed - list_instruct_models ignored the TLS and proxy settings

Both it and list_embedding_models used a bare httpx.get against the same endpoint the chat client talks to — so behind an internal certificate authority they failed while the model itself worked, and INSTRUCT_MODEL_PROXY never applied to them. Both now go through the new embedding.http_get.

Added - patch_dotenv: change one variable without rewriting the file

export_config and generate_ai_studio_env rewrite a .env whole, losing its comments, its ordering and the commented-out alternatives a generated file deliberately carries. tdfs4ds.patch_dotenv(path, updates, *, comment=None, allow_secrets=False) changes one line.

It targets the last uncommented occurrence — the loader assigns as it reads, so on a duplicate the last one wins, and patching any other would leave the file saying one thing and meaning another — else the first commented slot (#TDFS4DS_X=, uncommented in place), else appends. It never enables a commented alternative while a live line exists, since a generated file carries two MCP_SERVER_URL lines on purpose.

The comment is a trailing note, so re-running replaces it instead of stacking one per run, and it is dropped for a quoted value (read back verbatim, so a trailing comment would become part of it). Credentials are refused without allow_secrets: a general-purpose file patcher should not be how a password reaches a file someone later commits. None is written as the literal None and loads as None.

check_configuration(..., pin=True) uses it to record what the JSON-constraints check discovers, stamped with the endpoint and date it was probed against — so a pin that stops being true is visible rather than silent. Note a pinned STRUCTURED_OUTPUT_METHOD travels into a deployed pod's config.yaml.

Changed - generate_ai_studio_env asks, validates, then writes

It used to write first and offer to check afterwards, filling anything unanswered with <set-this>. That placeholder does not work: it loads as the literal string and fails far from the file that caused it.

  • A blank answer is re-asked, with the reason the value is needed.
  • The instruct model is chosen from the list the endpoint itself serves, defaulting to a Ministral 14B when one is offered.
  • No API key is written as None — correct for a gateway that needs none, and honest, unlike a fake key that fails as a 401 elsewhere.
  • The configuration is staged into the session and checked before the write (describe_config() then check_configuration()), then the session is restored.
  • Gaps or failing checks block the write until confirmed; a non-interactive caller gets nothing written and the preview returned. force=True writes anyway.
  • A successful write ends with load_config(), so the session is configured when the call returns.

check now defaults to True, and pin=True records the probe's finding into the file just written. Two commented slots are emitted — INSTRUCT_MODEL_CA_BUNDLE and STRUCTURED_OUTPUT_METHOD — so turning either on is an edit.

The superseded root-level probe_json_mode.py scratch script is removed; the packaged module and its python -m entry point replace it.

[0.3.1.188] - 2026-09-11

Added - generate_ai_studio_env(): stop writing the AI Studio .env by hand

Configuring tdfs4ds on Teradata AI Studio went wrong quietly — a mistyped gateway URL or an unset variable surfaces much later as a model that "isn't configured" or a Vector Store that refuses a credential — and the only remedy was a hand-maintained template.

But the platform already publishes most of the answer. Mapping a known-working configuration against a real notebook's os.environ: 16 of 22 variables need no human input at all.

tdfs4ds.generate_ai_studio_env() and tdfs4ds.detect_ai_studio() (ai_studio.py) — 9 derived, 7 constant, 6 asked:

Target Source
TD_USERNAME, TD_DATABASE the …/teradata_username JWT claim inside OAUTH_ACCESS_TOKEN
INSTRUCT_MODEL_URL TD_LITELLM_BASE_URL
VECTOR_STORE_AUTH_BASE_URL TD_VECTORSTORE_BASE_URL, minus its /data-insights suffix
INSTRUCT_MODEL_VERIFY_SSL TD_SSL_VERIFY
MCP_SERVER_URL https://{JH_FQDN}/one-td/mcp
TD_HOST, VECTOR_STORE_BACKEND, TERADATA_AI_API_TYPE, … constant for AI Studio
the password, the API key, the instruct model asked — no platform source exists

And in password mode three of those six are the same database password, so it is asked once and fanned out to TD_PASSWORD / VECTOR_STORE_AUTH_PASSWORD / MCP_PASSWORD. That is also why _JWT_DROP is defeated in practice: one secret travelling under three names.

It writes real values and returns a redacted preview (set (hidden, N chars)), so the output is safe to leave in a saved notebook. This deliberately departs from export_config's return-the-text idiom, for the reason v0.3.1.187 had to clean up: returning that text is how a live password and API key reached a stored notebook output.

Quoting is a correctness requirement, not a style choice. _parse_dotenv handles quotes and trailing comments in an if/else, so a quoted value is verbatim while an unquoted abc #def loses everything from the whitespace. Every secret is therefore written quoted, with a single-quote fallback when the value contains " and a bare write plus a warning when it contains both. A round-trip test drives this with a #-bearing password (the real shape), a whitespace-then-# value, and one containing a quote.

auth='jwt' writes no password at all — the platform token serves the database (TD_LOGDATA), the Vector Store (VECTOR_STORE_AUTH_AUTH_TOKEN, which is tried before Basic) and MCP (Authorization: Bearer). Stated plainly in the docstring and in the generated file: the token expires about two hours after it was issued (exp - iat = 7200), so this trades a durable secret for a short-lived one rather than making the file non-secret. The variant not chosen is emitted commented out, so switching is an edit rather than a rerun.

mcp='gateway' (default) is https://{JH_FQDN}/one-td/mcp with edition=enterprise — the verified 132-tool server. mcp='incluster' uses TD_MCP_BASE_URL, whose host implies the Community edition, which embeds its own credentials so tdfs4ds sends none. The alternative is always written commented, because the edition decides whether credentials are sent at all and the platform's own TD_MCP_BASE_URL is not the server that was verified.

Anything neither derived nor answered becomes <set-this> plus a # MISSING: block rather than an exception — so prompt=False, a declined prompt and a TTY-less script are all the same, testable path. check=True reloads the file and runs describe_config() + check_configuration().

The unprefixed-variable rule is respected, not reversed. tdfs4ds still never reads OAUTH_ACCESS_TOKEN at runtime (the v0.3.1.154 decision). This is the tool that performs the explicit, one-time, user-invoked mapping the docs already described by hand, writing a named TDFS4DS_* variable so the resulting configuration states its own sources.

Reuses config._is_secret as the single authority on what counts as a credential — the per-item flag can only add caution, never remove it. 30 offline tests (tests/test_ai_studio.py): no DB, no network, no real secrets.

Fixed - TDFS4DS_TD_HOST_POD was never a registered variable

Introduced in v0.3.1.187's notebook pass. It is in neither _ALL_ENV_VARS nor _PASSTHROUGH_ENV_VARS, so _apply_dotenv logged unknown .env variable 'TDFS4DS_TD_HOST_POD' — ignored on every load. Notebook 06 still worked, because it reads the value with a plain os.getenv rather than through the config layer, but the warning was noise.

It should never have worn the prefix: it is not a tdfs4ds setting but a notebook-side staging value that becomes TDFS4DS_TD_HOST in the deployment's runtime_config. Renamed AGENTOPS_POD_TD_HOST. Registering it instead would have been wrong — it would claim a configuration slot the package does not have. A test now asserts every name the generator emits is a registered variable, so the class of bug cannot recur.

Changed - notebook 00 demonstrates the generator; off-platform behaviour documented

notebook dev/agentops/notebooks/00-Setup.ipynb went from 18 to 26 cells. Its section 1 was a template to copy and fill in; it now calls detect_ai_studio() and generate_ai_studio_env(), with the old template demoted to a reference for reading the generated file. New section 1b lays out the password-vs-token tradeoff. Section 4 pairs describe_config() ("is it set?") with check_configuration() ("does it work?").

Two pre-existing defects fixed there: %gen_token was in a raw cell, so it never executed, and an empty code cell was left over from the previous pass. Both notebook cells that touch the generator degrade with a message on an older tdfs4ds rather than raising.

Off AI Studio nothing changes. The new module is stdlib-only, imported eagerly like read_federation, and consulted by no existing code path. detect_ai_studio() returns an empty report rather than raising; the generated file labels its AI-Studio constants (TD_HOST=dbc, VECTOR_STORE_BACKEND=teradata, …) as assumptions, because they are wrong elsewhere; and the summary names export_config(fmt='env', source='module') as the right tool there. deploy_to_agentops off-platform already failed in its own client and fails no differently.

Documented - $VAR is not expanded in a .env

.env.example, README.md and docs_src/getting-started/connect.md recommend TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN=$OAUTH_ACCESS_TOKEN. tdfs4ds's own parser does no variable expansion, so that line is read as the literal string — it works only because python-dotenv's load_dotenv() (or a shell, or a Docker --env-file) interpolates before tdfs4ds sees the file. Anyone relying on tdfs4ds's auto-discovery gets the literal. Now stated in connect.md and in a new "Quoting, comments and $VAR" section in docs_src/configuration/dotenv.md, which also documents the trailing-comment and quoting rules. generate_ai_studio_env writes resolved values for exactly this reason.

[0.3.1.187] - 2026-09-10

Version 0.3.1.186 was published to PyPI before these changes were made, so this work ships as .187. A published version is immutable - reusing the number would leave two different builds answering to it, and the deployment publication gate compares the running version against the index by name alone.

Added - describe_config(): is this variable set?

export_config(include_secrets=False) renders a secret as a commented-out placeholder, which reads identically whether the variable holds a password or nothing at all — so it cannot answer the question you actually ask when a deployment misbehaves. describe_config() (config.py, re-exported on tdfs4ds) does:

TDFS4DS_TD_HOST                      dbc
TDFS4DS_TD_PASSWORD                  set (hidden, 16 chars)
TDFS4DS_MCP_PASSWORD                 NOT SET (required)

describe_config(*, source='env', required=None, display=True). A credential reports its length — enough to catch a truncated paste or a stray quote, never enough to reconstruct. An unset variable is listed only when named in required=, so the report stays readable against ~60 possible variables. source='env' reads os.environ (and therefore works despite the load-order trap); source='module' snapshots the running config, matching export_config's existing source= semantics.

Three contracts, each with a test: it reuses _is_secret rather than re-deriving the rule (a second "is this a credential?" authority is what goes stale and leaks); every hidden row returns value=None, so a caller who prints the return value cannot leak one either; and the output is pure ASCII, per the _print_report convention — a diagnostic must never be the thing that raises UnicodeEncodeError on a legacy Windows console.

Why it exists: the leak it prevents already happened. Printing the whole environment to confirm that .env had loaded is what put a live database password and a live LiteLLM API key into the stored output of a demo notebook.

Fixed - MCP_USERNAME and MCP_AUTH were committed into config.yaml in clear

config.yaml ships inside the deployment zip and is committed, so it carries only what is safe to read. classify() routed a setting to runtime_config on _is_secret(attr) or a TD_ / VECTOR_STORE_AUTH_ prefix — and two credential-adjacent MCP settings passed neither test:

  • MCP_USERNAME is half a credential pair. Its partner MCP_PASSWORD is a credential and already routed to runtime_config, and MCP_USERNAME falls back to TDFS4DS_TD_USERNAME (the Teradata MCP server authenticates with the database identity) — so the archive carried the database username in clear while the password travelled separately. TD_USERNAME is shielded only by its prefix, not by being secret, so the same identity was protected under one name and exposed under another.
  • MCP_AUTH is an auth-mode override: build_mcp_headers() resolves Basic from it before falling back to the edition, so a committed file could change how the pod authenticates.

Both now route to runtime_config via a new _DEPLOYMENT_VARS (agentops/config_map.py), the sibling of the existing _DEPLOYMENT_PREFIXES and sharing its rationale ("runtime even when not secret, because it identifies a deployment"). Deliberately not added to config._is_secret: that is the package-wide authority on "is this a credential?" and neither of these is one — "may this be committed?" is a different question, which is precisely why _DEPLOYMENT_PREFIXES exists.

Backward-compatible by construction. Under mcp=False — the default, and the documented first deployment — every MCP_* setting is dropped before this branch is reached, so an existing package is byte-identical (asserted). The change is observable only under mcp=True.

Documented - runtime_config is not a vault

Verified against a live AgentOps deployment UI, and correcting wording in fs-deploy-agentops, docs_src/deployment/agentops.md and CLAUDE.md that implied otherwise.

The UI's Config.json panel is runtime_config rendered back by the platform (plus platform-injected OPIK_*) — not the zip's config.yaml, none of whose settings appear there. And it displays the values in clear, readable by anyone who can view the deployment.

So runtime_config's real guarantee is "not committed to source control", which is weaker than confidential; package.py's "not on disk" is accurate about the zip and says nothing about the deployment record. The destination still matters — git history is permanent and widely readable, whereas a deployment record can be deleted and its credentials rotated — but that is durability and blast radius, not secrecy. Treat a deploy as publishing its credentials.

Two further findings recorded from the same deployment:

  • _JWT_DROP can be defeated in practice. Under auth='jwt', TD_PASSWORD is dropped because "a password shipped to the pod is never used". But VECTOR_STORE_AUTH_PASSWORD commonly holds the same database password, so the identical secret reaches the pod under another name and is displayed anyway.
  • auth='jwt' is a mixed auth model, not a per-caller one. The database connection is per-caller (TD_LOGDATA + a fingerprint-keyed reconnect) and MCP gets the caller's Bearer per turn, but the Vector Store authenticates with a fixed shared username/password from runtime_config — so vector search runs as one service identity no matter who calls. The caller's token could serve it (resolve_auth_mechanism tries JWT first, and VECTOR_STORE_AUTH_AUTH_TOKEN is merely unset), but the generated agent never wires it and the obstacle is a lifetime mismatch: Vector Store auth is configured inside connect(), which runs once per process, while a token is per-caller and expires. Notebook 06 ships a probe cell for whether the service accepts an AgentOps-issued token at all; wiring it is gated on that result.

Changed - the AgentOps demo notebooks

notebook dev/agentops/. Notebook 06 was 4 code cells with zero markdown and 16 stored outputs holding a live password, a live API key, three *.svc.cluster.local addresses, an internal hostname, a private IP and a database id. The outputs were also stale — cell 1's four lines cannot have produced the environment dump attached to them — so they were stripped rather than edited. Nothing had reached git.

Rebuilt as 26 cells (12 markdown) in the series' voice, with the environment read from .env instead of hardcoded. The substantive change is that deploy and verify are now two separate, re-runnable steps: report["ok"] is assigned before the smoke test precisely so a verification problem cannot read as a failed deploy, AgentOps deployment is asynchronous, and under auth='jwt' the pod's lazy bootstrap means its startup banner is only written on the first caller turn — so verified: False with "no startup confirmation" straight after a deploy is the expected reading, not a broken demo. The notebook now says so and calls check_deployed_agent in its own cell.

Notebook 00: the internal preprod MCP hostname is replaced with a placeholder; the !pip install <local wheel> that sat in a markdown cell became a code cell pointing at PyPI (a local wheel satisfies the notebooks and is invisible to the pod's image build, which contradicts notebook 06's publication gate); the .env template documents TDFS4DS_TD_HOST_POD and the TD_LOGMECH constraint, and no longer mislabels a username/password group as the PAT mechanism.

[0.3.1.186] - 2026-09-10

Published to PyPI from an uncommitted working tree, so there is no commit carrying this version and no record here of what changed in it. The entry exists so the public version sequence has no silent gap: anything looking for 0.3.1.186 in the history will not find it, and that is the fact, not an oversight in the search.

[0.3.1.185] - 2026-09-01

Documented - RESOLVED: an AgentOps pod has no DNS for the database; use the IP address

The live deployment now works end to end. The blocker was never the credential, the auth mode or MCP: TDFS4DS_TD_HOST='dbc' does not resolve from inside an AgentOps pod, and neither did any qualified form - DNS for the database was unavailable there entirely. Setting TDFS4DS_TD_HOST to the database's IP address was the fix.

The v0.3.1.183/.184 diagnostics offered only the fully-qualified in-cluster name, which this proves insufficient. Both the host_unresolved insight and the generated agent's own diagnosis now name the IP address as the remedy when no name resolves, and both note the cost: an IP pins the deployment to an address that can change.

New Connecting to Teradata from a pod section in the docs and the fs-deploy-agentops skill, recording what the episode actually taught:

  • The notebook is not evidence for the pod. A bare name is completed by the resolver's search domains and the two workloads need not share them - so the same value succeeds in one and fails in the other, with nothing in the error pointing at a name. That the other in-cluster services are all reached by fully-qualified name is what made a bare dbc look normal.
  • Classify a TDML_2000 by elapsed time, since the message is identical for three faults: well under a second = the name never resolved (observed: 88 ms); seconds then failure = resolved but unreachable, a platform question; a round trip plus SQLSTATE 28000 = reached and refused, the credential.
  • A connection failure is never evidence about the credential. The identical failure appeared under auth='service' (TD2) and auth='jwt'. Switching auth mode to test connectivity spends a deploy cycle learning nothing.
  • Runtime: X | Mode: N/A is a consequence, not a second fault - under service the connection happens at import, so a host failure is fatal and the runtime never constructs; the pod serves /health while the agent does not exist, which reads as unhealthy. Under jwt the same fault appears per request, because that path connects lazily.
  • JWT Auth: ... | Authorization: disabled is inbound auth to the agent's HTTP endpoint and says nothing about the Teradata connection.
  • AgentOps does support a direct database connection. Nothing was blocked; there was no name to connect to. MCP was never a required data path, and could not be one - every non-MCP skill reads the catalog over the ordinary teradataml connection.

[0.3.1.184] - 2026-09-01

Added - the generated agent diagnoses its own connectivity failure

TDML_2000 Failed to connect to Teradata Vantage. renders identically for an unresolved name, an unreachable host and a rejected credential - three faults with three different fixes. A pod is not a shell you can poke at, so each hypothesis costs a full redeploy: on a live deployment, host=dbc failing in 88 ms took several cycles to attribute.

Both generated variants now wrap reconnect_teradata and, on the failure path only, log a diagnosis before re-raising:

  • resolve the configured host, and report what it resolved to or the resolver error verbatim;
  • if it resolved, TCP-probe the port - reachable there means the name and route are fine and the fault is the credential;
  • if it did not, print the resolver's own search domains from /etc/resolv.conf and what a bare name becomes under each. A bare name like dbc is completed by those domains, and an AgentOps pod need not share the notebook's namespace - so this prints the fully-qualified name that works instead of leaving it to be guessed.

Costs nothing on the happy path, uses only the standard library, and never raises: it runs inside an error handler, where raising would replace the real failure with its own.

Five tests, including an AST check that the handler both diagnoses and re-raises, and that a name which cannot resolve is never TCP-probed.

[0.3.1.183] - 2026-09-01

Fixed - reconnect_teradata logged the wrapper and threw the cause away

logger.error(..., str(exc).splitlines()[0]) kept the first line only. teradataml puts (TDML_2000) Failed to connect to Teradata Vantage. there and wraps the driver error rather than replacing it - so the actionable half lives in __cause__. A DNS miss, a refused connection and a rejected logon all render identically at the top level, and they have three different fixes.

Found on a live AgentOps pod log: reconnecting to Teradata host=dbc user=DM5344095 at 07:53:22,903, failed at 07:53:22,990. 87 milliseconds - far too fast for a TCP timeout, so a name that never resolved - and nothing in the log said so. _describe_exception now walks the __cause__/__context__ chain (bounded, cycle-safe, tolerant of a __str__ that itself raises, since it runs inside an error handler).

Third instance of this pattern after v0.3.1.176 and v0.3.1.179: do not truncate an error to its first line or a fixed short width - the half that names the fault is routinely the later half.

Added - host_unresolved, ahead of the generic connect_failed

Now reachable, because the chained gaierror finally reaches the log. A bare name like dbc resolves through the namespace's search domains, and an AgentOps pod need not share the notebook's namespace - so a host that works in the notebook can fail in the pod. The insight says to use the fully-qualified in-cluster form the other services already use (<service>.<namespace>.svc.cluster.local), and gives the timing tell: sub-second means unresolved, a TCP timeout means unreachable, SQLSTATE 28000 means refused.

Ordered before connect_failed so a specific fault is not reported as "could be the host, could be the token".

[0.3.1.182] - 2026-09-01

Fixed - auth='TD2' raised without saying which argument the user wanted

auth= names whose credential the pod uses ('service' = its own TDFS4DS_TD_*, 'jwt' = the caller's, per request). TD2 / LDAP / JWT name the Teradata logon mechanism, which travels separately in TDFS4DS_TD_LOGMECH. Two orthogonal axes whose vocabularies overlap on JWT - so a user wanting a username/password pod reasonably reaches for auth='TD2' and got only a list of the legal values, which does not say which one they want.

The error now distinguishes the axes and, for a recognised mechanism, names the fix: use auth='service' and set TDFS4DS_TD_LOGMECH=TD2. TD2 was always supported; it never had a name a user would guess.

Added - service identity with a leftover TD_LOGMECH=JWT is refused at preflight

The mirror of the v0.3.1.175 bug, in the other direction. Under auth='service' the session's TD_LOGMECH is passed through, so a JWT left over from experimenting with auth='jwt' ships a pod calling create_context(username=..., password=..., logmech='JWT') - incoherent, and it fails at startup with a message about credentials rather than about the mechanism. The connection_env check now fails when TD_LOGMECH=JWT accompanies a username and password with no TD_LOGDATA, and names both ways out. auth='jwt' is untouched: it sets the mechanism itself.

The OK detail also reports the resolved mechanism (service identity, logmech=TD2), so what the pod will actually do is visible in the report rather than inferred.

Eight tests, including that an unrecognised value still explains both modes without guessing a mechanism.

[0.3.1.181] - 2026-09-01

Changed - ok is the deployment's verdict; verified is the smoke test's

report["ok"] = bool(smoke["ok"]) made the two one flag, so a live run whose pod was active with a working URL reported ok=False because a smoke check that depends on pod warmth had failed - sending the user to debug a deployment that was fine.

They are now separate, and the failing-verify hint says so explicitly ("The DEPLOYMENT is active - this is the agent's behaviour, not the deploy"):

  • ok - the deployment reached active and has a URL. Set before the smoke test runs, so a verification problem can never read as a failed deploy.
  • verified - True/False, or None when it did not run. A tri-state, because "not checked" and "checked and failed" are different things.

Added - wait for the pod's startup block instead of reading the log a second after active

active is the platform's verdict on the pod, which it reaches before tdfs4ds has finished its module-scope work (connect(), the configuration check, the graph build). The pipeline read the log about a second later, found no startup block, and said no startup confirmation in the log on a healthy deployment - and that line sat next to a real failure, so it read as corroboration.

_wait_for_startup polls the log (5 s) up to startup_timeout=120 seconds, ending early as soon as the startup block appears or an error signature does - an error is a conclusion, and waiting past it delays the diagnosis by the whole timeout for nothing. Bounded and fail-open: a log that never confirms anything still returns whatever it held.

Added - verify=False, for deferring verification

Returns as soon as the deployment is active, and the skipped stage names the follow-up: check_deployed_agent(report['agent_url'], token=...). For a pod that is slow to warm, or when the caller holding a usable token is not the one running the deploy.

Six tests: the wait returns on arrival / on an error / at the timeout / not at all when there is nothing to wait for, an active deployment with a failing smoke test reports ok=True with verified=False, and verify=False never calls the smoke test.

[0.3.1.180] - 2026-08-31

Added - connect_failed: a session that never landed is not a refused credential

A live JWT deployment reached the Playground, loaded, answered - and returned [Teradata][teradataml](TDML_2000) Failed to connect to Teradata Vantage. No signature matched it, so analyze_pod_log reported nothing and the run said only "no startup confirmation in the log".

It sat in the gap between the two existing signatures: no_credentials (nothing was supplied) and token_rejected (supplied and refused). A bare TDML_2000 is neither - nothing was refused, because no session was ever opened - and the fix is the opposite one. The new signature names both candidates:

  • TDFS4DS_TD_HOST. The notebook and the pod are different workloads and need not resolve the same name, so a host copied from a working notebook is not evidence it works in a pod.
  • Under auth='jwt', whether the platform's auth_context.token is usable as a Vantage JWT at all. A token minted for the AgentOps API is not necessarily accepted by the database, and a pod cannot mint one of its own - so where it is not accepted, auth='jwt' is not deployable in that environment and auth='service' with a real account is the way in.

Ordered after token_rejected, because teradataml wraps a refused logon in TDML_2000 too: when both markers appear the specific one leads, and the wrapper follows. A test pins that order, and pins that a bare TDML_2000 yields connect_failed alone.

It advised TDFS4DS_STRUCTURED_OUTPUT_METHOD=json_mode as "the robust choice". Measured directly against a LiteLLM/Bedrock route: response_format is inert there in every shape - json_object included - so all five variants returned identical prose and json_mode fails exactly as json_schema does. tools succeeded on every attempt. The insight now says function_calling, and says why json_mode is not a safe default.

[0.3.1.179] - 2026-08-31

Fixed - get_logs relayed a clipped 422 instead of naming the mistake

client.get_logs() takes the deployment's UUID, but the generated deployment name ends with a uuid4 of its own (tdfs4ds-consumer-agent-<uuid>) - and the name is what the AI Studio Playground displays, so passing it is the obvious mistake to make. The platform answers 422 ... "field": "path -> agent_id", "message": "Input should be a valid UUID", whose useful half sits after a long echoed path and fell off the 300-character truncation, leaving Input should be a valid and no fix.

get_logs now checks the shape first and says what is wrong: this route takes the UUID, that looks like the name, use report['agent_id']. The fallback truncation was also raised to 800 characters, for the same reason it was raised in the structured-output probe in v0.3.1.176 - a 4xx names the offending field late in the sentence.

Fetching a log is a diagnostic step taken when something has already gone wrong; a diagnostic that fails obscurely costs a round trip at exactly the wrong moment.

[0.3.1.178] - 2026-08-31

Fixed - a failed smoke run showed a green row and buried its own finding

All three seen together in one live run against a deployment whose pod could not reach Teradata: reachable failed, and the report still reported latency ok and a second failure from grounded.

  • latency counted a failed call as a measurement. A 404 returning in 100 ms is not a fast agent. Only a HTTP 200 is timed now, so with nothing answering the check reports skipped rather than a green row underneath a failed run.
  • grounded gated on non-empty text instead of the status. A failed call leaves the error body in first_answer - non-empty - so the check ran, found no catalog name in a 404 page and raised a derived failure over the real one. It now gates on first_status == 200, as answers / session_memory / session_isolation already did.
  • reachable did not wait for the gateway route. active is the platform's verdict on the pod, and the pipeline calls /invoke about a second later - so a missing route reads as a dead agent. It now retries a bounded 6 attempts / ~25 s, but only on 404/502/503/504. An expired JWT (401/403) or an app-level 500 still fails on the first call: retrying those turns a clear answer into a wait.

Same anti-pattern as the v0.3.1.164/.167 Vector Store findings - ask what a check would still report with the failure present.

[0.3.1.177] - 2026-08-31

Fixed - a dry run stopped without saying so

deploy_to_agentops promises a numbered [k/7] line per stage, but _skip_rest appended the skipped stages to the report without emitting them. So a perfectly normal dry_run=True run printed four lines and fell silent - which reads exactly like hanging on the next HTTP call, the first thing the pipeline does after packaging. Observed live: stages 1-4 green, "and then nothing more".

Skipped stages now go through the same _stage emitter as every other outcome, so the log always accounts for all seven and states the reason (dry run - nothing sent, definition not registered, ...). This also covers the failure paths, where a FAILED line was previously followed by the same silence.

The regression test asserts one emitted line per stage and reproduces the old symptom exactly - four lines, then nothing.

[0.3.1.176] - 2026-08-31

Fixed - the structured-output probe failed an endpoint over its own class name

check_configuration() reported "structured output FAILED every way" against a LiteLLM gateway where function_calling in fact works perfectly, failing the whole configuration report and the AgentOps preflight with it.

The cause was in the diagnostic, not the endpoint. On the function_calling path the Pydantic schema's class name is sent as the tool name, and probe_structured_output declared its schema _Probe - a leading underscore, which a Bedrock-backed route refuses:

litellm.BadRequestError: BedrockException - {"message":"The tool _Probe at tool ...

Renamed to Probe. No real call site was affected - all six schemas already in use are letter-initial (IntentResult, ExplainOutput, CandidatesOutput, PlanComparison, StrategiesOutput, CandidateRewrite), and the nested _SubIntent / _ClarificationInterpretation travel in $defs, never as a tool name. So the agent and the query optimizer were working on this gateway throughout; only the report said otherwise.

Two things made this expensive to find, both fixed:

  • The error was truncated to 120 characters, clipping the body mid-phrase at exactly the word that named the cause. The probe's per-method detail now keeps 300 characters (400 for the default path). A gateway's 4xx names the offending field late in the sentence, and that string is the entire diagnosis a user gets.
  • It read as a capability failure. A new _INSTRUCT_HINTS entry recognises a tool-name rejection and says what it is: the schema class name, not the model's ability to produce typed output.

Two tests guard the class of bug rather than the instance: the probe's schema name must match ^[A-Za-z][A-Za-z0-9_]*$, and a source scan rejects an underscore-prefixed schema at any _structured(...) call site.

Live-diagnosed by probing the gateway directly: response_format is inert there in all four shapes (json_object, json_schema, +strict, +seed) - all five requests returned identical prose, so json_mode cannot work either and no client-side parser can rescue that path. tools succeeded on every attempt. seed was ruled out (seed + json_schema and seed + tools both behave exactly as without it), which retires the v0.3.1.145 provider='mistralai' workaround as a hypothesis for this failure.

[0.3.1.175] - 2026-08-25

Fixed - a JWT deployment shipped a conflicting logon mechanism and an unused password

Found driving a live deployment from a session configured for TD2 user/password auth.

runtime_config carried TDFS4DS_TD_LOGMECH=TD2 straight from the session. The generated JWT agent sets its own default with os.environ.setdefault(...), which does not override a value already present - so the pod would call create_context(logmech='TD2', logdata='token=...'), which is incoherent, and fail at startup with an error about credentials rather than about the mechanism. Under auth='jwt' the logon mechanism is now normalised to JWT in runtime_config; auth='service' keeps whatever the session configured.

The same deployment also shipped TDFS4DS_TD_PASSWORD to a pod that authenticates with the caller's token and never reads it - pure exposure. It is now dropped under auth='jwt'. TD_USERNAME is kept: some sites scope a token by user, and reconnect passes it through when present.

classify() gained an auth= argument for both rules. Five tests cover the pair, asserting the service path is unchanged.

[0.3.1.174] - 2026-08-25

Added - deploy_to_agentops: one call from a named store to a verified agent

The packaging layer stopped at a zip; everything after it was hand-run REST calls, and each mistake cost a full image build to discover. Driving one real deployment produced, in order: a payload field the API rejects (422 extra_forbidden), a missing docker_image, an empty runtime_config that shipped a pod with no configuration at all, and a version that cannot pip install inside the image.

tdfs4ds.deploy_to_agentops(dest, database=..., token=..., auth=..., mcp=False) (agentops/deploy.py) runs seven stages - validate target, preflight, publication gate, package+zip, register, deploy, verify - and a failing stage stops the pipeline, because registering a definition whose preflight failed only creates registry litter and hides the earlier cause. dry_run=True stops after the zip and returns the payload.

The feature store is now stated, never inherited. database= is required; database_view, databases_readonly and data_domain are explicit too. Previously the pod silently picked up whatever the session had configured - invisible at the call site and impossible to review. Model, embedding and Vector Store settings still come from the running session (you proved those work by asking the agent questions). Applied as overrides into split_config, never by mutating module globals: a deploy must not leave the caller's session pointing elsewhere, and a test asserts it.

The target is validated against a live connection before anything is sent: catalog views readable, business dictionary non-empty (an undocumented store deploys an agent that cannot answer anything, so it is refused), and each read-only store checked with the federation attach probe _validate_store - reused, not reimplemented.

Publication gate. The running version must be on PyPI, or a RuntimeError explains that the image build runs pip install -r requirements.txt, that a local or editable install is invisible to it, and the three ways out. require_published=False waives it for a private mirror; an unreachable index stays fail-open.

analyze_pod_log (agentops/log_insights.py) turns a pod log into findings, seeded with signatures actually observed: pip_unresolvable, agent_runtime_missing, no_credentials, token_rejected, schema_missing, structured_output, vector_store_auth, oom, and the positive startup_ok. The positive one is not decoration - "no known failure matched" and "the agent started correctly" are different conclusions, and only the second is evidence; without it a truncated log and a healthy one read the same. Logs are analysed whether or not the deploy succeeded.

MCP is off by default and Enterprise-only when on. mcp_enabled=False omits exactly two intents (DATA_QUERY, ALGORITHM); everything else works, so it is the right first deployment. When on, the generated agent sets MCP_SERVER_HEADERS to the caller's Bearer token inside the turn lock - build_mcp_headers resolves from module config only, and neither a static header nor Basic-from-a- password can carry a per-caller token in a pod. No change to mcp_connection.py; mutating a module global under the lock is the pattern _with_domain already uses. MCP settings are dropped from config.yaml when MCP is off, so an archive never announces a server edition and tool cap it does not use - nor carries an MCP credential it has no reason to.

agent_runtime no longer comes from a demo folder. AgentOps ships its runtime SDK as loose files inside the tutorial directory and it is on no package index - the PyPI names agentops (an agentops.ai observability SDK) and agent_runtime (a Microsoft AutoGen placeholder at 0.0.1) are unrelated projects, and installing either gives a green build followed by an ImportError at startup. Copying from Demo/ made an unversioned tutorial file an invisible input to every deployment. The contract is two symbols and ~60 lines with no server or framework, so tdfs4ds now emits its own implementation of the same interface when the destination has none (agent_runtime='auto', the default). A platform-supplied copy always wins and is never overwritten - its host may rely on details beyond the public surface. Both paths are fingerprinted so "did the runtime change under me?" is answerable. agent_runtime='generate' forces ours; 'require' refuses to write. Twelve behavioural tests exercise the generated runtime through the real contract (is_async for async and sync entrypoints, unchanged registration, double-register refusal, AuthContext defaults/frozenness/get_auth_header/roles).

Fixed

  • The entrypoint was registered wrapped. AgentRuntimeApp.is_async is inspect.iscoroutinefunction(self._entrypoint), and decorators apply bottom-up, so @app.entrypoint over @opik.track registered the opik wrapper - if that returns a plain function the platform drives an async agent synchronously. The entrypoint is now a bare coroutine with @app.entrypoint as its only decorator, enforced by a test; tracing is left to the platform collector, since a span is not worth risking the runtime contract.
  • The generated agent now uses AuthContext.get_auth_header() for MCP rather than formatting the Bearer by hand - the SDK's own docstring names the MCP server as its use case, which independently confirms the per-caller design.
  • deployment_payload sent _generated_at, which the API rejects outright (extra_forbidden), and omitted docker_image, which the platform needs. Both covered by a test.
  • check_agent_deployment now reports the resolved store (SCHEMA / SCHEMA_VIEW / DATA_DOMAIN), so "which database will the pod read?" is answered before deploying.

[0.3.1.172] - 2026-08-25

Added - JWT / per-caller identity for the AgentOps package, and token logon in reconnect

Found deploying to AI Studio: the environment authenticates users with a JWT (create_context(host='dbc', logmech='JWT', logdata=f'token={user_token}')) and has no username/password to give a service account. The v0.3.1.171 package assumed service identity, so runtime_config came out empty and the pod deployed with no tdfs4ds configuration at all.

_connect_kwargs accepts a token logon (utils/reconnect.py): TDFS4DS_TD_HOST + TDFS4DS_TD_LOGDATA (+ LOGMECH, defaulting to JWT) alongside the historical host/username/password. Without LOGDATA the kwargs are byte-identical to before, pinned by tests written as literals rather than as a comparison against the implementation. This was not cosmetic: every automatic reconnect funnels through this function, so a JWT deployment could previously open a context by hand but never recover one.

LOGDATA is a credential - 'LOGDATA' joins _SECRET_SUFFIXES, so export_config redacts it and the AgentOps split routes it to runtime_config, never into the committed config.yaml.

auth='jwt' generates a different agent (generate_agentops_package(auth=...), write_agentops_package(auth=...), check_agent_deployment(auth=...)). Three things follow from the token model, none optional: there is no credential at import time (tokens arrive per request), so the bootstrap becomes lazy on first invocation; the token expires, so the rebuild is keyed on the token rather than done once; and identity is per caller, so the context is rebuilt when the caller changes. That last one is only safe because turns are already serialized - teradataml keeps ONE process-wide context, so interleaved turns would run under the wrong identity. connect() still runs once per process: catalog views belong to the store, not the caller. Only a fingerprint of the token is kept, never the token. auth='service' (the default) is unchanged and produces byte-identical files.

check_agent_deployment(auth='jwt') requires only TDFS4DS_TD_HOST - demanding a password there would fail a correct JWT deployment.

Fixed - deployment_payload sent fields the API rejects

Live POST /agents returned 422 extra_forbidden on _generated_at, a generation timestamp the payload carried. Removed; the payload now contains only API-defined fields. It was also missing docker_image, which the platform needs to know the base image. Both are covered by a test asserting no _-prefixed key survives and that a base image is set.

[0.3.1.171] - 2026-08-24

Added - package the consumer agent for Teradata AI Studio (AgentOps)

tdfs4ds.agentops turns a working local session into a deployable AgentOps package, and then verifies the deployed result. Five new entry points, all re-exported on tdfs4ds:

tdfs4ds.check_agent_deployment(dest='./teradata_agent')   # before: is it deployable?
print(tdfs4ds.generate_agentops_package())                # inspect
result = tdfs4ds.write_agentops_package('./teradata_agent')  # files + zip + runtime_config
payload = tdfs4ds.deployment_payload(defn_id, result['runtime_config'])
tdfs4ds.check_deployed_agent(agent_url, token=user_token) # after: does it work?

The package is derived from the running session, never hand-written. By the time a user runs this they have already proved their configuration by asking the agent questions; the split between what is safe to commit (config.yaml) and what must be injected at deploy time (runtime_config) is computed from config.py's existing credential classification (_is_secret, _CREDENTIAL_VARS, _PASSTHROUGH_ENV_VARS) rather than a second copy of it -- a duplicated "is this a credential?" is what goes stale and leaks the thing it was written to protect. Connection variables (TD_*, VECTOR_STORE_AUTH_*) are routed to runtime_config even when not secret, because they identify a deployment: committing TD_HOST is how a package built against dev keeps pointing at dev. SERVE_* is dropped (the platform is the server), as is CHROMA_* on the Teradata backend.

Four structural mismatches with the platform, encoded in the generated code -- each invisible until it fails in a pod. (1) The entrypoint is async and consumer_agent() is synchronous and blocks on Teradata + LLM I/O, so it runs in asyncio.to_thread. (2) The platform calls its factory per invocation, but connect() and the Vector Store handshake must happen once, so setup is at module scope and the factory is a thin wrapper. (3) graph._with_domain mutates the module-global DATA_DOMAIN and restores it, so two concurrent turns in one process clobber each other -- an asyncio.Lock serializes turns, and throughput scales with pods. (4) The env mapping is emitted before import tdfs4ds, because config.py populates module attributes exactly once at import (the v0.3.1.155 trap); a fallback assigned afterwards is silently ignored.

thread_id never falls back to a constant. session_id if the caller sent one, else the caller's identity, else a fresh uuid per call. A constant default would put every caller in one shared conversation -- answers leaking between users, which reads as a working memory feature from the outside. check_deployed_agent's session_isolation check asserts a fresh session sees no trace of the previous one, precisely because a "does it respond?" test passes this bug happily.

grounded is the check only tdfs4ds can make. A pod whose Teradata connection failed still answers -- fluently, and about nothing -- so the smoke test asserts a real name from _known_catalog_names() appears in the response, distinguishing "the LLM replied" from "the agent replied from the feature store".

pip_resolvable queries the index. A package built from an unreleased working tree installs perfectly in the notebook and fails in the image build minutes later; the check reports it up front. PyPI is treated as a default rather than an authority -- an unreachable index is fail-open, and a private mirror can overrule the finding.

Documented limits, not defects (they follow from leaving agent/graph.py untouched): replicas: 1 is mandatory, enable_memory: False, session memory is lost on pod restart, and Opik tracing is one span per turn. Lifting the first three needs a checkpointer= passthrough; the fourth needs callbacks=.

Also: the generated artifacts are pure ASCII, following scheduling/script.py -- they are written and re-read across unknown default encodings. agent_runtime/ is the platform's and is never generated, only preserved into the zip, which (unlike the platform tutorial's own zip step, whose os.walk applies no filter despite the comment) really does exclude __pycache__/.pyc.

New skill fs-deploy-agentops, new docs page Deployment -> Teradata AI Studio (AgentOps), 47 offline tests in tests/test_agentops_package.py (no DB, no LLM, no network).

Changed

  • diagnostics._format_report / _print_report take an optional title=, defaulting to the existing string, so the two new reports render through the one formatter instead of growing a second copy of it. No behaviour change for existing callers.

[0.3.1.170] - 2026-07-31

Fixed — the consumer agent crashed on its first step behind a relaying gateway

Every typed call died with::

ValidationError: 1 validation error for IntentResult
  Invalid JSON: expected value at line 1 column 1
  [input_value='```json\n{\n  "intent": ...']

llm.with_structured_output(schema) defaults, in langchain_openai, to method="json_schema" — OpenAI's Structured Outputs API. Its contract is that the server guarantees a schema-valid reply via constrained decoding, so the SDK parses the content verbatim (model_validate_json), with no tolerance for anything around it.

A gateway can accept response_format without honouring it. LiteLLM in front of Bedrock degrades it to an instruction, and the model answers with markdown-fenced JSON. Column 1 is a backtick.

It is the gateway, not the model. This was first diagnosed per-model, on a report that Mistral worked where Anthropic failed. The raw content disproved it — both come back fenced from the same endpoint:

anthropic : '```json\n{\n  "intent": ...t with high confidence.'
ministral : '```json\n{\n  "intent": ...ull\n    }\n  ]\n}\n```'

A model-name heuristic would have fixed one deployment and missed the next. Nothing in the configuration states whether an endpoint enforces the schema — the only way to know is to ask it.

New agent/structured_output.py is the single seam all nine call sites now go through (one classifier, one EXPLAIN, seven in the query optimizer — all of which failed identically, so optimize_query=True was equally affected). It attempts the default path unchanged, then falls back through two alternatives that are deliberately not interchangeable — there is no single industry parameter for constrained decoding, and they fail for different reasons:

method request parsed fails when
(default) response_format={"type":"json_schema"} server-side, verbatim the gateway relays without enforcing
function_calling tools + tool_choice tool-call arguments the route has no tool calling
json_mode response_format={"type":"json_object"} client-side — JsonOutputParser, which strips markdown fences (verified) rarely; but no schema reaches the model, so field adherence rests on the prompt

json_mode is last because it is the weakest guarantee, yet it is the most robust mechanism: parsing happens in langchain, so it survives an endpoint that enforces nothing at all — precisely the failure being handled. That is why both fallbacks exist rather than just the first.

Backward compatible in the strong sense: the fallback is reachable only after a call that already raised, so a working deployment issues a byte-identical request — no method= at all, not even an explicit "json_schema" — and never reaches it. A blanket switch to function_calling was rejected: it changes the wire payload for every non-OpenAI deployment, and payload shape is demonstrably gateway-sensitive here (v0.3.1.145 moved openai-compatible onto response_format because vLLM-style guided_json was 422ing from a LiteLLM gateway).

The working method is remembered per process, so the extra round trip is paid once rather than on every classification. tdfs4ds.STRUCTURED_OUTPUT_METHOD (env TDFS4DS_STRUCTURED_OUTPUT_METHOD) pins it and skips the probe entirely.

include_raw=True was investigated as a gentler fix and does not work: in BaseChatOpenAI.with_structured_output the response format is bound to the model and the wrapper is RunnableMap(raw=llm) | parser_with_fallback, so the LLM runs first and the strict parse raises inside _generate, before the fallback parser is reached.

Added — check_configuration now proves structured output, not just that the model replies

The instruct check gained a second stage, the same split as the MCP check (listing tools proves nothing about calling them): after the hello-world it asks for a small typed answer and reports whether the schema was actually enforced. A plain reply stays green on exactly the endpoint where the agent's first step dies, so the two are separate claims.

An endpoint that relays without enforcing is reported as ok with a hint — tdfs4ds handles it — naming STRUCTURED_OUTPUT_METHOD and the method that actually worked here, so the wasted first attempt can be avoided. The probe tries every fallback rather than stopping at the first success: which mechanisms an endpoint supports is what tells you what to pin. Only an endpoint where no path works is a failure. A model object exposing no with_structured_output is not probed rather than failed: that is client plumbing, not an endpoint capability.

probe_json_mode.py (repo root, not packaged) is the standalone version for handing to a platform team. Beyond the three langchain methods it also tries the parameter names the same idea goes by in other stacks — guided_json (vLLM/SGLang), nvext.guided_json (NVIDIA NIM), grammar (HF TGI), format (Ollama) — because a gateway generally accepts only its own, so the accept/reject pattern identifies the engine behind it. Verified against api.mistral.ai: json_schema enforced, json_object accepted, all four others rejected with 422.

[0.3.1.169] - 2026-07-31

Fixed — refreshing the vector index used a call that could never succeed

build_teradata_index's reuse path called only add_datasets(), which means attach a NEW source object. tdfs4ds always names the same FS_V_BUSINESS_DICTIONARY_* view — the documentation tables never change identity, only their rows do — so that call was guaranteed to be refused, on every second and later build, with 400 "You can only add object names that do not exist in the collection". Newly documented rows therefore never reached the index. 0.3.1.167 fixed the reporting of that refusal; this fixes the cause.

The reuse path is now update(data_refresh=True) first, destroy + recreate as the fallback (_resync_attached_collection):

  • update(data_refresh=True) is documented as "refresh/regenerate vectors … using existing backing/source tables" — exactly this situation. Cheap, no admin rights, collection preserved.
  • If it does not apply, the collection is destroyed and recreated from the same source. That needs admin rights and costs a full re-embed, which is why it is second — but it always picks the rows up, so an older service degrades instead of breaking.

add_datasets is no longer tried here at all: it cannot succeed against an always-identical source by construction, and it is not free — ~50 s per store observed live before the 400 comes back. It remains right for a genuinely new source, so its classifier (_is_source_already_attached_error) is kept for collections whose source was attached outside tdfs4ds.

0.3.1.140 recorded data_refresh as unreliable. That finding stands for the service version it was made on and was never evidence about newer ones; a 2.2.17 service is reported to update incrementally without trouble, and the fallback covers the case where it does not.

Whether unchanged rows get re-embedded is the service's business. Verified against the installed teradatagenai, a hand-rolled per-row incremental is not safely expressible: get_ids() raises NotImplementedError ("Currently this is not being supported by service"), and delete_by_ids keys on TD_ID, which _inputdata recomputes as ROW_NUMBER() OVER (ORDER BY <text>) — a position, not an identity, so a surgical delete can leave _index's frozen embeddings aligned to different rows (the 0.3.1.140 corruption). data_refresh is assumed to handle it, as the best mechanism the service exposes.

When neither mechanism applies the collection is left intact and searchable, and the warning names both failures — the second one alone reads as a rights problem and hides that the incremental path was refused first.

[0.3.1.168] - 2026-07-31

Added — selftest_vector_store(): prove the Vector Store actually works, and emit a support reproducer when it does not

Every Vector Store check up to now is read-only — service up, credential accepted, version supported. None of that exercises what tdfs4ds needs: embedding rows and getting them back. And every Vector Store failure this package has hit in the field is invisible to a read-only probe: the ephemeral _inputdata view (0.3.1.140), update_failed collections written by a call that raised nothing (0.3.1.139), add_datasets refusing an attached source (0.3.1.167). All three only appear once something writes.

New vector_store_selftest.py, re-exported as tdfs4ds.selftest_vector_store(), walks the whole lifecycle on a throwaway collection over a throwaway table and records each step independently:

The step list is an audit of the API calls tdfs4ds actually makes — there are exactly seven distinct ones in vector_store_teradata.py, and each gets a step:

# step call exercised
1 service_version health() — version vs the supported minimum
2 create_source — (plain SQL)
3 create_collection from_datasets() + list(), and the collection really reaching ready
4 collection_details get_details() — the readiness probe every write waits on
5 search similarity_search()
6 parse_hits tdfs4ds's own _parse_one
7 update_source — (one sentence rewritten, coffee → cycling)
8 refresh_index update(data_refresh=True) and add_datasets()
9 search_after_update similarity_search()
10 major_finalize update(update_style='MAJOR')
11-12 cleanup_* destroy() (admin-only) and DROP TABLE

Step 8 is the point. Resyncing an already-attached source is the one operation with no reliably working mechanism: add_datasets answers 400 "You can only add object names that do not exist in the collection", and Collection.update(data_refresh=True) worked once in testing then failed consistently. Both are tried and the exact outcome of each is reported — so whether this is a bug in a particular service version becomes a one-minute question answered on the system in front of you, instead of an open investigation.

Step 6 is the one most easily left out, and the most valuable. _parse_one looks a hit's text up under a fixed list of column names (_TEXT_KEYS), so "the service answered" and "tdfs4ds can read the answer" are different claims — in 0.3.1.140 similarity_search returned eight real hits while the public entry point returned zero rows, silently. Consequently the throwaway table mirrors the real business-dictionary column shape (ID, DATABASE_NAME, OBJECT_NAME, OBJECT_TYPE, BUSINESS_DESCRIPTION) and passes metadata_columns, rather than inventing convenient names: a collection built over a column called TXT would return hits tdfs4ds parses to nothing while the step still passed, i.e. a self-test unable to reproduce the very bug class it exists for.

Step 9 exists because "the refresh returned without raising" is not "the index changed". A silently stale collection is worse than a refusal, so it is reported as a failure.

On failure it writes a standalone reproducer (report["repro"]) for a support ticket: a single .py using only teradataml/teradatagenai/langchain_teradata — never tdfs4ds, which support will not have — truncated at the failing step, with the service version, client package versions and the embedding configuration in the header. It reads every credential from environment variables, and is forced to plain ASCII: the user writes it out with open(path, 'w') on a machine whose encoding we do not control, so one em-dash echoed back from a server error would make saving it raise UnicodeEncodeError — the artefact that exists to report a problem must not be able to create one.

Contracts, each with a test: never raises (every step individually wrapped, the walk always reaches cleanup); never leaks a secret (variable names and non-secret config only); cleans up or names exactly what it left behind (destroy() is admin-only, so a failed cleanup is a legitimate outcome and must not be silent); unique names per run (tdfs4ds_selftest_<8 hex>), so a leftover from an earlier failed run can never collide with this one. A failure in one step skips the dependent steps rather than cascading, because a derived error from a step that never had a chance buries the real one.

Also wired into check_configuration(selftest=True) as a seventh check. It runs only when asked — it is the only part of that function that writes — but the row is always reported, saying so and naming the flag. Hiding it entirely (the first cut) made the feature undiscoverable: a passing report gave the reader no way to learn that everything it had just verified was read-only. Opt-in is about not writing unasked, not about hiding. The full per-step report and the reproducer travel intact under the check's selftest key rather than being flattened to one line.

Added — per-check timings and progress, so a slow check is not mistaken for a hang

check_configuration() printed nothing until every check had finished, which on a slow service is indistinguishable from a hang — and gave no way to tell which check was slow. Measured on a live system: set_auth_token ~11 s, the first Vector Store REST call ~20 s cold (~3 s warm), and one health() call observed at 83 s while the same call minutes later took 1.2 s.

Each check is now announced before it runs and logged with its outcome and duration when it finishes, and every check row carries elapsed_s. The rendered report appends [N.Ns] to checks taking a second or more — only those, since 0.0s against every instant check is noise that hides the one that matters.

[0.3.1.167] - 2026-07-31

Fixed — build_vector_index reported a routine no-op as "service unreachable / check licensing"

Seen live: a second build_vector_index(force_rebuild=True) run died with "Teradata Vector Store service is unreachable or not enabled on this system … Check that the Enterprise Vector Store feature is licensed/activated" — while check_configuration() in the same notebook was green.

Both were right, and the message was wrong. The chain:

  1. The collection exists, so from_datasets() raises already exists. Expected — that is the trigger for the reuse path.
  2. The reuse path calls add_datasets() with the same FS_V_BUSINESS_DICTIONARY_* view that is already attached, and the service answers 400 You can only add object names that do not exist in the collection — the KNOWN LIMITATION already written into build_teradata_index's own docstring. It is the expected steady state of every repeat call, not an edge case.
  3. _reraise_if_service_down then matched the bare substring "response code:" and raised the licensing message — on a system whose Vector Store had just answered the request. The same misclassification shape as the SQLState 28000 fix in 0.3.1.164 and the 409 fix in 0.3.1.137: every TeradataMlException from this API embeds Response Code:, so that substring can never distinguish a genuine gateway failure from an in-service one.

_is_source_already_attached_error now classifies it; _reraise_if_service_down returns for it (terminal but benign — "nothing to add" — so the caller decides); and build_teradata_index logs a warning naming the real remedy and continues to the remaining stores instead of aborting on the first one. Deliberately kept distinct from _is_transient_staging_object_error: both are 400s mentioning "not exist", but that one is retryable and this one is not, so conflating them would make the retry wrapper spin.

check_configuration() passing was consistent, not contradictory. Its Vector Store check is health() + list() — both read-only. This is a write-path-only condition. Same lesson as 0.3.1.164: ask what a check would still pass with the failure present.

Fixed — force_rebuild=True was a no-op on the Teradata backend

From 0.3.1.133 to 0.3.1.166 both branches of build_teradata_index ran identical code, so force_rebuild was "purely informational". A user setting it — which they do precisely because the index looks stale — got the same no-op, and once the source was attached, the same failure on the retry. It now really destroys and recreates each collection (_destroy_for_force_rebuild), which is currently the only working way to pick up business-dictionary rows documented since a collection was created, and also the way out of a *_failed collection (so it runs before the failed-status check, whose own prescribed remedy is this destroy + recreate).

destroy() is admin-only, so a refused destroy is a warning and a fall-through to the ordinary reuse path, never a raise — a non-admin session ends up exactly where it was before, not newly broken. force_rebuild=False still never destroys.

Added — the Vector Store SERVICE version is now checked

check_configuration()'s Vector Store check gained a third stage: the service version reported by health(), compared against _VECTOR_STORE_MIN_VERSION = (2, 1, 11). An older service is up and accepts the credential — stages 1 and 2 both pass — and only misbehaves once something writes, so the version is the one place this is visible before a real call fails.

The hint says what the fix actually is: the Vector Store service is a server-side component of the Vantage system, so upgrading it is a platform/admin action, not a pip install (and not a licensing hunt) — with VECTOR_STORE_BACKEND = 'chroma' as the meanwhile workaround.

Evidence, not inference: 2.2.29 verified working at a customer site; a service reporting 2.0.5 was seen failing in the field. The boundary between them was not bisected, so a version in between is untested rather than proven. Version parsing is fail-open — an unrecognised health() payload shape is reported as "version not reported, could not verify" and never fails the check.

[0.3.1.166] - 2026-07-30

Fixed — the query optimizer could silently change business logic when flattening

The optimizer accepted a rewrite on two conditions: EXPLAIN compiles, and user_score does not regress. Neither has anything to do with the query still meaning the same thing.

The alias mechanism. Simplification flattens nesting, which merges two previously-distinct name scopes. A bare column that resolved uniquely in the nested form can then either

  • rebind to a different relation — EXPLAIN-clean, score unaffected, silently wrong; or
  • become ambiguous (Error 3809), at which point the existing syntax-repair loop asks the LLM to fix it — and the LLM picks an alias with no information about which one was correct.

That repair path is the real vector: a coin flip gets written into the accepted SQL.

The fix is to remove the degree of freedom before the LLM sees the query. If a query compiles, every bare column in it already resolves uniquely in its own scope — Teradata rejects ambiguity outright — so the correct qualification is derivable, not a guess. New module agent/sql_qualify.py: qualify_columns(sql, schema) resolves each bare column against the relations visible in its scope and writes the binding down explicitly. A later flatten then cannot rebind it, because the alias travels with the column.

Design constraints, all load-bearing:

  • Never regenerates SQL. It only inserts alias. at computed character offsets; every other byte is preserved. Parsing to an AST and re-emitting (e.g. via sqlglot) would mangle CURRENT VALIDTIME, QUALIFY, PERIOD/INTERVAL and UAF constructs — the same class of silent corruption, from a new source. No new dependency was added.
  • Only qualifies what is provable. A prefix is written only when the name belongs to exactly one visible relation whose column list is fully known. Not found, found twice, or any relation in scope with an unknown column list → the token is left alone. Ambiguous names are reported, never guessed, and surfaced in the optimization report.
  • Fail-open per scope, so the worst case is the previous behaviour.
  • Schema comes from HELP COLUMN db.obj.* (not DBC.ColumnsV, whose ColumnType is NULL for view columns — and process views are the common case).

Plan comparison was deliberately NOT used to validate it. It is the wrong oracle in both directions: a wrongly-bound column very often yields an identical plan (same joins, same spools), so it misses the error it was meant to catch, while estimated row counts drift for reasons unrelated to semantics and would cause false rejections. The qualified form is EXPLAIN-checked only as a compile guard against a bug in our own parser.

Added — structural invariant gate (agent/sql_invariants.py)

Qualification closes the rebinding hole; it does nothing about the other flatten failures, which all compile cleanly. Worse, user_score is anti-correlated with safety: it is derived from the count of [You] items, so a candidate that drops a predicate or a DISTINCT produces a simpler plan, scores higher, and is accepted preferentially.

diff_invariants(original, candidate) compares a coarse structural fingerprint — source tables, join flavours, predicate count, DISTINCT/QUALIFY, aggregation grain, window PARTITION BY, aggregates, set operators. Only losses are blocking; doing more is a warning. Grain and partitioning are compared alias-insensitively, because alias renaming is a legitimate rewrite and comparing raw text would flag every one of them.

Wired into three acceptance points, before the score is consulted: skill_simplify_query's accept branch, every initial candidate in skill_generate_candidates, and every refinement round (compared against the original, so drift cannot accumulate across rounds).

Qualification is kept even when a simplification is rejected — it is a semantics-preserving hardening that makes every downstream strategy safer.

Live-verified (Vantage 20, 12/12)

HELP COLUMN on a table and a view; qualified SQL compiles and is row-identical to the original (bidirectional EXCEPT = 0); the naive flatten of the unqualified form really does raise Error 3809; the qualified form flattens cleanly and stays row-identical; and the wrong binding a repair-loop guess would produce compiles fine while changing 4 rows.

One finding worth recording: the lossy rewrite the invariant gate blocked (dropped predicate + narrowed PARTITION BY) returned identical rows on the fixture data — every provider had a non-null specialty, and the two claims of one patient shared a drug code. So a row-comparison EXCEPT probe would have passed it. An EXCEPT is only as complete as the data that happens to exercise the difference; on a bounded FilterManager partition — exactly how a process runs — a lossy rewrite can pass a row comparison and still be wrong on the next partition. The static structural gate is complementary to a row-comparison oracle, not redundant with it.

Offline coverage: tests/test_sql_qualify.py, tests/test_sql_invariants.py, tests/test_optimizer_safety_wiring.py (63 tests, no DB, no LLM).

[0.3.1.165] - 2026-07-30

Fixed — run() crashed after writing its rows when no primary index was registered

Reported from a live Lake run over an ordinary permanent table:

File "tdfs4ds\feature_store\feature_data_processing.py", line 108, in generate_collect_stats
    primary_index.sort()
AttributeError: 'NoneType' object has no attribute 'sort'

apply_collect_stats declares primary_index as Optional[str], _upload_features passes through whatever the process catalog returned — None for a process registered without an explicit primary index — and generate_collect_stats called .sort() on it unconditionally. The '' defaults in its own signature never applied, because both arguments are always passed explicitly. Passing primary_index= to upload_features did not avoid it: the value does not survive the round-trip through the catalog.

Ordering makes this worse than a crash. COLLECT STATISTICS runs after ingestion, so the feature rows were already written — a caller catching the AttributeError was left with correct data and a process whose statistics were never collected, rather than a clean abort.

None and '' are now both normalised to "not specified" (previously '' reached the SQL as a malformed COLUMN()), a bare string to a one-element list, and partitioning=None to '' — it would have raised on .upper() four lines later for the same reason. Sorting switched to sorted(): generating a query string was reordering the caller's own list as a side effect.

No signature change and no behavioural change for any call that currently succeeds.

[0.3.1.164] - 2026-07-30

Fixed — a rejected credential reported as "service not licensed"

Live on VCE/preprod: check_configuration() returned all-green while rebuild_vector_indexing() failed with

Teradata Vector Store service is unreachable or not enabled on this system ...
Check that the Enterprise Vector Store feature is licensed/activated ...
Detail: (TDML_2412) Failed to execute collections_list. Response Code: 500,
Message: [SQLState 28000] The UserId, Password or Account is invalid.

Two independent defects, one visible symptom.

The message was wrong. SQLState 28000 is the SQL standard's invalid authorization specification — the service answered and then the database refused the credential it presented. _reraise_if_service_down classified it on "Response Code:" alone, so it read as unreachable/unlicensed and sent the user after a feature flag. New _is_auth_rejected_error matches the 28000 markers before the generic branch (the same position and reasoning as the three existing transient pre-checks) and raises _AUTH_REJECTED_MSG, which names the TDFS4DS_VECTOR_STORE_AUTH_* variables and the three real causes, expiry first.

The check could not have caught it. _check_vector_store_service called only CollectionManager().health() — a service liveness endpoint that never opens a database session, so it passes with a credential the database would reject. And check 1 cannot cover the gap either: teradataml's JWT branch stores a ready-made token without validating it (the validation call is commented out upstream), so set_auth_token() succeeds on an expired token. Between them, nothing green had ever spent the credential.

The service check is now two-stage, mirroring the MCP check: health() for reachability, then list() — what every real path starts with — for the credential end to end. Stage 1 passing while stage 2 fails is now itself the diagnosis. The JWT auth check states in its own detail that the token is stored unvalidated, so a green auth line never again reads as a promise the stage below can contradict.

Tests: the live 28000 string drives both the classifier and the report; a 404 still classifies as service-down, so the new branch narrows the old one rather than replacing it.

[0.3.1.163] - 2026-07-30

Added — two-phase tool discovery, as a fallback

.162 bounded the MCP tool set with MCP_MAX_TOOLS, but that leaves a guess: pick it too high and the schemas still overflow, too low and the model loses tools it needed. This removes the guess by recovering from the failure instead of trying to predict it.

When binding the tool schemas raises a context-window error, the model is shown a one-line-per-tool index (name — description) — an order of magnitude smaller than the schemas, so ~130 tools fit in a couple of thousand tokens where their schemas fit in none — asked which tools the question needs, and only those are bound for one retry.

  • Nothing is paid on a working path. The extra model call happens only after a failure, so a setup whose tools already fit behaves exactly as before — asserted by a test that counts selection calls on the happy path.
  • tdfs4ds.MCP_TOOL_DISCOVERY (default True) disables it to fail fast; MCP_FALLBACK_MAX_TOOLS (default 8) bounds the retry, deliberately far below MCP_MAX_TOOLS — the larger set is what just overflowed.
  • One retry, never a loop. A second overflow means even the focused set does not fit, and looping would only burn model calls.
  • Never retried once output exists. An overflow on a later model call, after an answer or a chart has already been produced, re-raises rather than re-running and answering twice.
  • Never on an unrelated error. A 401 surfaces as a 401. is_context_window_error recognises how each gateway words it (LiteLLM/Bedrock ContextWindowExceededError, OpenAI maximum context length, Anthropic prompt is too long), including nested inside a generic BadRequestError body.
  • An empty selection re-raises rather than binding nothing — an agent with no tools fails more confusingly than the original error.
  • Selected names are verified against the real tool set, since a model asked for bare names will still number, quote or invent them.

Both the agent (_skill_mcp_tools_async) and check_configuration's MCP probe use the same path, so a passing diagnostic exercises the agent's recovery too. Helpers live in the shared agent/mcp_connection.py seam with the LLM injected, which keeps that module free of any dependency on the agent and makes the whole path testable without a model — see tests/test_mcp_fallback.py, which drives the real skill with a stubbed client, agent and LLM.

Still one-shot in nature: the retry re-selects from the same question. It is recovery, not mid-reasoning discovery — the model cannot reach for a tool it discovers it needs halfway through a task. That would need dynamic tool binding (the langgraph-bigtool pattern) and is not attempted here.

[0.3.1.162] - 2026-07-29

Fixed — the Enterprise MCP edition's ~132 tools overflowed the model's context

The first successful Enterprise connection listed 132 tools, several times what the Community edition exposes. Every tool's JSON schema is sent to the model on every turn, so the tool set is a context-window cost paid before the question is read — and the whole set did not fit:

BadRequestError: Error code: 400 - litellm.ContextWindowExceededError:
BedrockException: Context Window Error

This was not limited to the diagnostic. _skill_mcp_tools_async binds tools the same way (create_react_agent(llm, tools, …)), so the consumer agent's MCP skills would have failed identically against Enterprise. It is the first concrete consequence of the "different tool set" difference MCP_SERVER_EDITION was introduced to describe.

select_tools(tools, question, max_tools=None) (agent/mcp_connection.py) — the shared MCP seam both paths already import from — now bounds the tool set by tdfs4ds.MCP_MAX_TOOLS (default 32, 0 disables, env TDFS4DS_MCP_MAX_TOOLS):

  • At or below the cap the list is returned untouched, so a Community-edition install binds precisely the tools it always did. The cap engages only where the alternative is failing.
  • Above it, relevance is a deterministic token overlap against each tool's name and description — no LLM, no extra round trip. Ties keep the server's ordering, and unmatched tools still fill the remaining slots, so an unforeseen phrasing degrades instead of emptying the toolbox.
  • Tokenisation splits camelCase and matches crude plurals, because tool names are camelCase compounds (base_databaseList) while questions are English ("the databases"). Without both, every tool scores zero and the ranking silently collapses to "whatever the server listed first". Verified against the real Enterprise names with the useful tools deliberately listed last.

Fixed — a context-window error reported as "the model does not support tool calling"

That 400 mentions tools, so it matched the existing tool-calling hint and blamed the model's capabilities — when the model never got the chance to try. A dedicated hint is now matched first, naming MCP_MAX_TOOLS and stating plainly that this is not a tool-calling support problem.

[0.3.1.161] - 2026-07-29

Fixed — a trailing .env comment was being read as part of the value

_parse_dotenv skipped whole-line comments but never stripped a trailing one, so

TDFS4DS_MCP_PASSWORD=Secret#123   # prod password

authenticated with the literal Secret#123 # prod password. The damage surfaces far from its cause — as a rejected credential from whatever service received it — and the value looks correct in the file.

It now follows the python-dotenv rule, so a file means the same thing whichever loader reads it (users routinely call load_dotenv() themselves before importing tdfs4ds): in an unquoted value, whitespace followed by # starts a comment; a # with nothing before it stays in the value (PASSWORD=abc#def → abc#def, which generated passwords rely on); and a quoted value is verbatim, which is how to write a literal " #". Verified case-by-case against python-dotenv in tests/test_export_config.py.

Fixed — Error 8017 reported as a URL problem

A credential rejected by Teradata comes back through MCP as [Error 8017] The UserId, Password or Account is invalid — text carrying neither 401 nor auth, so it missed every entry in _MCP_HINTS and got the generic "check MCP_SERVER_URL and MCP_SERVER_TRANSPORT" fallback. That is actively misleading: reaching a database error proves the URL, the TLS and the Authorization header all worked. There is now a dedicated hint saying the account is the only thing wrong, and noting that a credential valid with logmech='TD2' can still fail if the MCP server logs on with a different mechanism.

[0.3.1.160] - 2026-07-29

Fixed — a partially-created dataset catalog was permanently unrepairable

DatasetCatalog had two existence predicates that disagreed. _exists() reported True only when all six objects (3 tables + 3 views) were present; create_catalog() refused — logger.error plus a raised ValueError — as soon as any of them existed. Every partial catalog therefore fell into the gap between them: the constructor sees "not all present", calls create_catalog(), and that rejects "some present".

Because the constructor self-heals on every instantiation, this fired on ordinary read paths (dataset_catalog(), the consumer agent's _resolve_dataset) and on connect()/setup() — an ERROR-level log and an exception for a condition the caller could do nothing about. Observed live with the three tables present and the views absent:

ERROR - The dataset catalog cannot be created because these tables already exist :
        ['DB.FS_FS_DATASET_CATALOG', 'DB.FS_FS_DATASET_ENTITY', 'DB.FS_FS_DATASET_FEATURES']
WARNING - _resolve_dataset: catalog query failed — …

create_catalog() is now idempotent: it creates the missing objects, leaves the existing ones alone (tables before the views that read them) and returns the list of what it created. Both it and _exists() read from one new _missing_objects(), so they can no longer disagree. This matches the self-healing convention the rest of the package already follows (documentation_business_dictionary_view_creation, time_manager_catalog_creation).

Also fixed, latent: create_catalog(schema_name=…) rebuilt the object names with a different convention than the constructor — {name}_CATALOG and V_ instead of FS_{name}_CATALOG and FS_V_ — producing a catalog no other part of tdfs4ds can find, since every other call site spells it FS_V_{DATASET_CATALOG_NAME}_*. Both paths now derive names from one _build_object_names(), and overriding the table database moves co-located views with it instead of silently splitting the catalog. setup() no longer re-probes with _exists() after the constructor already healed (one round-trip less), and logs what was actually created rather than claiming "created" unconditionally.

Not a bug, for the record: the doubled FS_FS_ prefix is the intended convention — DATASET_CATALOG_NAME is 'FS_DATASET' and DatasetCatalog adds its own FS_. All ~10 call sites agree on it; renaming would orphan every existing store.

New DB-free tests in tests/test_dataset_catalog.py.

[0.3.1.159] - 2026-07-29

Fixed — three defects surfaced by the first full live check_configuration() run

Two of them were in the diagnostic itself, which is the worst place for them: a tool whose job is to name the setting to fix was blaming a correctly-set variable, and reporting a cause it had discarded.

1. config_loaded failed on legitimately falsy values. _unloaded_env_vars classified by truthiness, so TDFS4DS_INSTRUCT_MODEL_VERIFY_SSL=false — which loads correctly as False — was reported as "set in the environment but NOT applied", failing the whole report. The live run contradicted itself: the instruct check printed TLS verification DISABLED in the same report, which it can only do by reading that False. It also hit DISPLAY_LOGS=false (common), DEBUG_MODE, USE_VOLATILE_TABLE, BUILD_DATASET_AT_UPLOAD, and EMBEDDING_BATCH_DELAY=0 / SCHEDULING_MAX_WORKERS=0. Both classifiers now share _env_attr_status(), which compares against the coerced expected value — what load_config itself applies — and yields applied / unloaded / diverged. False == False is applied; only a genuinely empty attribute (None, '', [], {}, missing) is unloaded; anything else is a divergence, still reported and still non-fatal.

2. A custom httpx client silently stopped following redirects. openai's own _DefaultHttpxClient sets follow_redirects=True; httpx defaults it to False, and a supplied http_client is used verbatim (the SDK only overrides redirects per request for endpoints that declare it, which chat completions does not). So building a client for any reason — INSTRUCT_MODEL_PROXY, VERIFY_SSL=False, a CA bundle — turned redirect-following off as a side effect, and an ingress redirect surfaced as a bare Error code: 302. Present for proxy users since .158 wired that client into build_llm. _build_http_client now sets it, the same line and the same reasoning as mcp_connection._ca_client_factory. Timeout needs no such mirroring — the SDK compensates structurally, adopting its own 600 s default whenever the supplied client is left at httpx's.

This also closes the 'str' object has no attribute 'model_dump' loop from .158: httpx rewrites a 302 POST into a GET, so with following on the SDK lands on the redirect target (typically an HTML login page → 200 + non-JSON → raw text → .model_dump() on a str) and with it off the raw 302 appears. One ingress fault, two symptoms — now one hint, plus a redirect probe that reports where the URL actually leads (GET … -> 302 -> https://…). The probe sends no credential and strips the Location's query string, since an SSO redirect carries tokens there.

3. ExceptionGroup hid every MCP stage-1 cause. An anyio task group stringifies as unhandled errors in a TaskGroup (1 sub-exception) — a count, not a cause — so expired certificate, 401, 404, wrong transport and DNS failure all collapsed into the same string and no hint could match it. _short now flattens groups to their leaves (Type: message, recursively, duck-typed so the exceptiongroup backport works too), fixing every except site at once. A new hint covers transport mismatch (404/405/not acceptable/text/event-stream) and points at MCP_SERVER_TRANSPORT.

Not a defect, for the record: an MCP failure is not caused by a failing instruct model. Stage 1 (get_tools) involves no model, and stage 2 is skipped, never failed, without one — the — N tool(s): suffix in the detail string is what distinguishes them.

[0.3.1.158] - 2026-07-28

Added — TLS control for the instruct model's HTTP calls

A LiteLLM gateway behind an internal certificate authority is only reachable with a custom httpx client:

ChatOpenAI(model=…, api_key=…, base_url=…, http_client=httpx.Client(verify=False))

tdfs4ds builds that client itself, so there was no way to express it — the deployment simply could not use the endpoint.

  • INSTRUCT_MODEL_VERIFY_SSL (default True) — False disables certificate verification.
  • INSTRUCT_MODEL_CA_BUNDLE — the safer fix for the same problem: verification stays on, anchored to your internal CA's PEM (the pattern MCP_SERVER_CA_BUNDLE established in .157). Prefer it when the certificate is obtainable.
  • Instruct-only by design. Embedding clients are untouched whatever the instruct model does, so a shared helper cannot leak one endpoint's TLS posture onto another.
  • Backward compatible by construction: the existing _proxy_http_client already returned None when nothing was configured, and that is exactly the guarantee wanted here. It is generalised (and renamed _build_http_client, old name kept as an alias) to build a client when any of proxy / verification / CA bundle applies — with proxy and TLS composing into one client rather than one discarding the other. With nothing set, no http_client argument is passed at all, asserted on the real ChatOpenAI kwargs.
  • Disabling verification is never silent: a warning names the variable and states that the API key then travels over a connection whose peer was never authenticated. check_configuration also appends TLS verification DISABLED to the instruct check's detail, so an insecure-but- working setup is visible in the report rather than buried in config.
  • A malformed or missing CA bundle raises a RuntimeError naming the variable instead of an opaque [X509] PEM lib error on the first request — and is never downgraded to default verification.

Added — a hint for the opaque 'str' object has no attribute 'model_dump'

Reported live. The traceback names langchain internals, but the actual event is that the endpoint returned a success status with a non-JSON body: the OpenAI SDK returns response.text verbatim when the content-type is not JSON (openai/_legacy_response.py), and langchain then calls .model_dump() on a str. Neither a TLS nor a model problem, which is what makes it so hard to place.

The new hint says so, and points at the two real causes: a wrong INSTRUCT_MODEL_URL path (the SDK appends /chat/completions), or an ingress answering with a redirect, login or error page. A companion hint covers genuine certificate-verification failures, pointing at INSTRUCT_MODEL_CA_BUNDLE.

Fixed — build_llm never passed an HTTP client, so INSTRUCT_MODEL_PROXY never applied there

Found while verifying the above end-to-end: build_llm (genai/documentation.py) constructs its own ChatOpenAI / AzureChatOpenAI and passed no http_client at all. Only the consumer agent's _get_llm ever injected one — so the documented outbound-proxy support silently did nothing for documentation, the query optimizer, and check_configuration's instruct check, and the new TLS settings would have missed exactly the paths that need them.

All three OpenAI-family branches in build_llm now inject the shared client (and its async counterpart) when one is configured, and still pass nothing when none is — so the fix cannot change behaviour for an install that configures neither. Regression-tested for both the proxy and the TLS setting.

Offline tests: tests/test_http_client.py (19).

[0.3.1.157] - 2026-07-28

Added — two MCP editions, and authenticated access to Enterprise

There are two Teradata MCP products. The one tdfs4ds has talked to until now is the Community Edition, which works because it embeds its own credentials — the connection was a bare {"url", "transport"} dict with no way to send headers. Enterprise differs in two ways: it requires caller-supplied HTTP Basic auth, and it exposes a different tool set. That makes the edition a real distinction rather than an auth flag.

TDFS4DS_MCP_SERVER_URL=https://your-enterprise-mcp/one-td/mcp
TDFS4DS_MCP_SERVER_EDITION=enterprise
#   -> Basic auth built from TDFS4DS_TD_USERNAME / TD_PASSWORD
  • MCP_SERVER_EDITION — 'community' (default) or 'enterprise'. Credentials default to the Teradata connection credentials: the Teradata MCP server authenticates with the database identity, so there is nothing to restate. Override with MCP_USERNAME / MCP_PASSWORD when the MCP account differs.
  • MCP_SERVER_HEADERS sends verbatim headers for any other scheme (Bearer, API key) and wins over the edition; MCP_AUTH ('none'/'basic') overrides the mechanism the edition implies.
  • MCP_SERVER_CA_BUNDLE re-anchors TLS trust for an internal CA. Verification is never disabled — and a bad bundle raises a RuntimeError naming the variable rather than an opaque [X509] PEM lib at connect time, or being silently downgraded to default verification.
  • New agent/mcp_connection.py is the single seam, used by the agent and by check_configuration, so a passing diagnostic describes the connection the agent actually makes. With none of the new variables set it returns exactly {"url", "transport"} — byte-identical to the dict it replaced.
  • Credential values are never logged or returned; describe_mcp_auth() names the variable each part came from. MCP_PASSWORD and MCP_SERVER_HEADERS are credentials, so export_config redacts them and JSON config rejects them.

Why the edition gates the credentials. TDFS4DS_TD_USERNAME/TD_PASSWORD are set in essentially every deployment. Deriving Basic auth from them unconditionally would make every existing Community install start sending the Teradata password to its already-configured MCP endpoint on upgrade — a credential reaching somewhere it does not reach today, opted into by nobody. Defaulting the edition to community makes the backward compatibility structural rather than incidental.

Capability differences are exposed, not guessed. The edition is added to the ReAct system prompt, and the MCP check lists the tool names it finds — that is how the CE/Enterprise delta gets documented, from the live server. Nothing name-specific changed: _make_guarded_tools guards on tool input text (name-agnostic) and chart_render._infer_kind already falls back from its plot_* map to the tool description, so an unfamiliar Enterprise tool set degrades rather than breaks.

Added — the MCP check runs in two stages, and takes your question

check_configuration() gains a sixth check, and it does more than ping:

  1. Reachability and auth — get_tools(), reporting the tool count and names.
  2. Can the model actually drive them? — a minimal create_react_agent over the guarded MCP tools, asking by default for the list of available databases (both editions answer it). Override with check_configuration(mcp_question="…").

The split is the diagnosis: stage 1 failing means URL / auth / TLS; stage 1 passing while stage 2 fails means the model side — listing tools proves nothing about tool calling, and a model (or the gateway in front of it) can chat perfectly while never forwarding tool schemas. Stage 2 is skipped, not failed, when no instruct model is configured.

The probe is deliberately not routed through consumer_agent: that agent answers business questions about the feature store, so a server-administration question would test its classifier rather than the MCP tool path.

Offline tests: tests/test_mcp_connection.py (22) plus the MCP cases in tests/test_check_configuration.py.

[0.3.1.156] - 2026-07-28

Fixed — "package too old" was the wrong diagnosis; it is the wrong environment

The two teradatagenai hints added in .155 were built on an assumption that turned out to be false. Checked against a real install: teradatagenai==20.0.0.7 ships vector_store.collection and does accept api_type='td_hosted' — its own docstring lists 'litellm', 'td_hosted' and 'guardrails'. Yet a notebook reporting exactly that version via !pip freeze failed with No module named 'teradatagenai.vector_store.collection' and an api_type list missing all three.

Both symptoms have one cause: the interpreter importing the package is not the one pip was inspected in. !pip resolves through PATH; the kernel imports from its own sys.executable. Advising an upgrade sends the user round in a circle — the version they would "upgrade to" is already installed, just not where they are running.

  • New _package_origin() reports where the package actually imports from and under which interpreter, and both checks now put that evidence in their detail. This is what distinguishes "too old" from "different environment" — two failures with identical tracebacks and opposite fixes.
  • Both hints rewritten to name the environment split, point at %pip install (the % magic installs into the running kernel, !pip does not), and mention a shadowing teradatagenai.py/directory in the working directory.

Fixed — the litellm hint sent users to install packages they don't need

A live run hit cannot import name 'ChatLiteLLM' from 'langchain_community.chat_models' (recent langchain-community releases have dropped it) and was told to install two packages. But that deployment points INSTRUCT_MODEL_URL at a LiteLLM proxy — which speaks the OpenAI API, so INSTRUCT_MODEL_PROVIDER='openai-compatible' reaches it through ChatOpenAI with no extra packages at all.

  • The hint now leads with that: proxy → 'openai-compatible', same URL/key/model. The package-install advice is kept for the other topology (the LiteLLM Python SDK in-process, no proxy, model strings like 'bedrock/amazon.nova-pro-v1:0').
  • Documented as a table in docs_src/configuration/llm-config.md — picking the wrong one of the two LiteLLM topologies is the common mistake.

The 'litellm' provider itself is unchanged and still supported.

Nothing else changed; .155's Config-load check and export_config() are unaffected.

[0.3.1.155] - 2026-07-28

Added — check_configuration() now catches the load-ordering trap

Found immediately in live use: check_configuration() reported the instruct and embedding models as "not configured" and the backend as chroma, on a system whose environment plainly set all of them — while the Vector Store auth check passed. The asymmetry is the explanation: auth variables are pass-through and read live from os.environ, but everything else reads tdfs4ds.<ATTR>, which config.py populates once, during import tdfs4ds. Anything exported after that import (or a .env outside the working directory) never arrives.

Telling the user to "set INSTRUCT_MODEL_PROVIDER" when they had set it is exactly the confusion this tool exists to remove.

  • New Config load check, run first: compares every TDFS4DS_* variable in the environment against the module config and fails when one is set but not applied, naming the variables and pointing at tdfs4ds.load_config().
  • It also reports variables whose value differs from the environment — without failing, since an in-session override is legitimate. This catches the quiet half: an attribute sitting at its default (VECTOR_STORE_BACKEND='chroma') while the environment asks for something else is not "unset", so nothing else would notice. That is what silently skipped the Vector Store service check in the report above.
  • The instruct and embedding checks now say "…but TDFS4DS_X is set in the environment" and give the load_config() hint instead of blaming the user.
  • Names only, never values — the comparison covers credential variables too.

Added — tdfs4ds.export_config(): capture a working configuration

tdfs4ds.export_config()                                  # inspect the text first
tdfs4ds.export_config('tdfs4ds.json')                    # reusable JSON config
tdfs4ds.export_config('.env', fmt='env', source='env')   # env vars -> a file
  • source='module' (default) snapshots the running configuration, in-session changes included; source='env' converts the TDFS4DS_* environment variables into a file even when they never reached the module config — the trap above.
  • fmt='json' writes what load_config() reads; fmt='env' writes TDFS4DS_* lines and is the only format that can carry connection and Vector Store auth variables (they are pass-through, never module attributes).
  • Secrets are redacted by default, written as a commented placeholder so the result is a shareable template; include_secrets=True writes real values. JSON never carries credentials at all — _apply_file rejects them on read, so writing them would leak and not work.
  • Returns the text whether or not a path is given (inspect before writing, as generate_execution_script does).

Fixed — three hints blamed a missing package for a version problem

All three surfaced in the same live run:

  • No module named 'teradatagenai.vector_store.collection' said "the package is missing" when the top-level package imports fine. A submodule ImportError is now matched first. (The advice given here — "upgrade" — was itself wrong; corrected in .156.)
  • No module named 'langchain_community' gave a generic list of LangChain integrations. It now names the actual requirement for the litellm provider: pip install langchain-community litellm — neither is a tdfs4ds dependency. The generic hint also maps each provider to its package.
  • Invalid value(s) 'td_hosted' passed to argument 'api_type' was read as a missing-argument problem. It is not — the build being imported does not know that api_type. (Attributed to the version here; corrected to the environment split in .156.)

Fixed — secret detection redacted a harmless setting

_is_secret matched TOKEN as a substring, so EMBEDDING_MODEL_MAX_TOKENS (an integer) would have been redacted out of exported .env files. Matching is now on the name suffix, plus the explicit _CREDENTIAL_VARS set.

Offline tests: tests/test_export_config.py (16 new) and the extended tests/test_check_configuration.py.

[0.3.1.154] - 2026-07-28

Added — Vector Store auth accepts a token you already hold (JWT mechanism)

tdfs4ds could build three teradataml.set_auth_token(...) mechanisms (PAT, Basic, OAuth), all of which make it generate a token. It had no way to express the case where the caller already holds one — the normal shape in-cluster, where the platform injects an OAuth access token into the pod:

tdml.set_auth_token(base_url='http://vectorstore.vectorstore.svc.cluster.local:8000',
                    auth_token=os.getenv('OAUTH_ACCESS_TOKEN'))
  • New TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN. In-cluster, map the platform's injected token onto it — TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN=$OAUTH_ACCESS_TOKEN.
  • Only TDFS4DS_-prefixed variables are read. Reading the platform's generic name directly was considered and rejected: such a name can be owned by an unrelated component in the same environment, and picking it up would forward that component's credential to the Vector Store endpoint with nothing in the configuration naming it. The one-line mapping above keeps the intent explicit — and keeps every tdfs4ds input inside its own namespace, so .env files behave the same way.
  • JWT is tried first, ahead of PAT → Basic → OAuth. Safe for every existing install (the variable is new, so nothing that resolves today changes), and it means an explicitly supplied token is never silently ignored in favour of a generated one.
  • The token value is never logged — only the base_url and the name of the variable it came from, so a wrong-token diagnosis stays possible without leaking it.
  • resolve_auth_mechanism() / configured_auth_variables() split out of configure_vector_store_auth(): one definition of the mechanism table, now also readable without side effects (no token generation, no network) — which is what check_configuration(live=False) uses.
  • _NOT_CONFIGURED_MSG (the error a user actually hits) lists the new group too.

Added — tdfs4ds.check_configuration(): validate the setup, and say how to fix it

Nothing answered "is my AI configuration actually usable?" until something failed deep inside build_vector_index() or a documentation run, with an error naming the symptom rather than the missing setting. New module tdfs4ds/diagnostics.py:

tdfs4ds.check_configuration()             # live: real handshake, one prompt, one embedding
tdfs4ds.check_configuration(live=False)   # resolve + build clients only, call nothing

Four checks, in dependency order:

Check Applies when What it does
Vector Store auth backend is teradata, or any auth variable is set resolves and applies set_auth_token, reports the mechanism
Vector Store service backend is teradata CollectionManager().health()
Instruct model provider + model set build_llm(), then a hello-world prompt
Embedding model always branches on the backend (below)

The embedding check deliberately differs per backend: with the Teradata Vector Store and TERADATA_AI_API_TYPE set it verifies only that TeradataAI(...) instantiates — in-database embedding needs nothing further from the client; with ChromaDB (or Teradata without TERADATA_AI_API_TYPE) it embeds a hello-world string and checks the vector length against EMBEDDING_MODEL_DIM, because a mismatch there never raises — it silently builds an index that cannot be searched.

  • Every non-passing check carries a hint naming the setting to change — matched from the error text (401 → which API key; 404 → list_instruct_models(); 422 on structured output → the LiteLLM/Mistral provider gotcha; 422 "Input should be a valid string" → the tiktoken-array bug; 502/503/504 → transient, retry; and so on), with a per-check fallback so an unrecognised failure still gets direction. A skipped check hints too, naming the variable to set.
  • Never raises and never leaks. Each check is wrapped individually; only variable names, URLs and model names reach the output — never a token, key or password. A legacy Windows console that cannot encode the emoji gets ASCII markers rather than a UnicodeEncodeError.
  • Reuses the real code paths (build_llm, get_embeddings, get_vector_store_embedding, configure_vector_store_auth), so a passing check means the actual path works.
  • The Teradata database connection is not checked — it is external to the platform and owned by the caller (in-cluster, typically tdml.create_context(host=…, logmech='JWT', logdata=…)).
  • Surfaced in the admin Parameters panel as a Configuration check card (GET /ui/params/check, panels.check_configuration), rendering the same results and hints.

Offline tests: tests/test_check_configuration.py (26) and the extended tests/test_vector_store_auth.py (15).

[0.3.1.153] - 2026-07-27

Added — run_now=False now validates the source with EXPLAIN before registering

run_now=False (added in .152) skipped the run — and the run is what normally proves the feature source actually executes, so a lazily-declared process could bank an invalid source and only fail much later, at its first roll. It now EXPLAINs first.

  • The check runs before any catalog row is written, so an invalid source never leaves a registered process behind. On failure nothing is registered: a prescriptive message is logged and None is returned, no traceback — consistent with the other recoverable paths.
  • Closes a real gap: _resolve_feature_source already EXPLAINs the sql_query form (the query, then SELECT * FROM the created view) and the 'schema.view' form. It has never EXPLAINed the teradataml DataFrame form — which is also the form most likely to carry a fragile ephemeral view (the crystallize_view case). That path is now covered.
  • UAF ART views: only the FIRST art_sql statement is EXPLAINed, and the ART is NOT materialised. art_sql order is authoritative and a later statement may read an ART an earlier one creates — so without materialising, statements 2..n would fail for a reason that says nothing about their own validity. The feature view is skipped for the same reason (it reads the ART → Error 3807), matching the EXPLAIN skip document_process_incremental already applies to this process type.
  • EXPLAIN is not a guarantee, and the docstring says so. It proves the statement compiles and its objects are reachable. It does not prove ingestion: run() additionally does Gettdtypes/validate_feature_types and rejects NULL entity keys not covered by entity_null_substitute, and the real ingestion statement is an unpivot CTAS, not SELECT *. A lazily-declared process can EXPLAIN clean and still fail its first run.

Compat: no signature change. No parameter added, removed or reordered; run_now keeps its True default, so every run_now=True caller is completely unaffected (no extra round-trip is issued on that path — running already proves the source). The behaviour change is confined to run_now=False, which was introduced in the immediately preceding, unreleased .152: it can now return None where it would previously have registered an invalid process. That is the intended fix, not a regression.

[0.3.1.152] - 2026-07-27

Fixed — upload_tdstone2_scores could never run (register_entity arity)

  • upload_tdstone2_scores(model) called register_entity(entity_id) with one argument, but register_entity(entity_id, feature_types, ...) requires two — so the function raised TypeError: register_entity() missing 1 required positional argument: 'feature_types' on every call. Confirmed by signature binding.
  • Fixed by hoisting feature_names_types = tdstone2_Gettdtypes(model, entity_id) above the call and passing it: register_entity(entity_id, feature_names_types). The hoist is safe — tdstone2_Gettdtypes reads only the model's own predictions/computed features, never the feature-store tables register_entity creates. register_entity still runs before register_features, as it must.
  • Compat: cannot break any caller. The function had no working behaviour to depend on — it raised TypeError before touching the database. Reported by the tdquiver session.

Added — upload_features(..., run_now=False)

  • upload_features gained an optional run_now keyword (default True). With run_now=False the process is registered without being executed: the process-catalog row, data-distribution row and any filtermanager row are written and the view is created, but run() is not called and no feature values are computed. Pair it with return_process_id=True for a lazy declare-now/compute-later flow:
process_id = tdfs4ds.upload_features(..., run_now=False, return_process_id=True)
tdfs4ds.roll_out([process_id], time_manager=tm)   # drives every vintage, including the first
  • Caveat: register_entity / register_features and the FS_T_* table creation happen inside run(), so until the first run the feature catalog has no entry for these features and their storage tables do not exist — get_feature_versions returns nothing for them. That is the intended lazy state, not an error. Dataset documentation is skipped too (the dataset view does not exist yet).
  • Compat: fully additive. run_now was appended to the end of the signature (after return_process_id) — no existing parameter moved, so no positional shifts — and it defaults to True = register and run, exactly as before.

[0.3.1.151] - 2026-07-27

Added — upload_features(..., return_process_id=True)

  • upload_features gained an optional return_process_id keyword (default False). When True it returns the PROCESS_ID (a UUID string) of the process it just registered, instead of the dataset — so a caller can hold the id for a later run(process_id) / roll_out([process_id], …) without a process-catalog lookup:
process_id = tdfs4ds.upload_features(..., return_process_id=True)
tdfs4ds.run(process_id)
  • Mirrors the existing run(return_dataset=…) idiom: the flag toggles what is returned, it does not wrap the result in a tuple. The default return (dataset) is already None unless tdfs4ds.BUILD_DATASET_AT_UPLOAD is True, so in the common case nothing is given up by setting it.
  • The recoverable input-error paths (missing schema, target view already exists, SQL that will not compile) still return None regardless of the flag, with a prescriptive message logged and no traceback — a None return still means "check the log".

Backward compatibility: fully additive. The parameter was appended to the end of the signature — no existing parameter moved, so no positional argument shifts — and it defaults to False, so every existing caller returns exactly what it returned before. No existing signature, default, or return value changed. (Note: "appended", not "last" — 0.3.1.152 later appended run_now after it. The guarantee is that nothing before it moved, which holds however many kwargs are appended afterwards.)

[0.3.1.150] - 2026-07-16

Added — cross-store MIGRATION / promotion (lab → production)

Where .148 attaches other stores read-only, this imports selected work from a distant (lab) store into the primary (production) store. Two governed phases, both re-exported on tdfs4ds.

  • plan_migration(seeds, source, *, all_domains=True, target_data_domain=None, data_strategy='recompute', attach=True) (migration/planner.py) — READ-ONLY, never writes. Attaches source read-only (§.148), resolves a mixed process / dataset / feature seed list in the source store, and expands the upstream process closure — the same self-contained unit analyze_parallel_plan computes, reusing the parallel planner's pure graph core (_resolve_seeds / _build_process_dag / _upstream_closure / _compute_waves) rather than reimplementing it; only the catalog reads are re-scoped to the source. Walks each closure view's DDL to inventory and classify every object (process_view / dataset / intermediate_view / source_table), detects conflicts against the primary (feature-name triplet collisions, object-name collisions), inventories the filter/time managers (content-hash equivalence → unify vs recreate), and returns a JSON manifest that requires approval.
  • The lab→prod downstream-safety gate. For each object proposed for replace, downstream_feature_loss walks the primary's descendants (real production consumers) and marks the conflict blocking when a dependent needs a feature the incoming version drops. Dataset dependents are checked against their exposed feature set; process/plain-view dependents are treated conservatively as needing everything.
  • migrate_objects(plan, approvals=None, *, dry_run=False, rebuild_vector_index=True, progress=None) (migration/executor.py) — the only writer. Refuses to run unless validate_approvals passes: every conflict decided skip/replace, and no approved replace still blocking. Executes wave by wave, upstream-first — recreate upstream non-FS objects, migrate managers, re-register each process with a fresh primary PROCESS_ID (a source_pid → primary_pid map threaded throughout), bring in data (default recompute = run(primary_pid)), rebuild datasets, re-home the business dictionary, re-derive lineage, rebuild the vector index.
  • Writes resolve PRIMARY-only. The executor temporarily detaches read-only stores (SCHEMA_READONLY=[]) for the duration: the planner attached the source, which would otherwise leak the source's feature-table location into build_dataset. It reads the source through explicit schema names, never federation, so detaching is safe (restored in a finally).
  • Datasets are rebuilt, not copied. A dataset view carries store-local FEATURE_ID / FEATURE_VERSION references, so it is regenerated via build_dataset against the primary's freshly registered features with source→primary process-id remapping — a blind view copy would carry identifiers meaning something else in the target.
  • The linchpin is a pure DB/name rewrite (migration/rewrite.py): rewrite_references swaps fully-qualified src_db.src_obj → dst_db.dst_obj (quote/case tolerant, qualified-only so bare column names are safe), remap_process_ids rewrites FEATURE_VERSION='<uuid>' dataset references, build_placement_map puts tables → primary SCHEMA, views → primary view_schema().
  • render_migration_report(plan, fmt='markdown'|'html') is the approval report; generate_migration_script / write_migration_script emit a standalone .py that re-plans and executes with a baked-in approval map. Every auxiliary step (BD / lineage / vector) is fail-open. Admin "Migration" panel: source picker → seeds → Build plan → per-conflict approval controls (replace disabled when the gate marks it blocking) → staged SSE run + script download.
  • Live-verified end-to-end on a demo cluster (DEMO_USER → throwaway DEMO_MIG): process migration with source-table copy and view-reference rewriting, re-registration with a fresh pid, run() recompute of 8 features, dataset migration (ds_provider_stats rebuilt — view, catalog rows and 200 correct rows), re-run in the migrated store, idempotency on re-plan, approval enforcement, and the blocking gate firing against real lineage. Source untouched.
  • Known gap: the copy data path is a documented stub — only recompute has been exercised. _copy_feature_values still needs the FEATURE_ID-remapping INSERT…SELECT.
  • Pure helpers unit-tested without a DB: test_migration_rewrite.py, test_migration_conflicts.py, test_migration_planner.py.

Trial-cluster gotcha: a freshly CREATE DATABASE'd database can transiently read as "does not exist" in the next session — do create + setup + migrate in one session.

[0.3.1.149] - 2026-07-15

Fixed — parallel-plan write conflicts on a store whose features are not yet registered

A MERGE locks its target FS_T_* table, so two processes may run in parallel only if they write different tables — and physical tables are keyed by (data_domain, entity, feature_type). _pid_to_tables resolved that from the feature catalog, which on a fresh parallel build is still empty: the exact table could not be resolved, no conflict was detected, and same-entity processes were free to race the same MERGE.

  • When a process's features are not yet registered, _pid_to_tables now substitutes a synthetic (DATA_DOMAIN, ENTITY_ID) conflict key (_entity_conflict_key, sentinel database __ENTITY__, rendered entity:DOMAIN::ENTITY), so same-domain/same-entity processes serialize instead of racing.
  • The fallback fires only when the exact table set is empty. Once features are registered the exact table drives the decision again, so same-entity but different-data-type processes keep their parallelism.
  • Accepted rare gap: a registered process (exact table) and an unregistered one (synthetic key) writing the same physical table are not matched.

Added — opt-in cap on parallel workers

  • max_workers on generate_execution_script / write_execution_script caps the number of units run in parallel per wave via a single ProcessPoolExecutor(max_workers=…). A wide wave's extra units queue in the rolling pool — each starts the instant a worker frees; they are not split into synchronized sub-waves, and the only barrier remains the one between waves. A serialize_together group counts as one unit (its processes run back-to-back inside a single worker), so the cap bounds concurrent units while same-table MERGEs never overlap.
  • tdfs4ds.SCHEDULING_MAX_WORKERS (env TDFS4DS_SCHEDULING_MAX_WORKERS / JSON scheduling_max_workers, nullable int) is the global default; an explicit max_workers= argument overrides it for one call. When both are None the generated script is unbounded — one worker per unit, byte-identical to the historical behaviour.
  • Surfaced in the admin "Execution plan" panel as a Max parallel workers field → &max_workers= on the /ui/parallel/script download route.

[0.3.1.148] - 2026-07-14

Added — cross-store read federation (read-only extended connections)

  • tdfs4ds.SCHEMA_READONLY (default []) + extend_read_stores(databases) / detach_read_stores(databases=None) / list_read_stores() and connect(..., databases_readonly=[...]) (also TDFS4DS_SCHEMA_READONLY env / JSON schema_readonly) — connect to a primary store read-write as usual and extend the session to one or more other feature stores read-only. Each store is a string ('DB_B') or a dict descriptor {'schema','schema_view','vector_prefix'}.
  • Reads federate, writes don't. When read-only stores are attached, catalog / business-dictionary / lineage reads become a UNION ALL across the primary + read-only stores (each row tagged with a synthetic FEATURE_STORE provenance column, primary first), and build_dataset can pull a read-only store's features into a dataset built in the primary. upload_features / documentation / lineage writes still target only tdfs4ds.SCHEMA.
  • Byte-identical default. SCHEMA_READONLY == [] ⇒ every read funnel keeps its historical single-store form (the union builders return None) — no behavioural change unless a read-only store is attached. Same opt-in shape as the database_view split.
  • Teradata vector search federates live — search_teradata_index searches each store's own materialized collections by (vector_prefix, target_database) and merges hits (no re-index; a store without collections is skipped fail-open, still answerable via catalog/BD paths). Chroma (single local index) does not federate — out of scope.
  • New module tdfs4ds/read_federation.py (attach/validate + the shared catalog_union_sql / federated_* builders) and accessors read_view_schemas() / read_table_schemas() / read_stores(). Attach validation rejects a version-skewed / ungranted store (fail-open); optional has_bd/has_lineage capabilities are probed so a store without docs or lineage never black-holes those unions.
  • Documented collision limit: the consumer agent's identity/counting/domain-resolution is store-blind — exact for stores with disjoint domains/entities/feature names (the common case); a same-(domain, entity, feature)-across-stores clash resolves to the primary. Full provenance-in-identity hardening is a named follow-up.
  • Offline unit tests: tests/test_read_federation.py.

[0.3.1.147] - 2026-07-13

Added — parallel-execution planner, FS_TIME_MANAGER, "Execution plan" admin panel

  • tdfs4ds.analyze_parallel_plan(seeds=None, all_domains=True) — seed a list of processes / datasets / features, resolve each to its producing process(es), expand to the upstream process closure (everything needed to (re)build them — a self-contained migration unit), and group the closure into dependency waves: a topological layering of the process DAG (derived from the persisted FS_LINEAGE_GRAPH) where every process in a wave is mutually independent and can run in parallel, and waves run in order. seeds=None plans the whole store. The result also reports per-wave manager groups, unify_candidates (equivalent but distinct FilterManager/TimeManager objects), a serialize_together write-conflict guard, and processes with unknown lineage.
  • tdfs4ds.generate_execution_script(plan, mode='process_pool', ...) / write_execution_script(plan) — turn a plan into a standalone .py that runs waves sequentially and processes within a wave in parallel via separate OS processes (each worker opens its own Teradata context from TDFS4DS_TD_* env — safe against the module-global and shared-connection-pool hazards that make in-process threading unsafe).
  • tdfs4ds.unify_filter_managers / unify_time_managers (opt-in, dry_run=True by default) — repoint parallel-candidate processes onto one canonical manager (rewrite the process view's manager reference + update the catalog row).
  • New FS_TIME_MANAGER catalog — a per-process TimeManager association mirroring FS_FILTER_MANAGER, exposed as TIME_DATABASE_NAME/TIME_VIEW_NAME/TIME_TABLE_NAME on FS_V_PROCESS_CATALOG. It is populated by roll_out — as roll_out(process_list, time_manager, …) runs it records each process's TimeManager (idempotent, non-fatal). A TimeManager is a roll-out driver, not a registration-time property, so upload_features is unchanged (no timemanager= argument; registration SQL is byte-identical). Fully backward compatible and self-populating: connect() creates the (empty) table on pre-existing installs before refreshing the catalog view, and simply running your existing rollout fills it in — no migration step, including for processes registered by older versions.
  • Admin "Execution plan" panel — pick datasets/processes, see an interactive wave-layered workflow graph (what runs in parallel vs sequentially), and download the generated .py.

[0.3.1.146] - 2026-07-13

Added — database_view: optional separate database for views vs tables

  • New opt-in configuration for environments that keep views in one database and tables in another (governance / access-control policy). setup(database, database_view=...) and connect(database=..., database_view=...) accept a second database dedicated to VIEW objects, exposed as the global tdfs4ds.SCHEMA_VIEW (also settable via the TDFS4DS_SCHEMA_VIEW environment variable / .env, or the schema_view JSON config key — same tier as SCHEMA).
  • Fully backward compatible. database_view=None (the default) makes SCHEMA_VIEW resolve to SCHEMA through the new tdfs4ds.view_schema() accessor (SCHEMA_VIEW or SCHEMA), so every generated statement is byte-identical to before — no behavioural change unless a view database is explicitly configured.
  • When set: every physical TABLE stays in SCHEMA (feature FS_T_* tables, all catalog/_HIDDEN, business-dictionary, and lineage base tables) and every VIEW — process views, dataset views, all framework FS_V_* public views, per-feature value views, and the business-dictionary / lineage / process-catalog / dataset-catalog views — is created and read in SCHEMA_VIEW. Implementation: table sites keep tdfs4ds.SCHEMA; view create/read sites resolve through tdfs4ds.view_schema(). A view's FROM that reads a table keeps SCHEMA (tables never move); only a view-over-view FROM resolves to the view database.
  • Unqualified names now default to the view database. Registering a process/dataset with an unqualified name (e.g. upload_features(..., target_view_name='my_view')) places the view in SCHEMA_VIEW automatically (relaxing the previous hard "must be schema-qualified" rejection); an explicit db.view is still honoured. Covers _resolve_feature_source, build_dataset, and get_process_id.
  • FEATURE_DATABASE (feature catalog) stays the table database so cross-database dataset views resolve their "{FEATURE_DATABASE}"."{FEATURE_TABLE}" sources; DATASET_DATABASE records the dataset view database. DatasetCatalog gained schema_name_view / existing_view_objects.
  • Operational precondition (not enforced in code): the view database must be granted SELECT (and WITH GRANT OPTION to expose views to downstream consumers) on the table database.
  • Migration for an existing single-database store: set database_view and call connect(create_if_missing=True) to (re)create the framework views in the new database. Stale views left behind in the old database are not dropped automatically.
  • Out of scope (unchanged): TimeManager / FilterManager backing objects (placed by their own explicit schema_name) and the Enterprise Vector Store collection's own target_database.

[0.3.1.145] - 2026-07-13

Fixed — openai-compatible provider now uses response_format, not vLLM guided_json (LiteLLM / Mistral gateways)

  • Bug: documenting/scoring a SQL query through an OpenAI-compatible gateway that fronts a non-vLLM backend (e.g. a LiteLLM proxy → Mistral) failed with UnprocessableEntityError: 422 … "loc":["body","guided_json"], "msg":"Extra inputs are not permitted". Root cause: the openai-compatible (and vllm) provider path sent vLLM's guided_json structured-output parameter — a vLLM-only extension, not part of the OpenAI API — which such gateways reject. A provider literally named "openai-compatible" must speak the OpenAI API, i.e. use response_format.
  • Fix: openai-compatible is reclassified into the response_format json_schema group (alongside openai/azure/mistralai; strict is set only for openai/azure, which Mistral and generic gateways reject). vllm is unchanged — a real vLLM server still gets guided_json. Use provider vllm for a raw vLLM server, openai-compatible for any OpenAI-protocol gateway.
  • Centralized the provider → structured-output decision (mechanism and schema wrapping) into a single source of truth, _build_provider_llm_caller(llm, provider, base_schema, schema_name) in genai/documentation.py. The scattered per-site provider branching (build_documentation_json_schema, build_sql_documentation_chain, build_explain_documentation_chain, and the inline dataset path) — the reason the branches had drifted — now all route through it.
  • One-shot fallback: if a backend rejects the structured-output parameter (_is_structured_output_rejection — matches the 4xx extra_forbidden / "Extra inputs are not permitted" / "unknown|unsupported field/parameter" shapes naming guided_json / response_format / format), the caller retries once unconstrained and relies on the fence-tolerant _robust_json_parser. A genuine model error (context length, rate limit, content policy) never names those parameters, so it is not swallowed. This makes any gateway work even if it supports neither mechanism, so this class of bug cannot recur.
  • No config change required for existing users; no behaviour change for vllm, openai, azure, bedrock, gcp, litellm, or ollama. Immediate workaround for anyone already hitting this on a Mistral-backed gateway: set INSTRUCT_MODEL_PROVIDER='mistralai' (or now 'openai-compatible').
  • Tests: tests/test_structured_output_provider.py (pure unit, no Teradata/network) — mechanism selection per provider, the rejection detector against the real ticket message, and the one-shot fallback (incl. that unrelated errors are not swallowed).

[0.3.1.144] - 2026-07-12

Added — OFS backing-table storage for TimeManager / FilterManager (Vantage Cloud Lake + tdstone2 Apply)

  • Opt-in storage= constructor kwarg on both TimeManager and FilterManager (time_management.py, filter_management.py): None (default) = system default (block storage), 'OFS' = Object File System (MAP = TD_MAP2, STORAGE = TD_OFSSTORAGE). A companion ofs_map= kwarg (default 'TD_MAP2') overrides the map. The choice is per manager, never a global — a user can run block-storage (full primary cluster) processes and OFS/Apply processes side by side in the same session.
  • Context: on VCL, a tdstone2 model scored via the Apply backend runs on the Analytical Compute Cluster (ACC), which reads from OFS. storage='OFS' lets a manager's backing table live on OFS instead of block storage.
  • Making a manager-referencing view score on the ACC needs BOTH (confirmed live on VCL2): (1) the manager table on OFS (storage='OFS') so the ACC can read it — a block-storage read raises Error 6881; and (2) the filter written as a JOIN to the manager view, not a correlated scalar subquery — the ACC rejects ... WHERE d <= (SELECT BUSINESS_DATE FROM <tm_view>) with 6881 but accepts a JOIN. Verified: OFS + JOIN scores (~114s); OFS + scalar subquery still 6881s. So storage='OFS' is necessary but not sufficient on its own — it is genuinely part of the working recipe (the ACC reads the manager table through the JOIN), and the JOIN filter shape is the user's responsibility (a property of the scoring view's SQL). A JOIN keeps the reference dynamic (time-travel via update() still works); a hard-coded literal also passes but freezes the date. No literal-inlining helper or roll_out integration is added — the query shape is authored by the user.
  • roll_out/run() still cannot drive a tdstone2-Apply model — its ingestion CTAS (CREATE VOLATILE TABLE AS SELECT * FROM <live scoring view>) also hits 6881; call model.score() first, then upload_features.
  • New helper module utils/storage.py: resolve_storage, storage_clause(storage, ofs_map) (returns the CREATE TABLE options fragment — '' for the default so DDL is byte-identical to before, or , MAP = …, STORAGE = TD_OFSSTORAGE for OFS), is_acc_ineligible_error, and ACC_6881_HINT. The clause is injected immediately after the table name in the 5 backing-table CREATE sites (TimeManager load_time_steps / clone_timer hard / take_ownership; FilterManager load_filter / clone_filter hard).
  • Clearer 6881: _compute_features_batch (execution.py) now logs ACC_6881_HINT (still re-raises the original) when its own ingestion CTAS hits a 6881.
  • Docs: the OFS + JOIN Apply pattern (and the score()-then-upload_features sequencing) is documented across the time-manager / filter-manager user guides, the fs-rollout / fs-filter skills, and the tdstone2-troubleshooting 6881 table. No score-hook / query rewriter is coded — the filter shape is the user's responsibility (guidance only).
  • Backward compatibility: default behaviour unchanged (no MAP/STORAGE clause). New kwargs are keyword-only with defaults; existing positional calls and run()'s read-only FilterManager(...) reconstruction are unaffected. Offline tests in tests/test_manager_storage.py.

[0.3.1.143] - 2026-07-10

Added — 'uaf art view' process type (UAF features whose value lives in a secondary ART layer)

  • New opt-in, additive PROCESS_TYPE 'uaf art view' for UAF (Unbounded Array Framework) features whose value lives in a secondary ART (Analytical Result Table) layer — ARIMA fit-metrics (R_SQUARE/MAPE/AIC in ARTFITMETADATA), residual tests over ARTFITRESIDUALS, TD_PACF, etc. Reaching such a layer needs two statements (materialise the ART, then TD_EXTRACT_RESULTS over a LAYER), which the single ingestion CTAS (CREATE VOLATILE TABLE AS (SELECT * FROM <view>)) cannot express — so the inline-EXECUTE FUNCTION-in-a-view trick that serves inline-output operators (TD_ACF, TD_DICKEY_FULLER) does not extend to it.
  • upload_features(..., art_sql=[...], art_layers=[...]) — new kwargs. art_sql is a list of EXECUTE FUNCTION ... INTO VOLATILE ART(<name>) statements; art_layers names the declared LAYER(...)s the feature view reads. df/sql_query+target_view_name still build the plain feature view (reading the layer via TD_EXTRACT_RESULTS(ART_SPEC(..., LAYER(...)))).
  • Storage: no DDL migration. art_sql/art_layers are stored in the existing METADATA JSON(32000) column of the process catalog (a JSON array preserves the execution order of art_sql, which is load-bearing — a later statement may read an ART an earlier one created). PROCESS_TYPE is parameterised via kwargs.get('process_type', 'denormalized view') in the registration builders, so existing callers produce identical SQL.
  • run(): a new elif process_type == 'uaf art view' branch reads art_sql/art_layers back from METADATA and threads art_sql through _upload_features → _compute_features_batch, which — per FilterManager partition, after filtermanager.update(), before the ingestion CTAS, in the same session — calls the new _materialise_art(art_sql): for each statement in list order, DROP TABLE <art> (idempotent re-run) then execute it. The session-scoped volatile ART is then visible to the CTAS reading the feature view. Partition-awareness comes from the store filter living in the series-input view's scalar subquery (same pattern as FilterManager/fs-filter).
  • AI-feature gates: optimize_query is force-disabled for this type (art_sql is an opaque UAF invocation and the view EXPLAIN needs the ART materialised first); document_process_incremental skips the view-EXPLAIN and feeds the art_sql operator + art_layers into the column-documentation LLM context; get_process_info additively surfaces ART_SQL/ART_LAYERS.
  • Lineage traces through the ephemeral ART: _extract_series_art_table_names (lineage/lineage.py) parses TABLE_NAME(...) out of SERIES_SPEC/ART_SPEC (inert on non-UAF SQL), and record_process_lineage(art_sql=, art_layers=) rewrites the dangling feature_view → <ephemeral ART> edge into a two-hop path feature_view → <operator/layer node> → series_input_source (fail-open).
  • Backward-compatible: art_sql=None takes the classic 'denormalized view' path byte-for-byte; the 'denormalized view'/'tdstone2 view' branches are untouched.
  • Live-verified end-to-end on Vantage Cloud Lake (Vantage 20): ARIMA fit-metrics R_SQUARE/MAPE from ARTFITMETADATA, 2-partition FilterManager — registration, METADATA, both-partition ingestion (values matching the ART ground truth), idempotent run(force_compute=True), and lineage-through-ART all pass. See the notebook under notebook dev/uaf art view/. Creation-time lifecycle constraint (verified): a view referencing a volatile ART cannot be REPLACE VIEW-created unless the ART already exists in the session (Error 3807) — materialise the ART once before creating the feature view; run() re-materialises it per partition thereafter.

Added — fs-tdstone2 skill

  • New Claude Code skill fs-tdstone2 (a tdfs4ds skill) documenting the one-call bridge tdfs4ds.upload_tdstone2_scores(model) — registering a trained tdstone2 model's per-partition predictions / computed features as governed features (entity derived from the model's mapper; snapshot vs recomputable-view trade-off; the ML-model lineage the ca-algorithm skill walks backwards).

Tests

  • tests/test_uaf_art_view.py — DB-free unit tests for the 'uaf art view' helpers (_extract_art_name, _extract_series_art_table_names, _uaf_art_rewrite).

[0.3.1.142] - 2026-07-10

Changed (Claude Code fs-* skills — sync verified gotchas + TDFS4DS_TD_* credentials)

  • New shared reference .claude/skills/fs-shared/gotchas.md: verified against tdfs4ds 0.3.1.141 and a live Vantage 20.0 instance. Covers connection, the setup(database, if_exists='fail') signature (and why if_exists='replace' is destructive), the "a data domain has no table of its own" create-vs-select semantics, the three upload_features input shapes, grouped=True not being persisted, the business-date-defaults-to-now behaviour, temporal-MERGE idempotency, Teradata SQL traps (ORDER BY in from_query, LIKE [...] character classes, CHAR-padding, BEGIN/END(VALIDTIME), copy_to_sql type binding), lazy tdml.DataFrame, and catalog object-name / return-type quirks.
  • All ten fs-* skills: connection boilerplate now reads credentials from TDFS4DS_TD_HOST/USERNAME/PASSWORD/DATABASE (the current convention used by reconnect/serve) instead of the obsolete TDML_* vars; each skill now ends with a ## Gotchas section linking to ../fs-shared/gotchas.md.
  • fs-setup: setup(database=...) is required (the old no-arg call would TypeError); never if_exists='replace'; explicit create-vs-select data-domain rules with a fresh-store hedge pattern; LIKE ... ESCAPE cleanup SQL. The Vector Store / TeradataAI config block is retained.
  • fs-inspect: accurate catalog column comments (process catalog uses ENTITY_ID, real dataset-catalog columns, FEATURE_VIEW vs VIEW_NAME) and select_data_domain(...) instead of a direct DATA_DOMAIN = assignment.
  • fs-upload: new "three input shapes of upload_features" section (DataFrame / view-name / sql_query+target_view_name), the silent-None-on-failure warning, the always-schema-qualify rule, and the data-domain hedge in the code sample.

Documentation/skills only — no package code changed.

[0.3.1.141] - 2026-07-09

Added (Teradata Enterprise Vector Store — retry transient gateway 502/503/504 errors)

  • Motivation: live VCL2 testing repeatedly hit genuine transient 502 Bad Gateway responses from the Vector Store REST service (and one collection_status='create_failed' after an otherwise-clean-looking from_datasets() call) — pure infrastructure flakiness unrelated to any code/config issue, that resolved on a simple retry a few seconds later. These were previously classified correctly as "service down" by _reraise_if_service_down but required the caller to notice and manually retry.
  • Fix (agent/vector_store_teradata.py): new _is_transient_gateway_error() (detects a bare HTML 502/503/504 body, distinct from a genuine unreachable/unlicensed service) + _retry_if_transient_gateway_error() (retries for up to 60s, then falls through to the existing clear _SERVICE_DOWN_MSG). Wired into every REST-facing call site that showed this live: _lookup_store_info's CollectionManager.list(), build_teradata_index's from_datasets/ add_datasets calls, reset_teradata_store's from_datasets call, and _safe_destroy's destroy() call. _reraise_if_service_down now returns normally (leaving the caller's retry loop to handle it) for this third transient-exception class, alongside the existing "not in a ready state" 409 and per-query staging-object race.

Added (Teradata Enterprise Vector Store — collection description=)

  • build_teradata_index()/reset_teradata_store() now pass a fixed, human-readable description= per collection at creation time (_SOURCE_DESCRIPTIONS in agent/vector_store_teradata.py), visible via CollectionManager().list()'s collection_description field — makes the three collections self-documenting when inspected outside tdfs4ds (e.g. via a shared VCL2 instance with many unrelated collections). Verified live.

[0.3.1.140] - 2026-07-09

Fixed (Teradata Enterprise Vector Store — _inputdata view permanently referencing a garbage-collected ephemeral view)

  • Symptom: on a fresh VCL2 test deployment, build_teradata_index() created all three collections without error and collection_tdfs4ds_<source>_index (the embeddings table) held real, non-empty rows (13/16/73 for objects/sections/columns) — but every similarity_search call, and even a plain SELECT COUNT(*) against collectionV_tdfs4ds_<source> from a completely unrelated client (a teradata-mcp5 MCP session, a fresh teradataml session), failed identically with Error 3807 / TDML_2412: Object 'data_scientist.ml___frmqry_v_<id>' does not exist — every time, not transiently (3/3 retries failed identically; the referenced object was confirmed absent from DBC.TablesV).
  • Root cause, found by inspecting the actual DDL of the backing Teradata objects: collection_tdfs4ds_<source>_inputdata is itself a view, and its stored SHOW VIEW DDL hardcoded FROM "data_scientist"."ml___frmqry_v_<id>" — a short-lived materialized-query view that Vantage garbage-collects almost immediately. collectionV_tdfs4ds_<source> (the join view search reads through) depends on _inputdata, so it inherited the same permanent breakage. Traced this to tdfs4ds's own call site (not just the teradatagenai client library, as the 0.3.1.131 fix assumed): _from_datasets_kwargs() built its data= argument via tdml.DataFrame.from_query("SELECT ... FROM <permanent view>") — and confirmed live that DataFrame.from_query(...) itself sets _table_name to a freshly materialized ephemeral view at construction time, before teradatagenai is even called. teradatagenai then bakes whatever _table_name it receives permanently into _inputdata's DDL — so reading from a permanent source view (the 0.3.1.131 fix) was necessary but not sufficient; the from_query() wrapper itself was creating a doomed ephemeral view on every call.
  • Fix (agent/vector_store_teradata.py): _from_datasets_kwargs() now passes data= as a plain fully-qualified "schema.view" string naming the permanent FS_V_BUSINESS_DICTIONARY_* view directly — never a from_query() result — guaranteeing no ephemeral materialization ever happens. Matches teradatagenai's own documented simplest input shape (its from_datasets docstring's own Example 1 passes a plain table name string).
  • No row-level filtering of data= — by design, not an oversight: checked directly that neither from_datasets nor add_datasets exposes any filter/predicate/where-clause parameter. Considered and rejected pointing data= at a filtered subset for scoped/incremental reindexing: _inputdata is a live view (re-evaluated on every query, not a frozen snapshot), so a filtered data= would permanently narrow what the whole collection can ever return. Separately, _index's embeddings are keyed by TD_ID, which _inputdata (re-)assigns via ROW_NUMBER() OVER (ORDER BY <text column>) on every evaluation — a stable-looking key that is actually recomputed by row position, not a persistent identity. Always pass the entire, unfiltered source view.
  • Required recovery step for existing installs: collections created before this fix have the broken ephemeral-referencing _inputdata/collectionV_* DDL baked in permanently — an incremental add_datasets on top will NOT repair it. Call tdfs4ds.agent.vector_store_teradata.reset_teradata_store() (requires admin) once to destroy and recreate the collections using the fixed code path.

Fixed (Teradata Enterprise Vector Store — search_teradata_index silently returning 0 hits on genuine matches)

  • Symptom: after fixing the _inputdata/ephemeral-view bug above and confirming similarity_search() genuinely returns real, relevant hits when called directly, tdfs4ds.search_vector_index()/search_teradata_index() — the actual public entry point — still returned 0 rows for the same query against the same collection, with no error or warning logged.
  • Root cause: _parse_one()'s _TEXT_KEYS (("content", "page_content", "text", "document")) assumed a generic/Chroma-like hit shape. A real Teradata Vector Store hit is flattened differently: the text lives directly under the source's actual business-dictionary column name — business_description (objects/columns) or section_content (sections) — not any of the generic keys, and metadata (object_name, section_name, column_name, etc.) is flattened into the same dict rather than nested under a metadata key. Every hit's text therefore stayed empty, silently dropped by the if not text.strip(): return None guard — 5 genuine hits parsed down to 0 rows with no diagnostic trace.
  • Fix (agent/vector_store_teradata.py): added "business_description" and "section_content" to _TEXT_KEYS. Verified live: the same query that previously returned 0 rows through search_vector_index() now returns 8 real, relevant hits across objects and sections, matching what similarity_search() returns directly.

Known limitation (Teradata Enterprise Vector Store — build_teradata_index's reuse branch cannot pick up new rows via add_datasets; data_refresh=True investigated but found unreliable)

  • Symptom: calling build_teradata_index() a second time (collection already exists, e.g. after documenting one more table/column) fails with TDML_2412 ... Response Code: 400, Message: You can only add object names that do not exist in the collection — and this 400 is then misrouted through _reraise_if_service_down's generic "response code:" catch and re-raised as the unrelated _SERVICE_DOWN_MSG ("service is unreachable or not enabled"), even though the service is clearly up (the same collection had just been created moments earlier).
  • Root cause: add_datasets()'s real semantics are "attach a new source object not yet registered on the collection," not "resync/upsert rows from an already-attached source." _from_datasets_kwargs() always names the SAME FS_V_BUSINESS_DICTIONARY_* view on every call, so any second call through the reuse branch is guaranteed to hit this 400 once the source is already attached — i.e. new business-dictionary rows are NOT currently picked up by a second build_teradata_index() call.
  • Investigated and rejected as a fix: Collection.update(data_refresh=True) — the documented mechanism ("refresh/regenerate vectors for the collection using existing backing/source tables," per its own docstring's Example 8) — looked like the correct replacement and succeeded once in isolated live testing (a new business-dictionary row correctly appeared in _index/_inputdata/ collectionV_*, all in lockstep, 74/74/74). However it then failed consistently on every subsequent attempt — with a fresh TeradataVectorStore instance, and even with a genuinely new, never-indexed row present in the source view — with "At least one of collection_index, collection_parameters, collection_description, or new_collection_name must be provided for update". This looks like an intermittent/stateful server-side issue in teradatagenai, not something reproducibly fixable from the client side, so it was not adopted as the fix — shipping it would trade one known-broken path for a differently unreliable one. build_teradata_index's reuse branch remains on add_datasets() (pre-existing behavior, known limitation). A real fix needs a dedicated follow-up once data_refresh's actual failure conditions are understood.

[0.3.1.139] - 2026-07-08

Fixed (Teradata Enterprise Vector Store — add_datasets() silently corrupting collections; "no exception" ≠ "write succeeded")

  • Symptom: a live deployment showed build_teradata_index logging "indexed 'tdfs4ds_objects' (reused existing collection)" (no exception raised) while CollectionManager().list() showed the SAME collection's collection_status as 'update_failed'. Searching for a document's own verbatim indexed text (search_vector_index(<verbatim text>, top_k=5)) returned zero hits at the same time "indexed successfully" was logged — proving the collections held no usable data despite the success log. A timeline pinned the corruption to build_teradata_index itself: all three stores were collection_status='ready' at 16:46, then update_failed/create_failed at 16:49 — a ~3-minute window matching exactly one build_teradata_index run.
  • Leading root cause: Collection.add_datasets()/update() accept an update_style parameter ("MINOR"/"MAJOR") that tdfs4ds never passed. Checked the teradatagenai client source directly: every single documented usage example of update()/add_datasets()/delete_documents() in the library's own docstrings explicitly passes update_style="MINOR" — none omit it, and no default is documented in the parameter metadata. This is a strong signal that omitting it is an unsupported code path server-side. Fixed: build_teradata_index's add_datasets call now passes update_style="MINOR" (routine incremental upsert), and a MAJOR finalize pass (collection.update(update_style="MAJOR")) runs once per touched store after the per-store loop completes — matching the client library's intended MINOR-for-churn / MAJOR-for-periodic-consolidation split.
  • Secondary, defense-in-depth fix: add_datasets()/from_datasets() are asynchronous — a call returning without an exception only means "the request was accepted," not "the write succeeded." Added _verify_write_succeeded() (agent/vector_store_teradata.py), which polls the real collection_status (via the generalized _lookup_store_info(), extracted from the existing _lookup_store_database() — which previously discarded this field) until it reports 'ready', called right after every write in build_teradata_index, before logging success. Raises clearly on timeout or if collection_status reports a terminal *_failed state instead.
  • New failed-state detection: _is_failed_status() matches any ..._failed collection_status (confirmed live values: update_failed, create_failed). Both build_teradata_index (pre-write) and search_teradata_index (pre-search) now check this first and raise/log a clear, actionable message pointing at reset_teradata_store() — instead of silently waiting/retrying into a collection that can never recover on its own, or (for search) silently skipping it behind an opaque "query failed (... not ready for asks ...), skipping" warning.
  • Usability fix: search_vector_index() now accepts k= as an alias for top_k= (the near-universal convention in the surrounding ecosystem — Chroma, FAISS, LangChain retrievers all use k) instead of raising a raw TypeError: unexpected keyword argument 'k'.

[0.3.1.138] - 2026-07-08

Added (proactive readiness wait before every Vector Store write)

  • Motivation: 0.3.1.137's _retry_if_not_ready fixed the classification of the transient "not in a ready state" 409 (async Collection writes can still be busy when the next write targets them) but only retried by repeating the write itself, capped at 120s — not enough for a slower collection (tdfs4ds_columns, in a real deployment, kept 409ing well past that window).
  • Fix (agent/vector_store_teradata.py): added _wait_until_collection_ready(collection, name, timeout_seconds=1800.0, poll_interval=5.0), which polls the read-only Collection.get_details() every 5s until it succeeds — treating any exception (including the 409) as "still processing a prior async operation" — before the real write is attempted at all. Unlike retrying the write, a read-only probe can't itself add server-side contention. Raises the last observed exception if the collection is still not ready after 30 minutes (a genuinely stuck/failed operation surfaces clearly instead of hanging forever).
  • Wired into both write paths in build_teradata_index(): before the initial from_datasets() create attempt (only when the collection is confirmed to already exist via _lookup_store_database — a genuinely nonexistent collection isn't "not ready", it's "not there", so the wait is skipped there), and before the add_datasets() reuse-branch call. The existing _retry_if_not_ready write-retry stays in place underneath as a belt-and-suspenders fallback.

[0.3.1.137] - 2026-07-08

Fixed (Teradata Enterprise Vector Store — transient "not in a ready state" 409 misclassified as service-down)

  • Symptom: build_teradata_index() (real deployment, after PAT auth started working per 0.3.1.136) failed indexing the third of three stores with Teradata Vector Store service is unreachable or not enabled ... Detail: [Teradata][teradataml](TDML_2412) Failed to execute collection. Response Code: 409, Message: Collection tdfs4ds_columns cannot be updated as it is not in a ready state. — even though the service was clearly reachable (the first two stores had just been indexed successfully in the same run).
  • Root cause: every Collection write operation (create/update/ destroy) on the Enterprise Vector Store is asynchronous — the earlier logs show "Collection update request is accepted and in progress" for each store, meaning the collection can still be busy processing that request when the next store's operation targets it (or when a rapid create→reuse retry targets the same collection). Calling add_datasets() while the collection is still busy raises a 409 whose message contains both "not in a ready state" AND "Response Code:" — the latter is exactly the substring _reraise_if_service_down used to detect a down/unlicensed service, so this transient, retryable condition was being raised as a permanent RuntimeError telling the user to check licensing.
  • Fix (agent/vector_store_teradata.py): added _is_not_ready_error(exc), checked first (before the generic "response code:" branch) inside _reraise_if_service_down so this specific 409 shape is never misclassified — _reraise_if_service_down now returns normally for it, leaving it to the caller to retry. Added _retry_if_not_ready(fn, name, timeout_seconds=120.0, poll_interval=5.0), which retries a callable while it keeps raising this transient 409 (up to the timeout, then re-raises), and wrapped the add_datasets() call in build_teradata_index()'s reuse path with it.

[0.3.1.136] - 2026-07-08

Added (tdfs4ds now manages teradataml.set_auth_token(...) automatically)

  • Motivation: VECTOR_STORE_BACKEND='teradata' authenticates against the Vantage Enterprise Vector Store REST API via teradataml.set_auth_token(...) — a separate auth channel from the regular tdml.create_context() DB connection. Previously tdfs4ds never called this; users had to call it themselves, and forgetting to do so produced the _NOT_CONFIGURED_MSG error (added in 0.3.1.135).
  • New module tdfs4ds.utils.vector_store_auth.configure_vector_store_auth() (utils/vector_store_auth.py): reads new TDFS4DS_VECTOR_STORE_AUTH_* pass-through environment variables (same tier as TDFS4DS_TD_*) and dispatches to teradataml.set_auth_token(...), trying three mechanisms in priority order (first fully-configured group wins):
  • PAT — BASE_URL + PAT_TOKEN + PEM_FILE (+ optional USERNAME)
  • Basic — BASE_URL + USERNAME + PASSWORD
  • OAuth — BASE_URL + CLIENT_ID

Returns the mechanism name used (or None), and never raises — any set_auth_token failure is caught and logged as a non-fatal warning. Never logs secret values (pat_token, pem_file path, password, client_id), only base_url/username. - Wired in at every point a DB context is already established: lifecycle.setup(), lifecycle.connect() (both log which mechanism was used, or that none was configured), and serve/bootstrap.py's _post_connect_setup() (called before the vector index build, so the build succeeds once auth is configured). - config.py: the six new vars added to _PASSTHROUGH_ENV_VARS, matching exactly how TDFS4DS_TD_* is handled (read directly via os.environ, .env support via the existing passthrough re-export mechanism). - _NOT_CONFIGURED_MSG (agent/vector_store_teradata.py) updated to describe the new automatic behavior and the three env var groups, since reaching this error now means either no TDFS4DS_VECTOR_STORE_AUTH_* vars were set, or set_auth_token itself failed (check the connect/setup log for a "Vector Store auth ... failed" warning). - JWT and Keycloak (two other set_auth_token mechanisms) are intentionally out of scope — only PAT/Basic/OAuth are implemented.

[0.3.1.135] - 2026-07-08

Fixed (misleading "service unreachable" message when the Vector Store client isn't authenticated)

  • Symptom: build_teradata_index/build_vector_index failed with Teradata Vector Store service is unreachable or not enabled ... check that the Enterprise Vector Store feature is licensed/activated, with Detail: Invalid URL 'None/api/v2/collections?page_size=100000': No scheme supplied — even though the service itself was fine.
  • Root cause: _reraise_if_service_down treated every requests.exceptions.RequestException as "service down/unlicensed." But Invalid URL 'None/...' is requests.exceptions.MissingSchema/InvalidURL — a client-side failure: the Vector Store REST client's base URL is None because teradataml.set_auth_token(...) (or equivalent session auth) was never called before using the backend. tdfs4ds does not call set_auth_token on the user's behalf (it's a connection/auth concern outside its scope), so this is a setup-order gap, not a licensing problem — but the error message pointed the user at the wrong fix.
  • Fix: _reraise_if_service_down (agent/vector_store_teradata.py) now checks for MissingSchema/InvalidURL (and the equivalent wrapped TeradataMlException message shape, "no scheme supplied"/"invalid url") before the generic RequestException check, raising a new, distinct _NOT_CONFIGURED_MSG RuntimeError that tells the user to call set_auth_token(...) — instead of the misleading "unreachable/unlicensed" message.

[0.3.1.134] - 2026-07-08

Fixed (reset_teradata_store — wait for async destroy() before recreating)

  • Symptom: even with an explicit destroy() that reported success ("Collection destroy request is accepted and in progress"), a following from_datasets() call could still raise TeradataGenAIException: Collection with name '...' already exists. — from Collection.create()'s own self.exists check.
  • Root cause: Collection.destroy() is asynchronous — it returns as soon as the request is accepted, not once the collection is actually gone server-side. reset_teradata_store() (added in 0.3.1.133) called _safe_destroy() then immediately from_datasets(), racing the still-in- progress destroy.
  • Fix: added _wait_until_destroyed(name, timeout_seconds=60.0) (agent/vector_store_teradata.py), which polls _lookup_store_database() until the collection is confirmed gone (or the timeout elapses, logged as a warning rather than raised — the subsequent from_datasets() call will surface a clear error itself if the collection is genuinely still there). reset_teradata_store() now calls this between _safe_destroy() and the recreate.

[0.3.1.133] - 2026-07-08

Fixed (Teradata Enterprise Vector Store — clear gateway errors + drop the admin-only destroy from routine rebuilds)

  • Symptom 1: tdfs4ds.rebuild_vector_indexing(force_rebuild=True) (and any build_vector_index/build_teradata_index call) could crash with a raw teradataml.common.exceptions.TeradataMlException (TDML_2412) wrapping an HTML 502 Bad Gateway body, several frames deep inside langchain_teradata/teradatagenai internals (Collection.exists() → CollectionManager().list()), instead of the clear "Vector Store service is unreachable or not enabled" message tdfs4ds already raises elsewhere.
  • Symptom 2 (surfaced once the gateway/auth issue above was resolved): the same call then failed with TeradataGenAIException: Collection with name 'tdfs4ds_objects' already exists. — raised by Collection.create()'s own authoritative self.exists check, immediately after build_teradata_index() had just attempted to destroy that same collection.
  • Root cause (symptom 1): the clear-message classification (_raise_service_unavailable, detecting a transport-level failure or an HTML/status body embedded in a TeradataMlException) previously lived only inside _lookup_store_database(). The actual TeradataVectorStore.from_datasets()/add_datasets() calls in build_teradata_index() had no exception handling at all, so a gateway error from those calls (which internally re-enter the same Collection.exists() check) bypassed the classification and propagated raw.
  • Root cause (symptom 2): build_teradata_index(force_rebuild=True) called _safe_destroy() before every rebuild, which only invoked destroy() when Collection.status() reported a non-empty operation history — but status() reports async create/destroy/update operations, not current existence, so an existing collection with no reported history was wrongly concluded "nothing to destroy" and left in place. Separately, Collection.destroy() requires admin privileges ("Only admin users can use this method" per its own docstring); any destroy failure (e.g. a non-admin session) was caught and only logged as a warning, silently leaving the stale collection for the following from_datasets() call to collide with.
  • Fix:
  • Extracted the service-down classification into a shared _reraise_if_service_down(exc) helper (agent/vector_store_teradata.py), reused by _lookup_store_database() and wrapped around every from_datasets()/add_datasets() call site.
  • build_teradata_index() no longer destroys anything. It always attaches-or-creates via from_datasets()/add_datasets() (which creates the collection on first use and upserts/appends rows keyed on ID afterward — no per-id duplication), sidestepping the admin requirement entirely. force_rebuild is now purely informational for logging; both branches behave identically. This means rows whose source business-dictionary entry was deleted are not purged by this call (no admin-free per-id deletion API is available yet — Collection.get_ids() is currently an unimplemented stub upstream).
  • Added reset_teradata_store(sources=None) — an explicit, admin-only destroy-and-recreate for when a true clean wipe (stale-row purge) is needed and the connected user has admin rights. The routine build_vector_index()/rebuild_vector_indexing() path deliberately does not call it, so non-admin sessions keep working.
  • rebuild_vector_indexing()'s and build_vector_index()'s docstrings updated to describe the Teradata backend's reuse-only behavior (the Chroma backend's force_rebuild=True still deletes and rebuilds from scratch, unchanged).

[0.3.1.132] - 2026-07-08

Fixed (business-dictionary views not (re)created on installs from an older tdfs4ds version)

  • Root cause: the three FS_V_BUSINESS_DICTIONARY_OBJECTS/_SECTIONS/_COLUMNS views (introduced in 0.3.1.131 for the Teradata Enterprise Vector Store) were only created inside documentation_tables_creation(), which connect(create_if_missing=True) only calls when the underlying business-dictionary tables are detected as missing. On any install where those tables were created by an older tdfs4ds version, connect() saw the tables already present, skipped the creation branch entirely, and the views were never materialized — causing Error 3807: FS_V_BUSINESS_DICTIONARY_OBJECTS does not exist the first time the Teradata vector store backend queried them.
  • Fix: extracted the view-creation loop out of documentation_tables_creation() into its own documentation_business_dictionary_view_creation() (genai/documentation.py, re-exported from tdfs4ds.genai). connect() now calls it unconditionally on every connect, right alongside the existing unconditional feature_store_catalog_view_creation() / process_store_catalog_view_creation() refresh in the "Refreshing catalog views..." step (lifecycle.py) — the same self-healing pattern those two already used, rather than depending on a table-existence probe to infer whether the views need (re)creating. REPLACE VIEW is idempotent, so this adds no meaningful cost and guarantees the views exist after any connect(), regardless of what version created the underlying tables. setup() was already covered (it always calls documentation_tables_creation(), which now includes the view creation via this refactor).

[0.3.1.131] - 2026-07-07

Fixed (Teradata Enterprise Vector Store — real fix for staging-database bug)

  • Root cause found: target_database (0.3.1.129/130) only tells the Vantage REST API where to register the FINAL collection's metadata — it does NOT control where langchain_teradata's from_documents()/add_documents() (LangChain Document-list input) stage their intermediate data. That path internally calls teradataml.DataFrame.from_dict(..., persist=True) / DataFrame.materialize(), which takes no database argument at all and — per teradataml's own docstring — may materialize as a view or a volatile table, "decided and taken care by teradataml" internally. A volatile table is session-scoped and invisible to the Vector Store's separate REST session, reproducing TDML_2412: Object ... does not exist regardless of target_database.
  • Fix: build_teradata_index() now calls TeradataVectorStore.from_datasets()/ add_datasets() instead of from_documents()/add_documents(), reading directly from three new permanent CURRENT VALIDTIME views — FS_V_BUSINESS_DICTIONARY_OBJECTS/ _SECTIONS/_COLUMNS (documentation_tables_creation(), tdfs4ds.DOCUMENTATION_BUSINESS_DICTIONARY_*_VIEW) — via plain SQL + tdml.DataFrame.from_query(). No Document conversion, no pandas round-trip, no ephemeral/volatile/temporary table of any kind — the vector store reads the same permanent business-dictionary table every other part of tdfs4ds treats as the source of truth. The views bake in LOCKING ROW FOR ACCESS, a synthesized ID column (matching the existing Chroma-backend doc-id convention), and a blank/NULL-description filter (mirroring _objects_to_docs/etc.'s if not text.strip(): continue guard).

[0.3.1.130] - 2026-07-07

Fixed (Teradata Enterprise Vector Store — orphan prevention + service-down handling)

  • target_database on every TeradataVectorStore call: the incremental-attach path in build_teradata_index() and search_teradata_index() now pass target_database (scoped to tdfs4ds.SCHEMA) alongside the creation path fixed in 0.3.1.129, so attach/search are explicit about which database they resolve against instead of relying on the Vantage session's ambient current database.
  • Pre-destroy visibility via CollectionManager: new _lookup_store_database(name) helper (tdfs4ds/agent/vector_store_teradata.py) calls teradatagenai's CollectionManager().list() to discover which database a same-named store actually lives in before attaching. _safe_destroy now destroys the store at its discovered database (not blindly at tdfs4ds.SCHEMA) and logs a warning when it's found somewhere unexpected — closing an orphan risk where a stale store from before the target_database fix would never be reached/destroyed on force_rebuild, leaving a duplicate behind. build_teradata_index also logs the intended target database once per run.
  • Service-down handling: _lookup_store_database distinguishes a genuinely unreachable or unlicensed Vector Store service (transport-level requests errors, or a gateway 404/502 HTML body embedded in a TeradataMlException) from real in-service errors, and raises one clear RuntimeError ("...unreachable or not enabled... check licensing/activation...") instead of a raw requests traceback or an HTML-in-exception message. This propagates correctly through build_teradata_index's incremental-attach path (previously it would have been misclassified as "store doesn't exist, rebuilding").

[0.3.1.129] - 2026-07-06

Fixed (Teradata Enterprise Vector Store)

  • target_database on vector store creation: TeradataVectorStore.from_documents(...) in build_teradata_index() now passes target_database=tdfs4ds.SCHEMA explicitly. Without it, the Enterprise Vector Store staged its intermediate object in the Vantage session's ambient "current database" (the login user's default DB), not tdfs4ds.SCHEMA / the teradataml context's database=/temp_database_name= — causing TDML_2412: Object ... does not exist when those differ.

[0.3.1.116] - 2026-06-22

Fixed (eval + dashboard + agent + API)

  • Eval progress bar: shows elapsed time and estimated remaining time during SSE run (⏳ 2/44 — 0:12 elapsed, ~0:45 remaining; done shows total duration)
  • Eval report width: #ev-report now has width: 100% so the report fills the full panel
  • Judge — EXPLAIN grounding: _verified_names_from_context now extracts source_columns table/column names and calculation_sql formula tokens from the trace, so EXPLAIN-derived facts (e.g. CLAIMS_RAW, quantity) are verified rather than treated as fabrications; S05-style all-1 misscores fixed
  • Judge — EXPLAIN prompt rule: added conditional "EXPLAIN GROUNDING RULE" clause to _judge_turn prompt for EXPLAIN turns
  • Judge — DEFINITION completeness: added "DEFINITION COMPLETENESS RULE" clause so judge does not penalise for omitting navigation metadata (domain, entity_name) from the answer body
  • Visualization — radar chart: legend moved below chart with wider bottom margin (height=520, margin b=140, horizontal y=-0.22); labels no longer truncated to 1 character
  • Visualization — bar chart: increased height and bottom margin (height=420, margin b=100, y=-0.28); metric labels no longer truncated
  • Dashboard run history: x-axis now shows view names (vw_aggregate_features) instead of process UUIDs; _dash_run_history_figs accepts pid_to_view mapping from caller
  • Query optimizer API: new POST /v1/optimize endpoint — accepts {"sql": "...", "user_request": "..."}, returns {optimized_sql, optimization_strategy, token_consumption}; /v1/chat/completions for tdfs4ds-query-optimizer model now includes optimized and score_delta in the usage block
  • Domain typo: node_detect_domain now fuzzy-matches object_name against known data domains when detect_data_domain_for_object returns nothing (e.g. "anomalu" → "anomaly"); _build_context_update normalises a typo domain against the existing graph's known domains as a last-resort guard, preventing duplicate domain nodes in the knowledge graph visualization

[0.3.1.115] - 2026-06-22

Fixed (admin web shell — 13 UI rendering issues)

Multi-fix release addressing visual and functional bugs in the Tera admin shell:

Chatbot: removed the "Loading…" spinner flash when the Chatbot nav item is first selected (the iframe replaces content immediately so no spinner is needed).

Query Optimization: - "Current SQL" left column now shows the actual view SQL (PROCESS_SQL key, not the missing SQL key). - "Process detail" card renders styled HTML directly (badges, warnings, SQL block) instead of passing HTML-injected markdown through marked() which escaped the tags. - "Process context" footer (_Process context — Entity: … | Features: …) is now a styled light-grey info bar instead of raw italic text. - Score Summary and other ## sections with GFM pipe-tables are now converted to real <table> HTML before rendering.

Evaluation: - "Generate scenarios" button now shows an animated orange progress bar while the request is in flight and re-enables on completion. - By-skill score table replaced with a grouped bar chart (0–5 scale, one bar per metric per skill) — more readable at a glance. - Radar chart legend moved to the right side (orientation="v", x=1.02) so it no longer overlaps the polar area. - Scenario list table header <th> cells now carry color:#fff; background:#00233C inline — previously only the <tr> had these styles, which some CSS resets overrode. - Agent answers and judge rationale in the HTML report are now rendered as HTML (bold, inline code, bullet lists) instead of raw escaped text.

Dashboard: - "Run history" charts: follow_up_report_many().to_pandas() moves PROCESS_ID to the pandas index (it is the PRIMARY INDEX); added .reset_index() so the column is available for grouping — previously all queries returned empty. - "Compute row counts" button now shows an animated progress bar while the (potentially long-running) per-table COUNT queries are executing.

Layout: - All panels no longer shrink horizontally when content loads; #td-main now has width: calc(100% - 280px); box-sizing: border-box and .td-panel has width: 100% — panels were previously relying on default width: auto (shrink-to-fit), which collapsed when wide content was injected. - Added @keyframes td-slide animation to td.css for the progress bars.

[0.3.1.114] - 2026-06-21

Fixed (context knowledge graph — orphan feature nodes on a multi-entity SUMMARY)

Follow-up to 0.3.1.113. Asking "what are the available features in the claims anomaly detection data domain?" still rendered some features (e.g. pharmacy_lifetime_avg_amount) as isolated nodes — no edge to their entity or data domain — even though the catalog has both. Invariant: a feature must ALWAYS link to an entity AND a data domain.

Root cause: _build_context_graph_pyvis (agent/graph.py) renders features in three places. A feature_list pre-seed loop minted each feature's node id from _focus_entity (EMPTY for a multi-entity SUMMARY — node_skill_summary only sets it when there's exactly one entity) and no domain, producing a blank-identity node feature___NODOM____NOENT__<name> with zero edges. The real render loops mint the SAME feature with its true (entity, domain) triplet → a different node id, correctly linked. The node-merge net never merges two Feature nodes with different ids (triplet identity is intentional, so same-name/different-entity features stay distinct), so the blank-id pre-seed node survived as the floating orphan.

Fix: the pre-seed loop now recovers each feature's REAL (entity, domain) from the accumulated graph (features keyed by the DOMAIN::ENTITY::NAME triplet) and mints one node per entity variant, each linked to its entity (and the entity to its domain) — so the node ids collide with the real loops (one node, properly linked) and all same-name/different-entity/domain variants are preserved. A feature with no accumulated entry yet (pure group-pronoun-freshness) is anchored to the domain node so it is never isolated. Shared edge logic factored into a _link_feature_node helper.

Defense in depth: skill_summary's cg_features (agent/consumer_agent.py) now stamps each feature's domain (active tdfs4ds.DATA_DOMAIN), and node_skill_summary passes the resolved summary domain (not the possibly-stale resolved_data_domain, and never the unscoped "all") to _build_context_update — so features are never domain-less in the accumulated graph.

[0.3.1.113] - 2026-06-21

Fixed (context knowledge graph — entity-aware feature↔dataset linking)

Two bugs in the chatbot "Context knowledge graph" (the pyvis viz built from accumulated_context_graph), both visible after a SUMMARY question ("what are the available features in a data domain?"):

  • Orphan single features. A feature like unique_drugs rendered as an isolated node — no edge to its entity, dataset, or the data-domain node.
  • Dataset linked to the wrong same-named feature. ds_pharmacy_stats (grain pharmacy_id) drew dataset→feature edges to BOTH the pharmacy-entity AND the provider-entity features of the same name (total_amount, avg_amount, …).

Root cause (single): _feature_dataset_names_map() (agent/consumer_agent.py) keyed its feature→dataset map on FEATURE_NAME alone, dropping the entity dimension. A feature's identity is the pair (FEATURE_NAME, ENTITY_NAME), so a name-only key conflated two distinct features and gave each the union of every dataset exposing that name.

Fix: the dataset _FEATURES table carries FEATURE_ID (the feature's identity); the feature catalog (FS_FEATURE_CATALOG, keyed by FEATURE_ID BIGINT) resolves it to the authoritative ENTITY_NAME/DATA_DOMAIN. _feature_dataset_names_map() now joins the two on FEATURE_ID (in pandas over single materializations — no per-row DB query) and returns two maps: by_entity keyed on (ENTITY_UPPER, FEATURE_NAME_UPPER) (primary) and a by_name fallback for features whose FEATURE_ID has no catalog match (entity-less / cross-domain). skill_summary looks up the (entity, name) key first, falling back to name-only only when the feature has no entity — so a dataset links only to the entity it actually exposes, and entity-less features stay connected. Per-turn memoization and fail-open behavior unchanged.

[0.3.1.112] - 2026-06-21

Fixed (reverse-proxy path prefix — Gradio chatbot gradio_api URLs 404'd behind /tdfs4ds2)

Follow-up to 0.3.1.111. The FastAPI shell + /ui/* routes resolved correctly under the prefix, but the Gradio chatbot did not: its page emitted browser-facing asset/API URLs as root + api_prefix (e.g. /gradio_api/queue/join) with no prefix, because that root comes from the mount's root_path, not from the ASGI scope the _strip_prefix middleware rewrites. So the browser called https://…/gradio_api/... (un-prefixed) → nginx had no matching location → 404. (0.3.1.111 mounted Gradio plainly at /chatbot with no root_path, which is what caused this.)

Fix, both in serve/app.py:

  • Mount Gradio at the full prefixed path with a matching root_path: gr.mount_gradio_app(app, blocks, path=root_path+"/chatbot", root_path=root_path+"/chatbot"). In Gradio 6.x, root_path governs both the browser-facing root URL and the inner ASGI app's own route matching — they must be equal or every Gradio route 404s (mounting plainly OR passing only root_path both break). With them equal, Gradio emits the prefixed root (…/tdfs4ds2/chatbot) so the browser calls …/tdfs4ds2/chatbot/gradio_api/....
  • The _strip_prefix middleware now SKIPS the Gradio mount path ({prefix}/chatbot) — Gradio owns the full prefixed path and matches it directly; only the prefix-less shell//ui/*//v1/* routes are stripped. The mount path is stashed on app.state.td_chatbot_mount.

The iframe src="chatbot" is relative, so it resolves under <base href="/tdfs4ds2/"> to the prefixed mount automatically. No nginx changes needed.

Verified via TestClient in all three modes (prefix forwarded unstripped / stripped / root): shell, /ui/*, static, and /chatbot/config (with root = …/tdfs4ds2/chatbot) all resolve.

[0.3.1.111] - 2026-06-21

Fixed (reverse-proxy path prefix — landing page 404'd behind /tdfs4ds2)

The admin UI returned {"detail":"Not Found"} when served behind a reverse proxy under a path prefix (e.g. https://api-dmproject.myddns.me/tdfs4ds2). Two causes, both fixed in serve/app.py:

  • The app passed root_path to the FastAPI constructor, which (a) does not make routes match a forwarded-but-unstripped prefix and (b) broke the /ui/static StaticFiles Mount and the Gradio /chatbot mount. Removed the constructor root_path.
  • The prefix is now handled by a per-request _strip_prefix ASGI middleware: when the proxy forwards /tdfs4ds2/... unstripped, it strips the prefix so the (prefix-less) routes + mounts match, and stashes the external prefix on request.state so the shell injects the correct <base href="/tdfs4ds2/">. Works whether or not the proxy strips the prefix, and at root.
  • Gradio is mounted plainly at /chatbot (no root_path= kwarg) — the middleware already handles the external prefix; passing both made Gradio's internal routes 404.

Verified: shell, CSS/JS, /ui/*, /health and /chatbot all resolve under /tdfs4ds2/..., under a stripped prefix, and at the root.

[0.3.1.110] - 2026-06-21

Fixed / Added (triplet identity, judge, bidirectional lineage, admin UX, resilient startup)

  • Context graph respects the feature TRIPLET (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME) (agent/graph.py). _build_context_graph_pyvis keys feature nodes by the full triplet (_feat_node_id) and entity nodes by (domain, entity), so the same feature name under two entities (e.g. amount[provider_id] vs amount[pharmacy_id]) renders as distinct nodes (same label) instead of collapsing into one. The post-pass dedup no longer merges Feature/Entity nodes by label. _feat_cache_key now includes the active domain (DOMAIN::ENTITY::NAME) — since the domain can change mid-conversation, a cached feature from domain A is never served for domain B; _feat_cache_lookup stays back-compatible with legacy keys.
  • Within-domain entity disambiguation in name-only skills (agent/consumer_agent.py): skill_usage and skill_dataset constrain by the hinted/named entity (or return the ambiguous_feature clarification) instead of silently taking the first row; multi-feature freshness fans out per (entity, process) so a name under two entities reports both.
  • LLM-as-judge no longer scores correct counts as fabrication (agent/evaluation.py). _expected_facts for DATA_DOMAINS / SUMMARY now carries the real feature_count and feature_count_by_entity (triplet-deduped); the judge registers those numbers as fact-hits and the prompt states catalog-derived counts are real. Fixes the S17 class of straight-1s scoring.
  • Bidirectional, object-level lineage over the persisted graph: skill_lineage's primary path now positions the resolved view in FS_LINEAGE_GRAPH via _position_object_from_graph (both get_ancestors + get_descendants), so "what depends on X?" (downstream) works, not just upstream. A _lineage_direction(question) signal (EN/FR) selects upstream/downstream/both.
  • Admin UI: panels are now persistent (each rendered once into its own container, toggled by display) so switching panels no longer destroys content — the chatbot iframe is never reloaded, so conversation survives. The Evaluation panel regained skills selection, scenario table view, export, import (replace/append) and gained live per-turn SSE progress plus a radar chart + color-coded per-skill table (serve/static/app.js, serve/panels.py, serve/app.py, _ev_radar_chart_standalone/_ev_score_table_standalone in agent/chatbot.py).
  • Resilient startup + in-app Connect (serve/bootstrap.py): bootstrap() no longer raises when Teradata is unreachable — the app boots disconnected, /health reports connected, and a Connect page (top-bar Account icon) lets the user enter credentials and connect at runtime (POST /ui/connect → connect_teradata). Data panels degrade gracefully when disconnected; a status dot reflects the live state.
  • New packaged skill eval-inspect (tdfs4ds/skills/eval-inspect/): read-only stdlib query tool for an eval results JSON — by scenario id, worst N turns, intent mismatches, per-skill aggregates (with judge rationale).

[0.3.1.108] - 2026-06-20

Changed (admin app — Gradio only for the chatbot; custom Teradata-styled web shell for everything else)

The Gradio tabbed admin app (launch_chatbot_app) is replaced by a server-rendered HTML/JS admin web shell served by the FastAPI app, reproducing the Teradata "Tera" console look. Gradio now renders only the conversational chatbot, embedded in an iframe at /chatbot. The whole UI runs on TDFS4DS_SERVE_PORT. The app supports a reverse-proxy path prefix (TDFS4DS_SERVE_ROOT_PATH, e.g. /tdfs4ds2), and for backward compatibility the standalone Gradio chatbot is also still served on its own port (TDFS4DS_SERVE_CHATBOT, default on; TDFS4DS_SERVE_CHATBOT_PORT, default 7860).

  • tdfs4ds/serve/app.py — _add_admin_ui(app) mounts the chat-only Gradio Blocks at /chatbot (gr.mount_gradio_app), serves the shell at /, static assets at /ui/static, and /ui/* JSON/HTML endpoints for the Dashboard, Lineage, Query optimization, Evaluation, Parameters and Skills panels.
  • tdfs4ds/serve/panels.py (new) — Gradio-free panel logic returning JSON/HTML; delegates to the existing module-level builders in agent/chatbot.py.
  • tdfs4ds/serve/templates/index.html, serve/static/td.css, serve/static/app.js (new) — the Teradata-styled shell (top bar + teradata. wordmark, ~280px left rail with lavender active pill + BETA badges, per-panel content area).
  • agent/chatbot.py — build_chat_blocks() builds the embeddable chat-only Blocks (Tera hero + rounded-pill input + suggestion chips); _build_chat_ui(with_skills_pane=False) omits the skills column (skills + Parameters move to the shell). launch_chatbot / launch_chatbot_with_index are unchanged.
  • Path-prefix support — set TDFS4DS_SERVE_ROOT_PATH (e.g. /tdfs4ds2); create_app passes it to FastAPI root_path + Gradio's mount, and the shell injects a <base href> so every asset/fetch/iframe URL (relative on the client) resolves under the prefix. uvicorn runs with proxy_headers=True. Example target: https://api-dmproject.myddns.me/tdfs4ds2.
  • Deployment — Dockerfile, docker-compose.yml, .env.example keep ports 8000 (UI + API), 7860 (standalone chatbot) and 8080 (docs). Run with python -m tdfs4ds.serve.

[0.3.1.99] - 2026-06-17

Fixed (consumer agent — compound EXPLAIN+ALGORITHM question routed to EXPLAIN only when MCP on; invented STO SQL syntax)

Two issues with the ALGORITHM skill when MCP is enabled:

1. Compound question routed to EXPLAIN only. A question like "How is ml_risk_score computed? What algorithm is behind it?" has both EXPLAIN signals ("how is … computed") and ALGORITHM signals ("algorithm"). With MCP enabled, the classifier anchored on the EXPLAIN shape and routed to the EXPLAIN node — the user got the SQL formula explanation but not the ML model details.

  • Routing guard (_node_classify_impl in agent/graph.py): when MCP is on and the classifier returns EXPLAIN, re-route to ALGORITHM if _looks_like_algorithm_question also matches. Symmetric with the MCP-off guard that routes to ALGORITHM_UNAVAILABLE.
  • Dual-intent execution (node_skill_algorithm): detects both signals (_EXPLAIN_SHAPE_RE
  • _looks_like_algorithm_question) and runs skill_explain first when both are present, prepending its answer above the ALGORITHM answer with a --- divider. Column sources from the EXPLAIN pass are persisted to resolved_column_sources so EXPLAIN→DEFINITION drill-down works on the next turn.

2. Fabricated CALL sto.sto(...) SQL syntax. When asked about the STO invocation or how to build a similar model, the LLM reconstructed a CALL sto.sto(PARTITION BY ..., SCRIPT_COMMAND '...', RETURNS (...)) — syntax that does not exist in Teradata. The real invocation is FROM Script(ON ... PARTITION BY ... ORDER BY ... SCRIPT_COMMAND(...) RETURNS(...) CHARSET('LATIN')) AS d inside a REPLACE VIEW.

Root cause: _algorithm_system_prompt included parsed STO fields (partition_by, order_by, script_command, returns, charset) as JSON — the LLM had reconstruction material and invented a plausible-looking but wrong call syntax.

  • _raw_ddl anti-fabrication guard (agent/consumer_agent.py): skill_algorithm stores the raw training STO view DDL as sto_invocation["_raw_ddl"]. _algorithm_system_prompt pops _raw_ddl before JSON serialisation (removing the parsed fields as reconstruction material) and injects the raw DDL as a clearly-labelled authoritative block with explicit instruction "quote the FROM Script(...) block verbatim — NEVER write CALL syntax or fabricate SQL".

Test: tests/test_algorithm_params.py::test_algorithm_system_prompt_raw_ddl_injected_and_stripped_from_json.

[0.3.1.98] - 2026-06-16

Fixed (feature ingestion — CHAR entity-column length truncated; NULL in PI keys causes cartesian MERGE)

Two data-correctness bugs in upload_features when entity_id is passed as a list or string (type inference path):

1. CHAR entity-column length silently truncated. A source view with CLI_CD CHAR(14) produced an FS_T_* feature table with CLI_CD CHAR(1). Root cause: get_column_types (utils/info.py) applied the length-preserving path only for VARCHAR, not CHAR. The raw _td_column_names_and_types value for a CHAR column drops the length, and the FS_T_* DDL was built verbatim from that dict.

Fix: get_column_types now delegates all character columns (CHAR and VARCHAR) to get_feature_types_sql_format — the existing exact-type extractor used for feature columns. It is type-agnostic and length+charset-preserving (reads CAST(col AS <type>) from show_query()). Fail-open: if the exact extraction raises, falls back to the previous behaviour for the affected columns.

2. NULL in PI key columns causes cartesian-product MERGE / skew. NULLs in entity/primary-index columns were only handled when entity_null_substitute was passed. Without it, the MERGE ON clause used plain equality so un-substituted NULL keys never matched, causing rows to fan out (cartesian-like) with massive runtime and storage cost.

Fix: a NULL-count check runs immediately after the existing duplicate check in prepare_feature_ingestion (feature_store/feature_data_processing.py). If any entity column still has NULLs in the volatile table (i.e. after any COALESCE from entity_null_substitute), raises ValueError naming the column(s). Mirrors the duplicate-row guard exactly: same location, same log+raise pattern, same DEBUG_MODE sample query.

Tests: tests/test_entity_type_inference.py, tests/test_null_entity_guard.py.

[0.3.1.97] - 2026-06-16

Fixed (consumer agent — OUT_OF_SCOPE regression on compound EXPLAIN/ALGORITHM questions)

v0.3.1.96's tightened ca-explain description (an adversarial "SKIP when the user asks which ML ALGORITHM …" clause) made the small classifier (ministral-14b) bail to OUT_OF_SCOPE on a compound, in-scope question under multi-turn history — e.g. "How is ml_risk_score computed? What algorithm is behind it?" (MCP on) wrongly refused, where v0.3.1.94 answered it as EXPLAIN. The conflicting EXPLAIN + ALGORITHM cues plus 9 turns of context tipped the model over.

  • Description softened (skills/ca-explain/SKILL.md): dropped the adversarial ALGORITHM-SKIP line; kept the "how/formula is ALWAYS EXPLAIN, even for ML-backed features and even when the sentence also mentions the algorithm" anchor. The ca-algorithm classifier example already disambiguates the algorithm-only case.
  • Deterministic safety net (_rescue_out_of_scope in agent/graph.py, wired into _node_classify_impl): when the classifier returns OUT_OF_SCOPE but the question names a KNOWN catalog feature/object (_known_catalog_names, word-boundary match), re-route — EXPLAIN for a how/formula shape (EN/FR), DEFINITION otherwise. Grounded in the catalog so a genuinely off-topic question (e.g. "what's the weather?") still refuses, and the algorithm-only question still routes to ALGORITHM (MCP on) / the "enable MCP" message (MCP off).

Verified live (demo, MCP on/off): the compound question answers (EXPLAIN), algorithm-only → ALGORITHM, off-topic → refusal. Test: tests/test_out_of_scope_rescue.py.

[0.3.1.96] - 2026-06-16

Fixed (consumer agent — EXPLAIN "Variables involved" accuracy)

For rule_based_score, EXPLAIN's "Variables involved" listed 13 columns when the formula references only 5, and rendered subquery aliases (flags, base_data) as if they were tables.

  • Over-listing → transitive-closure pruning. The LLM populates source_columns from the whole DDL; _validate_source_columns (agent/consumer_agent.py) only appended, never pruned. New _formula_column_closure(calculation_sql, ddl) keeps the formula's direct variables plus the grain keys reached through their derivation (window PARTITION BY/ORDER BY and GROUP BY keys — e.g. patient_id, drug_code, claim_date behind a ROW_NUMBER()-based duplicate flag), and drops the plain arithmetic operands of an already-listed derived feature (provider_avg, amount, …). Wired into skill_explain via _prune_source_columns. Fail-open: no pruning when formula/DDL absent.
  • Alias-as-table → resolve or drop. _resolve_source_table_alias + _extract_ddl_table_aliases resolve an LLM-claimed source_table to the real base table (FROM claims_raw c → c ⇒ CLAIMS_RAW), or return None for a subquery/wrapper alias (flags, base_data, ps). Applied inside _validate_source_columns so a non-table alias never survives into result["source_columns"] — the renderer and the accumulated context graph both inherit clean data. graph.py _build_context_update additionally normalizes LINEAGE source_tables via _normalize_source_tables (drop names not in the view's FROM/JOIN set). Fail-open on missing DDL.

Added (consumer agent — ALGORITHM: input grounding + reproduction-grade parameters)

The ALGORITHM skill can now describe everything needed to rebuild a similar tdstone2 process, and grounds the algorithm in what its inputs mean:

  • Input grounding (D): skill_algorithm resolves the business descriptions of the model's column_names_X inputs (reusing _explain_input_column_context) and injects them, so the explanation grounds each SQL input in its meaning. result["input_meanings_resolved"].
  • Reproduction-grade parameters (E): parses the whole TDS_MODEL_REPOSITORY.ARGUMENTS (adds sto_parameters — column typing + output_format); reads the training grain from TDS_MAPPER_REPOSITORY (model_structure: id_row/id_partition/id_fold/dataset); and parses the training STO view DDL via new _parse_sto_invocation (sto_invocation: partition_by/order_by/script_command/returns/charset). The LLM prompt (now the DB-free seam _algorithm_system_prompt) frames these as a tdstone2 HyperModel configuration recipe, grounded in the real package API. All bounded + fail-open.

Changed (classifier clarity — DEFINITION vs EXPLAIN vs ALGORITHM)

Tightened the ca-definition / ca-explain / ca-algorithm SKILL.md descriptions and classifier examples so DEFINITION=meaning, EXPLAIN=formula/"how computed" (the cheap path), and the SQL-intensive ALGORITHM fires ONLY on explicit ML-model/algorithm/code/package/ parameter requests. "How is X computed / what is the formula" always routes to EXPLAIN, even for ML-backed features.

Tests: tests/test_explain_formula_closure.py, tests/test_explain_alias_resolution.py, tests/test_algorithm_params.py.

[0.3.1.95] - 2026-06-16

Added (consumer agent — ALGORITHM skill: explain the ML model behind a feature)

A new ALGORITHM intent/skill answers data-scientist questions about the machine-learning model behind an ML-backed feature: which algorithm, which Python packages, the model code, the hyperparameters, how many partitions were trained. It is distinct from EXPLAIN (which explains the feature's SQL formula) — ALGORITHM explains the ML model.

ML features are produced by tdstone2 via a Teradata Script Table Operator (STO). skill_algorithm (agent/consumer_agent.py) walks the tdfs4ds → tdstone2 metadata chain entirely via tdml.execute_sql:

  • Resolve the feature → its process view (same resolver as EXPLAIN) → SHOW VIEW DDL.
  • Detect the STO (_is_sto_backed) and parse the scoring-mapper UUID from the TDS_STO_SCORING[_LIVE]_<uuid> reference (_parse_sto_mapper_uuid). A non-ML (SQL-rule) feature returns error=not_an_ml_feature with a message pointing the user to EXPLAIN.
  • TDS_MAPPER_REPOSITORY (ID = uuid) → trained-models table; a capped DISTINCT join (_build_distinct_model_join_sql, ≤ tdfs4ds.ALGORITHM_MAX_CODES=20) resolves ID_TRAINED_MODEL → ID_MODEL → ID_CODE + ARGUMENTS in one round trip.
  • ARGUMENTS JSON → hyperparameters + column_names_X (model inputs). The CODE blob is read only when the question is about code/packages/algorithm internals, via FROM_BYTES(CODE,'ascii') (_read_code_blob; a plain CAST(... AS VARCHAR) errors), truncated to tdfs4ds.ALGORITHM_CODE_MAX_CHARS=8000; packages via _extract_packages_from_code.
  • Trained-models summary (_read_trained_models_summary) reports partition count + STATUS. Freshness caveat: when TD_TIMECODE is the open-end sentinel (year ≥ 9999, _is_sentinel_timestamp) the exact training date is reported as unavailable — never printed as 9999-01-01. Exact package versions are not stored in metadata; the answer says so.
  • The skill returns its final answer directly (no second LLM pass).

MCP-gated availability: the ALGORITHM intent is registered in the graph/classifier only when mcp_enabled=True (build_graph adds local_intent_map["ALGORITHM"] under the gate; the existing intent filter then keeps it out of the dynamic Pydantic Literal when MCP is off). When MCP is off, an algorithm-shaped question is detected deterministically (_looks_like_algorithm_question) and answered with a fixed "enable MCP" message (EN/FR) via the ALGORITHM_UNAVAILABLE pseudo-intent.

New files/config: tdfs4ds/skills/ca-algorithm/SKILL.md; tdfs4ds.ALGORITHM_MAX_CODES, tdfs4ds.ALGORITHM_CODE_MAX_CHARS. Tests: tests/test_algorithm_{sto_parse,packages,sentinel,join_sql,gating}.py.

[0.3.1.94] - 2026-06-16

Fixed (consumer agent — multi-feature DEFINITION misrouted to RECAP; "(from flags)" annotation broke lookup)

A multi-feature question that LISTS named features — e.g. "what are these features: flag_provider_zscore_3 (from flags), provider_z_score (from flags), …" — was misclassified as RECAP (returning "no features in conversation context") instead of DEFINITION. The "what are these features" phrasing collides with the RECAP few-shots/the "these"-pronoun hint, and a small classifier (ministral-14b) anchors on it and discards the explicit list. The trailing "(from flags)" annotation additionally broke the exact-match catalog lookup.

Fixes (in agent/graph.py and agent/consumer_agent.py): - Deterministic RECAP→DEFINITION guard in _node_classify_impl: when the question lists ≥2 known catalog feature names (or ≥1 after a colon), re-route to DEFINITION and set object_name to the matched names. A token counts if it EXACTLY matches a catalog name OR fuzzy-matches one with high confidence (tdfs4ds.FUZZY_NAME_MATCH_CUTOFF, default 0.90), so a single typo in the list still routes correctly. Typo'd tokens are passed as typed, so skill_definition_multi's existing per-element correction emits "did you mean …?". Exact- or-high-cutoff gating keeps a genuine RECAP ("what did we discuss?") from being hijacked. - _clean_object_name extended to also strip ONE trailing parenthetical annotation (provider_z_score (from flags) → provider_z_score), applied per comma-split element at every multi-name exact-lookup node (node_skill_definition, node_skill_lineage, node_recognize_entity). Interior characters are never touched (names have no parens; composite grain claim_id,region is unaffected). - Size-adaptive fuzzy backend _best_name_similarity (the user's scale point): Python difflib for small per-turn-cached catalogs (zero DB round-trip — the consumer-agent case), Teradata in-database StringSimilarity (jaro_winkler) above tdfs4ds.FUZZY_INDB_THRESHOLD (default 500) for large catalogs. - Prompt hardening (both classifier paths + ca-definition/SKILL.md): DEFINITION now states "one or more, comma-separated; an explicit list is DEFINITION not RECAP"; RECAP states "only when no explicit names are listed"; plus a contrasting few-shot.

Verified live: the failing question now defines all 5 features (no "(from flags)" leakage), a typo'd element ("provdr_z_score") re-routes and surfaces the "did you mean" correction, a real RECAP stays RECAP, and the prior 7-turn session is unchanged. Covered by tests/test_recap_definition_routing.py and tests/test_clean_object_name.py.

[0.3.1.93] - 2026-06-16

Fixed (consumer agent — EXPLAIN crashed in the streaming / chatbot path)

Asking an EXPLAIN question in the Gradio chatbot (e.g. "How is provider_z_score calculated? Walk me through the formula.") returned "An error occurred: 'NoneType' object is not iterable". The streaming entry point consumer_agent_stream (agent/graph.py) iterates graph.stream(…, stream_mode="updates") and did final_state.update(node_delta) for every node. The EXPLAIN path includes the no-op trace node node_pre_skill_explain which returns {}; LangGraph surfaces an empty-dict update as a None delta in updates mode, so final_state.update(None) raised and aborted the whole turn before skill_explain ran. (Pre-existing — only the streaming path is affected; consumer_agent() via .invoke always handled EXPLAIN correctly. Not a regression from 0.3.1.91/0.3.1.92.)

Fixed by guarding the merge: if node_delta: final_state.update(node_delta) — empty/None deltas are skipped (state is unchanged; the real work node runs next) while the trace step is still emitted. EXPLAIN now returns the full formula walkthrough through the chatbot, identical to the .invoke path. Covered by tests/test_stream_none_delta.py; verified live across a 7-turn session (overview → counts → search → definition → explain → compare → explain).

[0.3.1.92] - 2026-06-16

Fixed (consumer agent — DEFINITION failed on markdown-quoted feature names)

A DEFINITION question that wrapped the feature name in markdown backticks or quotes — e.g. "What exactly is provider_z_score?" — could return "feature not found … did you mean …?" instead of the definition. The LLM intent classifier sometimes returns object_name with the wrappers kept (`provider_z_score`), and every catalog lookup is an exact case-insensitive match, so the wrapped name matched nothing and the skill fell through to the fuzzy near-miss path. (Pre-existing fragility, exposed by classifier nondeterminism — not introduced by 0.3.1.91.)

Fixed at the single chokepoint: _node_classify_impl (agent/graph.py) now passes the classifier's object_name, the explicitly-named entity_name, and each low-confidence interpretation's object_name through a new _clean_object_name helper that strips surrounding backticks / single / double quotes / whitespace only (interior characters — _, digits, and composite-grain commas like claim_id,region — are untouched, so matching is not loosened). Every downstream skill now receives a clean name. Covered by tests/test_clean_object_name.py; verified live (the quoted question now defines correctly, while a genuinely missing feature still gets the graceful "did you mean" response).

[0.3.1.91] - 2026-06-16

Performance (consumer agent — collapsed N+1 query loops, per-turn SQL memo)

Multi-feature questions (summary, multi-feature definition, feature-group freshness, lineage) issued one Teradata round-trip per feature / entity / process / dependency node in loops, and re-materialized whole catalog / business-dictionary tables several times within a single turn. Measured live on the demo feature store, four representative turns went from 232 → 67 SQL round-trips (−71%) with identical answers (e.g. domain summary 61→7, 3-feature definition 91→18).

  • Batched business-dictionary helpers (_query_bd_columns / _query_bd_sections / _query_bd_objects in agent/consumer_agent.py) now accept list kwargs (table_names / column_names / object_names) and emit a single UPPER(col) IN (…) predicate via the shared _in_clause builder. The singular kwargs remain back-compatible.
  • _get_feature_catalog_with_docs rewritten from per-entity get_list_features + per-feature _query_bd_columns + a full all-domains process_catalog materialization (tens of round-trips) to ~3 scoped reads: one feature-catalog read, one batched get_feature_versions per entity, one scoped PROCESS_ID IN (…) map (_processes_by_id), one scoped BD_COLUMNS IN (…) read joined in pandas.
  • _build_multi_object_context, skill_lineage, and both freshness paths now issue one batched query for all views / datasets / dependency nodes / process_ids instead of looping. New follow_up_report_many(process_ids) in process_store/process_followup.py fetches all follow-up history in one PROCESS_ID IN (…) query.
  • SUMMARY no longer scans the whole BD_SECTIONS / BD_OBJECTS tables — the reads are scoped to the in-scope process views and registered datasets.
  • Per-turn SQL memo (_TURN_CACHE / _turn_cached, reset at the start of every turn in node_classify): scoped catalog / BD reads, and the unavoidable whole-store scans used for object-name resolution (_all_processes_cached / _all_features_cached), are materialized at most once per turn. Strictly intra-turn — the cross-turn accumulated_context_graph cache is unchanged.

Behaviour is preserved: same aggregates, same answers. Covered by DB-free unit tests in tests/test_agent_query_batching.py.

[0.3.1.90] - 2026-06-15

Fixed (pyvis lineage graph — tables and datasets misclassified as plain views)

FS_LINEAGE_GRAPH.PARENT_OBJECT_TYPE / CHILD_OBJECT_TYPE are CHAR(1) columns. Teradata pads fixed-width CHAR values, so the Python driver returns "T " / "D " instead of "T" / "D". The lookup {"V":…, "T":…, "D":…}.get("T ", "view") fell through to the "view" default in get_persisted_graph, making every table and dataset node appear as a plain (blue) view in the pyvis diagram. Fixed by stripping whitespace from both type columns immediately when loading rows in load_lineage_digraph.

Added (pyvis lineage graph — legend)

A fixed-position legend (bottom-left corner, semi-transparent white card) is now injected into the generated HTML, showing the four node types with their colors: orange (process view), blue (plain view), green (table), purple (dataset).

[0.3.1.89] - 2026-06-15

Changed (pyvis lineage graph — three-tier node coloring)

The interactive lineage graph (plot_lineage_pyvis) previously used a single color for all view nodes, making it impossible to distinguish feature-engineering process views from plain intermediate views at a glance.

  • Three-tier color scheme in _PYVIS_COLOR (lineage/lineage_store.py):
  • Blue #6baed6 — plain / intermediate view (not a registered process)
  • Orange #e6550d — registered process view (present in FS_V_PROCESS_CATALOG)
  • Green #31a354 — source table
  • Purple #756bb1 — dataset node
  • Process-view resolution: plot_lineage_pyvis now queries process_catalog(all_data_domains=True) once before the node loop and builds a (db, obj) lookup set. Each "view" node is classified as "process" or "plain view" against that set. Degrades gracefully (falls back to plain-view color) when the catalog is unavailable.
  • Tooltip labels updated to "type: process view" / "type: plain view" / "type: table" / "type: dataset".

[0.3.1.85] - 2026-06-14

Fixed (lineage graph — valid-time temporal writes, no history churn)

FS_LINEAGE_GRAPH is a valid-time temporal table, but record_process_lineage wrote it with a blind CURRENT VALIDTIME DELETE + INSERT of every edge on each call — so re-running on an unchanged graph closed and re-opened identical rows, inflating the validity history (observed: 46 active edges accumulating to 952 rows across repeated rebuilds). A row's ValidPeriod must reflect when an edge actually existed, not when the rebuild happened to run.

  • Diff-based writes (record_process_lineage): read the current edges for the parents in the new graph, then only DELETE edges that disappeared, INSERT edges that are new, and UPDATE the OBJECT_TYPE of matched edges whose type changed. Unchanged edges are left untouched (their ValidStart is preserved). Re-running on an unchanged graph now writes zero rows. Returns the number of edge changes (0 = no change).
  • rebuild_lineage_graph no longer wipes. The unconditional CURRENT VALIDTIME DELETE FROM <table> at the start of a full refresh was removed (it defeated the diff by making every edge look new). Pruning of edges left by since-dropped objects is now a targeted, in-memory desired-set diff (prune=True, default) that closes only the edges no longer produced by any seed — never re-opening still-valid ones.
  • Teradata syntax fix: the existing-edges read used the unsupported row-constructor predicate WHERE (a, b) IN ((..),(..)) (which threw, leaving the diff empty so everything re-inserted); replaced with a flat WHERE UPPER(PARENT_OBJECT_NAME) IN (...) + a Python (db, obj) re-check.

Verified live on the demo: rebuild from empty → 46 edges; rebuild ×2 more → 0 changes, table stays 46/46; delete one edge then scoped-refresh → exactly 1 change (re-inserted), confirming real changes are still detected.

Changed

  • Demo notebooks (claims-anomaly-detection) now call tdfs4ds.connect(..., create_if_missing=True) so the lineage graph (and all catalog tables) auto-create on a fresh schema and the write hook always has its table.

[0.3.1.84] - 2026-06-14

Added (persisted, temporal lineage graph)

Lineage was previously dynamic and downstream-only — every query re-derived the graph by walking SHOW VIEW / SHOW TABLE DDL, and there was no way to cheaply ask "what depends on this?". This release persists the graph so both directions are available, and drives documentation from it.

  • FS_LINEAGE_GRAPH — new VALIDTIME temporal edge table (one row per directed parent → child object edge; nodes reuse the business-dictionary identity (DATABASE_NAME, OBJECT_NAME) + OBJECT_TYPE; no PROCESS_ID on edges — an edge connects objects, not processes). Public view FS_V_LINEAGE_GRAPH. Created when missing at setup() / connect(create_if_missing=True). New module tdfs4ds/lineage/lineage_store.py.
  • Write hook — upload_features now records the process view's lineage subgraph (record_process_lineage) after the optimize/document step (non-fatal). Edges come from the existing build_teradata_dependency_graph (so feature-store technical tables are skipped and datasets are nodes); refresh is idempotent per parent object. Seeds from the view's SHOW VIEW DDL, not SELECT * FROM view — a plain SELECT has no target and yields 0 edges.
  • tdfs4ds.rebuild_lineage_graph(objects=None, all_data_domains=True, check=True) — full refresh (wipe + re-derive from every process view + dataset) or scoped refresh of just the objects you edited (no wipe; runs the impact check after).
  • In-memory query layer (networkx) — load_lineage_digraph reads the table once into a DiGraph; get_descendants (impact / "what depends on this") and get_ancestors (upstream sources) traverse in memory. Falls back to an on-the-fly build when the persisted table is empty/unavailable, so query/visualisation always works (logs a hint to run rebuild_lineage_graph()).
  • tdfs4ds.genai.rebuild_documentation(objects=None, force_update=False) — document the whole store from the persisted graph (topo-sorted leaves-first, shared visited-sets → each process/dataset documented exactly once), or re-document only the impact set of edited objects. Shares the new _document_from_graph walk with document_feature_store_incremental.
  • tdfs4ds.check_downstream_processes(objects) — deterministic EXPLAIN SELECT * FROM <view> (compile-only) on every registered process downstream of a change, so a changed view's result-column change that breaks a consumer is caught: returns {checked, ok, broken:[{view, process_id, error}]}.
  • tdfs4ds.resolve_objects(objects, require_exists=True) — identity-resolution helper: normalises "db"."NAME" / db.name / NAME (any case) and verifies against the graph then DBC.TablesV, raising on any unmatched name (no silent no-op).
  • plot_lineage_pyvis(graph, output_path=...) — interactive HTML network for the persisted graph (UTF-8 safe on Windows).
  • Opt-in cascade re-documentation — upload_features(..., cascade_redocumentation=True) re-documents downstream processes/datasets, but only when the process's own documentation actually changed (document_process_incremental now reports _doc_changed).
  • Consumer agent — impact direction — ca-lineage / skill_lineage now also answers "what depends on / consumes / would be impacted by this feature?" (new impacted_objects from the persisted graph). New developer skill fs-lineage-graph covers the admin operations (rebuild, scoped refresh, impact check, re-document, pyvis).

Fixed

  • resolve_process_id_for_object read the process catalog via a hand-rolled CURRENT VALIDTIME SELECT … FROM FS_V_PROCESS_CATALOG, which errors on some installs (the public view already applies validtime) and returned None for every process. Now reads via process_catalog().to_pandas() — also fixes the cascade resolver.

[0.3.1.81] - 2026-06-13

Fixed (dataset catalog — two registration bugs surfaced by the 0.3.1.79 split)

Re-running build_dataset for a flat view (ds_investigation_queue) under the corrected entity/feature split (0.3.1.79) exposed two latent DatasetCatalog bugs. Both were masked before the split fix because flat views previously had an empty dropped_entity set.

  • add_dataset deleted the features it had just inserted. The feature-cleanup loop iterated dropped_entity instead of dropped_features, so every column moved out of the entity grain by the new split was valid-time-closed in the FEATURES table seconds after being MERGE-inserted. Result: the dataset registered with the correct 2-key entity but 0 current features (verified live: 16 feature rows each living ~4 s before being re-closed). One-word fix: iterate dropped_features. After the fix, ds_investigation_queue registers cleanly as 2 entity / 16 features.
  • drop_dataset silently did nothing. (1) The guard if self.catalog[…].shape[0] == 1: read a stale snapshot bound in __init__ and had no else, so a miss dropped nothing with no log or error. (2) It only CURRENT VALIDTIME DELETEd (soft close), with no way to purge closed history. (3) The module-level drop_dataset() wrapper constructed DatasetCatalog with the default name='DATASET' → wrong tables (FS_DATASET_* instead of FS_FS_DATASET_*). Fixed: re-query the catalog view fresh, raise ValueError on a miss, add purge=True (NONSEQUENCED VALIDTIME DELETE) for a full reset, and pass name=tdfs4ds.DATASET_CATALOG_NAME in the wrapper.

Live-verified against the demo: T10 (score-direction note agrees with EXPLAIN), T12 (pharmacy_z_score surfaced), and T15 (dashboard uses the real ds_investigation_queue, CASE-orders risk_band, reports provider/pharmacy as uncovered rather than fabricating a join) all pass.

[0.3.1.80] - 2026-06-13

Version bump to ship the dataset-registration fix (0.3.1.79) and the consumer-agent / serve / reconnect work (0.3.1.77–78). Re-run the affected build_dataset calls to re-register datasets with the corrected entity/feature split.

[0.3.1.79] - 2026-06-13

Fixed (dataset registration — authoritative entity/feature split)

  • build_dataset now registers the entity/feature split it already knows, instead of re-deriving it from the dataset view's DDL. Dataset._retrieve_entities_and_features classified a column as a FEATURE only when the view DDL matched the canonical feature-store subquery (B1.FEATURE_VALUE AS … WHERE FEATURE_ID=… AND FEATURE_VERSION=…) and treated everything else as an ENTITY. A flat / non-canonical dataset view therefore registered 0 features with every column as an entity (observed: ds_investigation_queue). The role of a column is dataset-specific (the same name may be a feature in one process and an entity in another), so it must come from the build, not the catalog or DDL.
  • Dataset.__init__ gained optional entity= / features= parameters; when both are supplied they are used verbatim and the DDL-regex path is skipped. build_dataset computes the split via the new compute_dataset_split(entity_id, selected_features) (same column aliases the view builder emits) and fills SQL types from the built view's columns. Backward compatible: Dataset(view_name=…) / Dataset(df=…) without the split still use the regex derivation; build_dataset falls back to it if the split can't be computed.

[0.3.1.78] - 2026-06-13

Follow-ups from the live demo replay of the T10/T12/T15 fixes, plus a connection-resilience tool.

Fixed (consumer agent — dashboard grounding, live-verified)

  • Entity columns now drive the dataset cover (T15). skill_compose_dashboard read the wrong catalog column (ENTITY_ID; the live catalog uses ENTITY_NAME) so available_entities was always empty and provider/pharmacy never reached the cover. Fixed to use ENTITY_NAME, and _build_dataset_entity_keys now reads the actual ENTITY column returned by get_dataset_entity (was probing non-existent names → empty join keys). _extract_feature_list_from_question gained a deterministic entity-key fallback (scans the question for known entity stems when the LLM returns none).
  • Cross-grain identifiers handled honestly. _cover_features_with_datasets only JOINs an entity-providing dataset when it shares a key with the chosen set; different-grain stats datasets (e.g. provider_id-only) are reported as cross_grain (rendered as a grain caveat) instead of being force-joined or triggering a false "cannot compose". The dashboard now answers with the real claim-grain dataset, CASE-ordered band, and projected entity ids.
  • MCP no longer bypasses the grounded dashboard path. With MCP enabled, dashboard-shaped questions classified as DATA_QUERY previously routed to free-form MCP SQL (which fabricated columns / used ORDER BY band DESC / LIMIT). node_skill_mcp_tools now short-circuits dashboard requests to skill_compose_dashboard, and synthesize_answer routes DATA_QUERY results carrying dashboard_intent through the grounded R5 renderer. MCP-on and MCP-off now produce the same grounded answer.

Added (Teradata reconnection)

  • tdfs4ds.reconnect_teradata() — re-establishes the teradataml context from TDFS4DS_TD_* env / .env credentials (tdfs4ds/utils/reconnect.py). execute_query now auto-detects a dropped-connection error (is_connection_error), reconnects once, and retries the query transparently; the helper is also callable manually. SQL/logic errors are not misclassified as connection errors.

[0.3.1.77] - 2026-06-13

Consumer-agent quality fixes from the business-user-journey eval (T10, T12, T15). New deterministic helpers replace fragile LLM/substring heuristics; all are pure and unit-tested in tests/test_agent_pure_helpers.py (no Teradata needed).

Fixed (consumer agent — dashboard query synthesis, eval T15)

  • Grounded dashboard composition. synthesize_answer's dashboard path no longer lets the LLM pick the dataset, the ranking column, or invent a join. New pure helpers do it deterministically: _choose_ranking_metric() prefers a numeric risk/score column (inverted scores → ORDER BY … ASC) and, when only a categorical band is available, emits a CASE-expression ordering instead of an invalid ORDER BY risk_band DESC; _cover_features_with_datasets() greedily covers the requested features across datasets and derives the JOIN keys from the shared entity columns. When no requested feature is exposed, columns are reported as unavailable rather than backed by a fabricated dataset. The dataset allow-list is verified against _known_catalog_names(), and the LLM is constrained to present the Python-composed SQL verbatim.

Fixed (consumer agent — SEARCH feature omission, eval T12)

  • Depluralizing, entity-aware token matching. skill_search's catalog pre-filter previously substring-matched question tokens against FEATURE_NAME + DATA_DOMAIN only, with no stemming — so "pharmacies" missed the feature pharmacy_z_score. New _normalize_tokens() / _depluralize() map plurals to singulars, _row_token_set() also matches the entity name, and matches are bidirectional prefix-aware (provider ↔ provider_id). Matched rows are now ranked by match strength so the strongest survive the SEARCH_CATALOG_MAX_ROWS cap.

Fixed (consumer agent — score-direction contradiction, eval T10)

  • DEFINITION ↔ formula reconciliation. DEFINITION trusted the stored business description, which could claim "higher = riskier" while the formula inverts the score (1 - …, so 0 = highest risk) — contradicting EXPLAIN. New detect_score_inversion() (shared single-source util in genai/documentation.py) + _direction_correction_note() append an authoritative "lower = higher risk" clarification when the prose contradicts an inverted formula. The DDL fetch is gated on a cheap prose check so the cache fast-path stays O(1) for non-risk features.

[0.3.1.76] - 2026-06-13

Added (HTTP server — tdfs4ds[serve])

  • OpenAI-compatible API + LangServe playground — new optional tdfs4ds.serve submodule (pip install tdfs4ds[serve]) exposes both agents over HTTP. A single FastAPI app serves GET /v1/models + POST /v1/chat/completions where the model field selects the agent (tdfs4ds-consumer → consumer_agent, tdfs4ds-query-optimizer → query_optimizer_agent), so any OpenAI client can drive them. The query optimizer takes its SQL inline in the chat message. LangServe routes (/consumer, /query-optimizer) are mounted alongside for the interactive playground.
  • Run modes — python -m tdfs4ds.serve, or the new root Dockerfile / docker-compose.yml (docker run --env-file .env -p 8000:8000 tdfs4ds-serve). Configuration is entirely TDFS4DS_* env vars / .env; the new TDFS4DS_TD_HOST / _TD_USERNAME / _TD_PASSWORD (+ optional _TD_LOGMECH, _TD_DATABASE) open the Teradata context at startup, and TDFS4DS_SERVE_BUILD_INDEX toggles the vector-index pre-build. See .env.example and the new docs page Deployment → OpenAI-compatible API server.
  • Conversation continuity maps the OpenAI user field to the agent's thread_id (stateless per request otherwise); stream: true is supported (single content chunk + [DONE]).
  • OPTIMIZE answers are now a compact summary + a downloadable report link. query_optimizer_agent's OPTIMIZE path previously returned the entire Markdown report as the answer. It now writes the full report to disk (under QUERY_OPTIMIZER_REPORT_DIR/<DATA_DOMAIN>/, same naming convention as the upload_features optimizer) and the answer carries only the score delta, optimized SQL, FilterManager applicability, and a reference to the report (report_path on the skill result; the full text stays available as report_full). Benefits every caller — the OpenAI API, the LangServe playground, and notebooks.
  • tdfs4ds.serve exposes the reports for download. GET /reports lists available reports and GET /reports/{path} downloads one (path-traversal-guarded). The OpenAI /v1/chat/completions response rewrites the relative report_path into an absolute …/reports/… URL using the request's base URL.

Added (admin chatbot app — launch_chatbot_app)

  • launch_chatbot_app() — new additive wrapper in tdfs4ds.agent.chatbot that embeds the existing consumer-agent chatbot under a collapsible ⚙️ Parameters panel for managing, live: the feature store database (switch validated by connect(..., create_if_missing=False) — a DB without a feature store is rejected, no switch), the vector store (backend + Chroma path + a Refresh/rebuild button), and the instruct / embedding model settings (provider, model, URL, API key, proxy, dimension). Applying a data/model change rebuilds the index when needed, calls reset_consumer_agent(), rotates the thread, and clears the chat. The chat UI is factored into a shared _build_chat_ui helper; launch_chatbot / launch_chatbot_with_index are unchanged (backward compatible).
  • Served on a second port in Docker. python -m tdfs4ds.serve now also launches the admin chatbot in the same process (shared Teradata context + index), on TDFS4DS_SERVE_CHATBOT_PORT (default 7860), toggled by TDFS4DS_SERVE_CHATBOT (default true). The Dockerfile / docker-compose.yml expose 7860 alongside 8000.
  • Per-user feature-store database = one container per user. Configuration (SCHEMA + catalog globals + Teradata context) is process-wide, so the panel/DB switch applies to everyone in a process. Documented guidance: for per-user databases, run a separate container per user (same Teradata login, different TDFS4DS_SCHEMA) — no code change, no per-session isolation within a process.

Added (outbound proxy for LLM / embedding calls — HTTP or SOCKS5)

  • INSTRUCT_MODEL_PROXY (with EMBEDDING_MODEL_PROXY falling back to it) routes the OpenAI-family LLM and embedding HTTP clients through a proxy; the URL scheme selects the protocol (http://… or socks5://…), empty = direct. Injected via an explicit httpx client (http_client= / http_async_client=) so it stays scoped to those calls and never touches the Teradata connection. socks5:// support comes from the new httpx[socks] dependency. Configurable via env / .env / JSON and live in the chatbot admin panel.

[0.3.1.64] - 2026-06-11

Consumer-agent reasoning fixes driven by the LLM-as-judge evaluation run of 2026-06-11 (scenarios E01, E06, E10, S01).

Fixed (consumer agent — hallucinated lineage, eval E06)

  • Degenerate lineage cache hit — after a DEFINITION turn the accumulated context knows only feature → process_view (no upstream objects). _query_accumulated_context_for_lineage() served this as a cache hit, the LINEAGE fast path skipped the real DDL scan, and the LLM — given a dependency list containing only the process view itself — confabulated source table names. The helper now returns a cache miss when the process view is absent from source_views or has an empty depends_on, routing the question to the real build_teradata_dependency_graph scan. The fast-path LINEAGE prompt additionally forbids inventing object names and instructs the LLM to say so when no upstream is listed.

Added (consumer agent — explicit entity short-circuit, eval E10)

  • entity_name extraction in the intent classifier — both IntentResult and the dynamic intent model now carry an optional entity_name; the classifier prompt extracts an explicitly named entity/granularity ("total_amount for pharmacy_id"). _node_classify_impl writes it to resolved_entity_name (cleared on turns with no explicit entity), which has top priority in the _hint_entity chain — so an ambiguous feature name resolves without a clarification round-trip when the user already said which entity they mean.
  • _entity_from_question() deterministic safety net — before raising ambiguous_feature, skill_definition, skill_freshness, skill_lineage, and skill_explain scan the raw question for the candidate entity names (case-insensitive, _/space-normalised, multi-key entities match on any key component) and auto-resolve when exactly one matches.
  • Deterministic ambiguous_feature clarification — the clarification message in synthesize_answer is now a template (EN/FR via the new _question_is_french() heuristic) instead of an LLM call, which was observed embellishing the clarification with a guessed generic definition of the feature.

Added (consumer agent — typo recovery, eval E01)

  • Classifier rule: intent follows question shape — "what is X" classifies as DEFINITION even when X matches no known object (it may contain a typo); previously the unfamiliar token flipped the intent to SEARCH whose generic synthesis answered speculatively without touching the business dictionary.
  • Single-candidate fuzzy auto-correct — when _fuzzy_match_feature returns exactly one candidate, skill_definition and skill_freshness now answer for it directly with a correction_note ("'ml_rsk_scre' was not found — answered for the closest match 'ml_risk_score'") instead of a did-you-mean round-trip — mirroring the existing behaviour of skill_lineage / skill_explain. Multiple candidates keep the did-you-mean prompt.

Added (consumer agent — answer verifiability, eval S01)

  • grounding_context on DEFINITION results — skill_definition returns the raw retrieved evidence (context_parts) the explanation was generated from, so the trace panel and LLM-as-judge evaluations can verify the answer is grounded rather than scoring a fully-grounded answer as unverifiable.

Fixed (consumer agent — crashes in the 0.3.1.63 work)

  • consumer_agent(return_context=True, return_trace=False) crashed with UnboundLocalError: trace_md; it now returns the documented 2-tuple (answer, accumulated_context_graph) (the 0.3.1.63 note has been corrected accordingly).
  • _build_context_update() crashed with UnboundLocalError: _feat_cache_key when a skill result carried a context_graph but no resolved_triplet (the import was nested inside the single-feature branch); the import is now function-level.

[0.3.1.63] - 2026-06-10

Changed (consumer agent — accumulated context keying)

  • Composite feature cache keys — accumulated_context_graph.features is now keyed by FEATURE_NAME::ENTITY_NAME (both uppercased) instead of a bare FEATURE_NAME, so the same feature name registered under two entities no longer collides in the cross-turn cache. New helpers _feat_cache_key() and _feat_cache_lookup() (in consumer_agent.py) centralise key construction and a 3-tier lookup: exact composite → FEATURE:: prefix scan → legacy plain-FEATURE key (backwards-compatible with older snapshots). All six _query_accumulated_context_for_* fast-path helpers and both skill_definition fast-paths route through _feat_cache_lookup.
  • Uppercase-keyed context sections — _merge_accumulated_context() (graph.py) now normalises keys to uppercase for the features, datasets, entities, and source_views sections for stable lookup. processes and freshness_timeline stay keyed by their process-id UUIDs.

Added (consumer agent — EXPLAIN feeds the context graph)

  • EXPLAIN now accumulates context — _skill_explain_multi() returns a context_graph (one entry per explained feature, carrying process_id + view_name) and a resolved_triplet focus pointer; node_skill_explain merges this via _build_context_update / _merge_accumulated_context, mirroring node_skill_definition. Follow-up turns after an EXPLAIN can therefore resolve from cache. skill_explain now also captures PROCESS_ID per feature.

Added (consumer agent — entity-scoped SUMMARY)

  • skill_summary(entity_filter=…) — a new entity_filter argument (and a new entity / data_domain recognition path in _recognize_object_type) scopes the summary to a single entity, answering "what features belong to entity X?". node_skill_summary derives the entity from conversation state and passes it through.

Changed (consumer agent — API signature)

  • consumer_agent(return_context=True) returns (answer, accumulated_context_graph); combine with return_trace=True for the 3-tuple (answer, trace_md, acc_ctx). return_trace=True alone is unchanged ((answer, trace_md)). (Corrected in 0.3.1.64 — this entry originally claimed a 3-tuple for return_context=True alone, which never matched the docstring, tests, or code.)

[0.3.1.59] - 2026-06-10

Added (consumer agent — conversation-management intents)

  • RECAP / OUT_OF_SCOPE intents now actually fire — these built-in intents were wired into the graph but had no classifier few-shot examples, so "what are the features we are talking about?" mis-routed to SUMMARY. _build_classifier_examples() now injects explicit high-confidence examples (3 RECAP variants, 2 OUT_OF_SCOPE variants) plus a contrasting SUMMARY/RECAP pair that teaches the scope-of-data boundary (SUMMARY = full catalog that exists, queries Teradata; RECAP = only what was already discussed, reads memory). build_graph() injects descriptions for both built-in intents into the dynamic Pydantic intent model and the classifier prompt.

Added (consumer agent — entity-clarification recovery)

  • Deterministic recovery from ambiguous-feature replies — when skill_definition returns error="ambiguous_feature" (a feature name exists under multiple entities), node_skill_definition now raises a tracked awaiting_feature_entity_clarification pending action (via the new _ambiguous_feature_pending() helper) carrying the valid entity list, the original feature, and the origin intent. The next reply is resolved deterministically by _node_classify_impl through the existing _resolve_clarification_choice (digit / ordinal / fuzzy) — the LLM classifier is short-circuited entirely.
  • Role separation + intent inheritance: the entity is taken from the reply while the feature (object_name) and intent (DEFINITION) are inherited from stored context, so mixed-concept replies ("the total_amount feature of the provider_id entity") and a bare "1" both resolve correctly. The chosen entity is written to a new resolved_entity_name state field (preferred over _focus_entity) and into accumulated_context_graph._focus_entity.
  • The ambiguous_feature synthesis prompt now renders a numbered list so a digit reply maps 1:1 to stored options.
  • Scope: DEFINITION only; node_skill_lineage keeps its existing early-return. Deferred: a general "feature X of entity Y" phrasing arriving with no prior pending state still can't role-separate (IntentResult has no entity_name field yet).

[0.3.1.41] - 2026-06-08

Changed (chatbot UI — conversation management)

  • Separate Clear and Reset buttons: The single "Clear conversation" button is now split into two distinct actions:
  • Clear Conversation — resets visual display (chat history, trace, viz panel, step indicator, message input) but preserves the LangGraph thread_id, so accumulated_context_graph and all agent memory persist. Use this to refresh the UI while maintaining context across turns.
  • Reset Conversation — resets visual display and rotates to a new thread_id UUID, starting a fresh LangGraph session with no prior context. Use this for a complete restart.
  • Gradio UI enhancement — the "Reset Conversation" button uses variant="stop" for a red/destructive visual cue to distinguish it from the safer "Clear" action.

[0.3.1.38] - 2026-06-02

Changed (skills — semantic collision fixes + v0.3.1.35–37 content updates)

  • Semantic collision disambiguation — added SKIP when … use <skill> instead clauses to 10 SKILL.md files so Claude Code selects the right skill in ambiguous situations:
  • td-explain ↔ qo-skill-explain (both analyze EXPLAIN plans; differ in output format and workflow role)
  • ca-lineage ↔ fs-lineage (both surface lineage; differ in audience and output — NL Q&A vs Python code + Sankey diagram)
  • ca-dataset ↔ ca-build-dataset ↔ fs-dataset (discovery vs agent-driven creation vs code generation)
  • ca-explain ↔ ca-definition (SQL formula → plain language vs business meaning)
  • fs-inspect ↔ fs-analyze (catalog audit vs single-process performance analysis)
  • fs-setup/SKILL.md — added Vector Store Backend optional config block: VECTOR_STORE_BACKEND, TERADATA_VECTOR_STORE_PREFIX, and all five TERADATA_AI_* globals (v0.3.1.37).
  • fs-agent/SKILL.md — added Option B (Teradata Enterprise Vector Store), AGENT_ANSWER_STYLE, and a Chart rendering section describing automatic Chart.js → Plotly conversion for teradata-mcp-server plot_* tools (v0.3.1.35–37).
  • fs-document/SKILL.md — added document_view_incremental() step, formula consistency validation columns table (MONOTONIC_DIRECTION, VALUE_DOMAIN, IS_ABSOLUTE_VALUE, FORMULA_CONSISTENCY_WARNING), and migrate_documentation_columns_schema() migration snippet (v0.3.1.35).

Added (skills — structural completeness)

  • .claude/skills/ expanded from 11 → 24 directories: all 10 ca-* skills and qo-skill-explain, qo-skill-optimize, qo-skill-simplify copied from tdfs4ds/skills/.
  • tdfs4ds/skills/td-analytics-workflow/ (new) — copied from global ~/.claude/skills/ so the canonical package directory is complete (25 skills total).

[0.3.1.37] - 2026-06-01

Fixed (consumer agent — distinct feature count)

  • Feature counts no longer inflated by row duplicates (agent/consumer_agent.py, agent/graph.py) — a feature's identity is the pair (FEATURE_NAME, ENTITY_NAME); the same name registered under two entities is two distinct features and a feature appearing in several datasets must still count once. New shared _distinct_feature_count(df, group_by=None) helper de-duplicates on the identity pair (tolerant of ENTITY_NAME / ENTITY_ID column naming). Applied at get_data_domain() (was .size() over raw rows), skill_summary() domain-wide (total_features = len(feat_doc)), and skill_search() (was nunique("FEATURE_NAME"), which also referenced a non-existent ENTITY_ID column for the per-entity breakdown — now uses ENTITY_NAME with ENTITY_ID fallback). The two groupby("ENTITY_NAME").size().idxmax() dominant-entity picks in graph.py get the same dedup so they are robust to duplicate rows.

Added (vector store — Teradata Enterprise Vector Store backend, opt-in)

  • Backend selector (__init__.py, config.py) — new tdfs4ds.VECTOR_STORE_BACKEND config global ('chroma' default, 'teradata' opt-in) plus TERADATA_VECTOR_STORE_PREFIX (base name for the three named stores: {prefix}_objects, {prefix}_sections, {prefix}_columns). Registered in _FILE_VARS so .env, JSON config, and TDFS4DS_* env vars all work.
  • agent/vector_store_teradata.py (new) — Teradata backend mirror of the Chroma path. Reuses the existing backend-agnostic fetch + doc-builder helpers from vector_index.py (_fetch_objects/_sections/_columns, _objects_to_docs, etc.). build_teradata_index(force_rebuild) calls TeradataVectorStore.from_documents(name=, documents=, embedding=) (rebuild path) and tries add_documents (incremental path) with destroy + rebuild fallback. search_teradata_index(...) calls similarity_search(question=, top_k=, return_type="json") and parses results into the same six-column DataFrame schema (SOURCE, OBJECT_NAME, SECTION_NAME, COLUMN_NAME, TEXT, SCORE); the parser absorbs lists of Documents, dicts, .similar_objects objects, and JSON strings. section_names filter is applied post-hoc.
  • Backend dispatch in vector_index.py — build_vector_index and search_vector_index now branch on VECTOR_STORE_BACKEND; callers (chatbot.launch_chatbot_with_index, consumer_agent._vector_search_context, public agent.__init__ exports) are unchanged.
  • get_vector_store_embedding() (agent/embedding.py) — embedding factory for the active backend. When tdfs4ds.TERADATA_AI_API_TYPE is None, returns the existing LangChain Embeddings from get_embeddings() so vLLM / TEI / OpenAI-compatible / Bedrock setups keep working unchanged. When set, returns teradatagenai.TeradataAI(api_type=..., model_name=..., region=..., api_base=..., access_key=...); TeradataVectorStore accepts both object types at its embedding= argument.
  • New TeradataAI config globals (__init__.py, config.py) — TERADATA_AI_API_TYPE ('aws'|'azure'|'gcp'|'nim'|'hugging_face'|'onnx'), TERADATA_AI_MODEL_NAME, TERADATA_AI_REGION (aws), TERADATA_AI_API_BASE (nim), and TERADATA_AI_ACCESS_KEY (registered as a credential — .env / OS env var only; rejected from JSON config).
  • Dependencies (setup.py) — langchain_teradata was already present; teradatagenai added so the TeradataAI opt-in works out of the box (still lazy-imported so the package is optional at runtime).

Notes

  • The Enterprise Vector Store API does not yet guarantee per-id upsert, so a non-force_rebuild build attempts add_documents and falls back to destroy + rebuild.
  • Live-system items to verify when first switching to the Teradata backend: exact similarity_search(return_type="json") shape, Document.metadata round-trip, and add_documents availability. The parser and builder absorb either form; the rebuild fallback is automatic.

[0.3.1.36] - 2026-05-21

Added (consumer agent — interactive charts from MCP tools)

  • agent/chart_render.py (new) — host-side rendering of MCP chart data. teradata-mcp-server's plot_* tools return Chart.js JSON ({"data": {"labels", "datasets"}, "metadata": {...}}), not MCP-UI / MCP Apps resources. _chartjs_to_plotly_html(tool_name, spec) maps that JSON to a Plotly figure (plot_line_chart→Scatter, plot_pie_chart→Pie, plot_polar_chart→Barpolar, plot_radar_chart→Scatterpolar) and emits a self-contained HTML document with plotly.js inlined (include_plotlyjs=True) so it works air-gapped — no CDN. wrap_iframe() embeds it in a sandbox="allow-scripts" iframe (no allow-same-origin → isolated origin); charts_to_panel_html() stacks multiple charts.
  • Chart capture in the MCP skill (agent/consumer_agent.py) — _extract_chart_from_tool_output() recovers Chart.js payloads from the LangChain ToolMessage .artifact (structured_content) that langchain-mcp-adapters 0.2.2 already exposes. _skill_mcp_tools_async collects charts during on_tool_end (independent of the thread-local trace buffer — charts travel back via the return value) and returns them under skill_result["charts"].
  • Chart rendering in the chatbot (agent/chatbot.py) — new collapsible "Visualizations" panel (gr.HTML, same pattern as the context graph). Charts also render inline in the assistant's chat bubble (Chatbot(sanitize_html=False), each chart a sandboxed iframe). The panel resets per turn and on Clear.

Notes

  • Only the Chart.js path is handled (generic ui:// MCP-UI / MCP Apps host rendering and the postMessage host↔iframe bridge are out of scope). Charts are self-contained interactive (Plotly native zoom/pan/hover); each inlined chart carries the full plotly.js bundle (~3.5 MB).

[0.3.1.35] - 2026-05-21

Fixed (round-2 agent quality)

  • Feature count wrong on non-overview SEARCH turns (agent/consumer_agent.py, synthesize_answer) — the LLM was computing its own count from context (e.g. "44 features" on entity-scoped turns) despite n_features being in the skill result. The synthesizer now quotes the exact value inline ("The total registered feature count is EXACTLY N") so the LLM has no room to invent a different number.
  • Dashboard query uses metric columns as identifiers (agent/consumer_agent.py, synthesize_answer) — when skill_compose_dashboard returns dashboard_intent=True, the synthesizer now builds a structured SQL template that places entity_keys in the SELECT's identifier columns and feature_names as measure columns, with an ORDER BY <ranking_metric> DESC clause. The LLM no longer picks arbitrary metric columns (e.g. provider_lifetime_avg_amount) for the provider/pharmacy slot.
  • Overview answers truncated to 4 sections (agent/consumer_agent.py, synthesize_answer) — post-LLM hard truncation when len(answer) > 1500 and is_overview=True. Splits on markdown headings and appends a "truncated" note. The existing soft prompt cap was ignored by ministral-14b.
  • $$\nYES\n$$ LaTeX block rendered (agent/consumer_agent.py, EXPLAIN direct-answer path) — calculation_latex is now validated before wrapping in $$ ... $$. Strings that contain no LaTeX-ish characters (=, +, -, \, ^, _, (, ), *, /) are silently skipped instead of rendered as $$\nYES\n$$.
  • Group-pronoun freshness empty after SEARCH turns (agent/graph.py, _capture_feature_group_from_search) — when SEARCH matches contain no type:'feature' entries (LLM emitted prose), the helper now falls back to scanning available_context for Feature: <NAME> lines and intersects against _known_catalog_names() to populate resolved_feature_list. Fixes turn-14 pharmacy freshness after a SEARCH turn.

Added (documentation pipeline — formula-grounded documentation)

  • _extract_interpretation_hints(sql) (genai/documentation.py) — full-text regex scan of the entire SQL string (WHERE, GROUP BY, subqueries, CTEs included) for ABS(, 1.0 -, STDDEV, PERCENTILE_CONT, SIGN(, and aggregation functions. Returns a dict of boolean signals passed to the LLM as a constrained context section. Avoids the AST-only approach which misses formulas embedded in subqueries.
  • Formula-grounded prompt (skills/qo-skill-explain/prompts/qo-document-sql.md) — upstream column docs are now "context, not authority." New FORMULA CONSISTENCY RULES section maps each interpretation_hints signal to a mandatory prose constraint (is_absolute_value → never use "signed"; is_inverted → explicitly state direction; etc.).
  • Post-validation gate with retry (genai/documentation.py, document_sql_query_columns) — after LLM returns descriptions, _find_hint_violations checks for prose that contradicts active hints (e.g. "signed" when is_absolute_value=True). On violation: one retry with explicit correction note in context. On persistent violation after retry: description prefixed with [FORMULA CONSISTENCY WARNING: ...] and logged as error.
  • migrate_documentation_columns_schema() (genai/documentation.py) — idempotent SHOW TABLE-based migration that adds MONOTONIC_DIRECTION, VALUE_DOMAIN, IS_ABSOLUTE_VALUE, and FORMULA_CONSISTENCY_WARNING columns to FS_BUSINESS_DICTIONARY_COLUMNS. Same pattern as migrate_documentation_explain_schema.
  • document_view_incremental(database, view_name, ...) (genai/documentation.py) — public entry point to document any Teradata view and its upstream dependencies, even when the view is not registered in the process or dataset catalog. Mirrors the inner loop of document_process_incremental but seeded from an arbitrary SELECT * FROM {db}.{view}. Exposed on tdfs4ds.genai.
  • Refresh notebook (notebook dev/genai/14 - Refresh Feature Documentation.ipynb) — three-cell workflow: (1) schema migration, (2) document_feature_store_incremental(force_update=True) for all catalog objects, (3) optional loop over additional non-catalog views using document_view_incremental. Includes a shared visited set so multi-seed runs don't re-document shared dependencies.

[0.3.1.34] - 2026-05-21

Fixed

  • MCP astream_events crash on multi-feature dashboard questions (agent/consumer_agent.py, _skill_mcp_tools_async) — data["output"] for inner-chain on_chain_end events is sometimes a list[BaseMessage], not a dict, so data.get("output", {}).get("messages", []) raised 'list' object has no attribute 'get' and killed the React loop. Output is now defensively unwrapped (dict, list, or other) and the top-level chain is gated via metadata.langgraph_node so intermediate tool messages no longer clobber the final answer.
  • skill_compose_dashboard ignores entity keys and partial matches (agent/consumer_agent.py) — _extract_feature_list_from_question previously returned a case-sensitive feature list only, so questions like "top 50 highest-risk claims with risk band, top driver, provider and pharmacy" matched <2 features and fell through to a literal single-feature search. The helper now returns (features, entities) (case-insensitive, parses ENTITY_ID for entity-key tokens), and skill_compose_dashboard handles the single-feature path by delegating to _skill_dataset_single with dashboard_intent=True and entity_keys set.
  • SEARCH does not populate resolved_feature_list (agent/graph.py, node_skill_search) — only SUMMARY captured the feature group, so follow-up group-pronoun freshness ("are these pharmacy features fresh?") had nothing to aggregate after a SEARCH turn. New _capture_feature_group_from_search() helper lifts feature names from skill_search matches and skill_compose_dashboard results into resolved_feature_list, and resolves resolved_entity_name when all matched features share one entity.
  • Hallucinated feature counts in overview answers (agent/consumer_agent.py, skill_search + synthesize_answer) — the LLM was counting its own bullets (52 in one turn, 125 in another for the same catalog). skill_search now emits catalog-grounded n_features and n_features_by_entity, and the synthesizer prompt instructs the LLM to quote those numbers verbatim instead of computing its own.
  • MCP React agent hallucinates feature/dataset names (agent/consumer_agent.py, _skill_mcp_tools_async) — the MCP system prompt previously listed dataset names only, leaving the React loop free to invent feature names (claim_amount_zscore_provider, ds_scoring observed in earlier runs). The prompt now lists every valid FEATURE_NAME from the catalog and forbids invented variants.
  • Lineage / EXPLAIN speculate about source-table provenance (agent/consumer_agent.py, skill_lineage + _skill_explain_multi) — when a source table has no business description, the LLM was inventing facts ("synthetic 20,000 records" vs "real healthcare claims"). Both system prompts now strictly forbid inventing facts about undocumented sources.
  • Overview SEARCH ignores AGENT_ANSWER_STYLE='standard' (agent/consumer_agent.py, synthesize_answer) — turn-1 overview answers were 1500+ words. A hard 200-word / 4-bullet-group budget is now enforced when the question matches overview cues (overview, what's available, new here, vue d'ensemble, ...).

Added

  • Source citations on FRESHNESS and LINEAGE answers (agent/consumer_agent.py) — skill_freshness, skill_freshness_for_feature_list, and skill_lineage now emit a structured sources dict (process_ids, follow_up_method, seed_query, ddl_objects, edges_total, ...). New _render_sources_footer() in synthesize_answer appends a **Sources** — ... line that lets the user verify or escalate the claim. Suppressed under AGENT_ANSWER_STYLE='concise' to keep short answers short.

[0.3.1.33] - 2026-05-21

Added

  • AGENT_ANSWER_STYLE global (__init__.py) — controls verbosity of every consumer-agent response ('concise' | 'standard' | 'verbose'). Threaded into every _llm_call via a new _style_prefix() helper in agent/consumer_agent.py.
  • skill_freshness_for_feature_list (agent/consumer_agent.py) — new helper that aggregates freshness across a list of feature names. node_skill_freshness (agent/graph.py) now detects group pronouns ("these features", "the ones I just saw", "ces features") and, when a multi-feature list is remembered from a prior turn, routes the question to this aggregator instead of failing to resolve a single object.
  • skill_compose_dashboard (agent/consumer_agent.py) — new SEARCH-side composition path. When the question looks like a dashboard request (mentions dashboard / top N / SELECT and references multiple feature concepts), an LLM call extracts the implied feature names from the catalog, then re-uses multi-feature skill_dataset to find datasets exposing the whole set (or report a JOIN gap when none does). Wired in node_skill_search (agent/graph.py).
  • EXPLAIN formula completeness check (agent/consumer_agent.py) — new _extract_formula_identifiers, _extract_ddl_tables, and _validate_source_columns helpers. After skill_explain calls the LLM, every identifier in the formula is cross-checked against the LLM's source_columns list (missing ones appended with source_table='UNKNOWN') and each claimed source_table is annotated with source_confidence (high when it appears in the view's FROM/JOIN set, low otherwise).
  • _known_catalog_names helper (agent/consumer_agent.py) — returns the union of registered feature, dataset and process-view names. Used to drop unverified hits returned by the vector index and to filter LLM-generated search matches so fabricated feature names never reach the user. Dropped names are logged and returned as dropped_unverified on the skill result.
  • detect_data_domain_for_object(..., prefer_domain=) parameter (agent/consumer_agent.py) — when an object name exists in multiple DATA_DOMAINs, returns the previously-resolved one if it is among the matches. This eliminates silent domain switching on ambiguous names. node_detect_domain now passes the sticky current_resolved_domain as the preference.

Fixed

  • DEFINITION silently fails for near-miss feature names (agent/consumer_agent.py, skill_definition) — when the requested feature was not found in any catalog and the vector index returned nothing, the skill returned a generic "No documentation found" message. It now runs _fuzzy_match_feature first and returns feature_not_found_did_you_mean with up to 3 candidates so the synthesiser surfaces the near-matches (handles cases like provider_z_score → provider_amount_zscore / pharmacy_z_score).
  • USAGE answers list only one dataset, no SELECT (agent/consumer_agent.py, skill_usage) — extended to enumerate all datasets that expose the feature and to emit a paste-ready SELECT <entity_keys>, <feature> FROM <db>.<dataset> for each. Entity-key columns are derived from feature_catalog.ENTITY_ID so the projection is join-ready. The new datasets field on the skill result carries this information for downstream consumers.

[0.3.1.32] - 2026-05-21

Added

  • MCP_TIMEOUT config variable (__init__.py) — total seconds allowed for one MCP skill call covering connection, tool discovery, and ReAct loop. Default 120 s.
  • Live growing trace panel (agent/chatbot.py) — the "Agent trace" accordion now updates on every step: each graph node appends a row to a live markdown table (# | Step | Detail) instead of remaining blank until the answer arrives. MCP tool calls appear as indented sub-rows (↳ MCP · tool_name | input). The final structured trace still replaces the live table when the answer arrives.
  • MCP tool-call side-channel (agent/consumer_agent.py, agent/graph.py) — a threading.local() buffer (_mcp_trace_local) captures on_tool_start/on_tool_end events from the MCP ReAct agent; consumer_agent_stream drains the buffer after the MCP node and emits ("mcp_tool", tool_name, tool_info) events for the live trace.

Fixed

  • MCP calls hang indefinitely (agent/consumer_agent.py) — client.get_tools() had no timeout; agent.ainvoke() had no end-to-end timeout. Both are now wrapped with asyncio.wait_for (30 s for get_tools, MCP_TIMEOUT for the full ReAct loop). The non-Jupyter asyncio.run path was also missing a timeout; both paths now go through ThreadPoolExecutor.result(timeout=MCP_TIMEOUT + 5).
  • MCP ainvoke replaced with astream_events (agent/consumer_agent.py) — enables per-tool-call event capture for the live trace without breaking the existing answer extraction.

Changed

  • Trace accordion open by default (agent/chatbot.py) — changed from open=False to open=True so the live trace is immediately visible without a click.
  • consumer_agent_stream step events now 4-tuples (agent/graph.py) — ("step", node_name, partial_state, step_num) replaces the old 3-tuple; step_num is a 1-based counter used by the live trace row formatter.

[0.3.1.31] - 2026-05-21

Fixed

  • Redundant "Dataset availability" section on multi-feature questions (agent/graph.py, node_synthesize) — when a question asked about multiple features and mentioned "dataset", node_synthesize ran a secondary skill_dataset lookup and appended a full "Dataset availability" block even though the primary multi-object answer already contained complete per-feature dataset membership. The secondary block was suppressed when skill_result["_multi_object"] is set, eliminating the duplicate narrative.

[0.3.1.30] - 2026-05-21

Fixed

  • Multi-feature questions crash (agent/graph.py, _build_context_update) — when a skill result contained a context_graph with shared_datasets stored as a list (the format returned by _build_multi_object_context), the code called .items() on it expecting a dict, raising 'list' object has no attribute 'items'. Now handles both list-of-names and dict forms gracefully.

[0.3.1.29] - 2026-05-20

Fixed

  • Graph canvas height (agent/graph.py, _build_context_graph_pyvis) — Network(height=750) was passing an integer; pyvis embeds the value verbatim in its HTML template producing invalid CSS height: 750; (no unit). Changed to Network(height="750px") so the canvas renders at the intended height.
  • Duplicate nodes in context graph (agent/graph.py, _build_context_graph_pyvis) — inconsistent node-ID schemes across code paths caused the same logical object to appear as two disconnected nodes (e.g. a dataset shown once connected and once isolated). Root causes:
  • Feature: feature_{name} (most paths) vs context_feature_{name} (multi-object context_graph path)
  • Source view: view_{name} (triplet path) vs context_view_{name} (context_graph path)
  • Entity: bare "entity" (early state path) vs entity_{name} (accumulated graph paths)
  • Dataset: raw UUID string (early state path) vs dataset_{name} (accumulated + context_graph paths) Fixed by normalizing all node IDs to a single uppercased name-based scheme throughout: feature_{NAME_UPPER}, source_view_{NAME_UPPER}, entity_{NAME_UPPER}, dataset_{NAME_UPPER}. Process nodes keep their UUID-based IDs (already unique). All edge references updated accordingly.
  • Post-processing consolidation (agent/graph.py, _build_context_graph_pyvis) — added a safety-net pass after nodes_dict is built that merges any remaining nodes sharing the same (type, label) pair. Duplicate edges and self-loops created by the merge are removed. This prevents visual duplicates even if future code paths introduce new ID inconsistencies.

[0.3.1.28] - 2026-05-20

Added

  • accumulated_context_graph state field (agent/graph.py) — new persistent AgentState field storing a structural relationship map that grows across conversation turns: features → {entity, domain, process_id, view_name, dataset_names}, processes → {view_name, entity_name, feature_names}, datasets → {dataset_id, database, feature_names, entity_names, process_ids}, entities → {domain, feature_names}, source_views → {referenced_by_process_ids}. Persisted via MemorySaver so it survives turn boundaries.
  • _merge_accumulated_context helper (agent/graph.py) — non-destructive deep merge for updating the accumulated structural map without overwriting existing entries.
  • _build_context_update helper (agent/graph.py) — extracts structural relationship updates from skill results (single-feature triplet path and multi-feature context_graph.features path) into the accumulated map format.
  • Context-first navigation in skill_definition (agent/consumer_agent.py) — before running the full feature catalog scan, checks if the feature is already mapped in accumulated_context_graph. If found, navigates directly to the known view via targeted BD_COLUMNS/BD_SECTIONS queries, skipping the catalog scan entirely. Trace shows ACCUMULATED_CONTEXT_MAP as a docs source.
  • Context-first navigation in skill_freshness (agent/consumer_agent.py) — checks accumulated_context_graph.features[X].process_id before querying feature_catalog → process_catalog. When process_id is already known, calls follow_up_report() directly.
  • Context-first navigation in _build_multi_object_context (agent/consumer_agent.py) — per-feature check against accumulated map; skips per-feature catalog scan and view/dataset lookups for already-mapped features.
  • Dataset membership lookup in skill_definition (agent/consumer_agent.py) — after resolving a single feature's triplet, looks up which datasets include that feature (via get_dataset_features() + dataset_catalog()), unless already present in the accumulated map. Returns resolved_dataset_id and dataset_name for graph rendering.
  • Process nodes in context graph (agent/graph.py, _build_context_graph_pyvis) — Process nodes rendered from accumulated_context_graph.processes with light sea green color (#20B2AA) and hexagon shape. Edges: Entity → Process and Feature → Process (replaces raw "Feature → View" for process views). Legend updated with Process entry.

Changed

  • Full state for graph visualization (agent/graph.py, consumer_agent_stream) — graph visualization now reads graph.get_state(config).values (full MemorySaver-persisted state) instead of current-turn deltas, so nodes from all prior turns remain visible in the context graph.
  • Pyvis tooltip plain text (agent/graph.py, _build_context_graph_pyvis) — replaced HTML tags (<b>, <br>) in title attributes with plain text (\n separators) since vis.js renders title as plain text, not HTML. Previously showed raw HTML markup on hover.
  • Graph panel height (agent/graph.py) — increased Network(height=...) and iframe height from 500px to 750px for better visibility of complex graphs.
  • Dataset nodes from multi-feature context graph (agent/graph.py, _build_context_graph_pyvis) — now reads cg_feat["datasets"] when processing context_graph.features, adding Dataset nodes (purple triangle) with edges to their features.
  • Accumulated context written after each skill (agent/graph.py) — node_skill_definition, node_skill_freshness, and node_skill_summary all call _build_context_update + _merge_accumulated_context and persist the result in AgentState after each successful skill execution.
  • "Source view" legend label (agent/graph.py) — renamed "Computed by view" → "Source view" in the graph legend to distinguish raw/lineage views from process views and dataset views.

Technical Details

  • accumulated_context_graph keys are uppercased for case-insensitive lookup (e.g., features["FEAT_NAME"]); canonical casing is stored inside the value dict.
  • View name is stored directly in feature entries (denormalized from processes) so skills can navigate features[X]["view_name"] in one step.
  • Process view names label Process nodes (readable) with process_id in the tooltip; no separate "Process → View" edge since the view_name is the label.
  • Dataset views and process views are kept in separate sections; raw/lineage views (from LINEAGE/EXPLAIN traversal) go in source_views.

[0.3.1.25] - 2026-05-20

Changed

  • Context graph visualization (agent/graph.py, agent/chatbot.py) — replaced static Plotly scatter plot with dynamic Pyvis network graph. Nodes are now draggable with physics-based layout, smooth edge arrows, and physics stabilization (150 iterations, Barnes-Hut gravity). Rich HTML hover tooltips on all nodes (type + name) and edges (relationship description: "entity has feature", "dataset exposes feature", etc.). HTML legend table below the graph shows all 6 node types with color swatches. Graph wrapped in <iframe srcdoc=...> for sandboxed rendering.
  • Dataset support in DEFINITION, LINEAGE, and FRESHNESS skills (agent/consumer_agent.py) — extended all three skills to handle registered dataset names as query objects:
  • DEFINITION: Quaternary path now calls _resolve_dataset() and _get_dataset_with_features() after BD_OBJECTS/BD_SECTIONS/BD_COLUMNS checks succeed, so dataset documentation is returned alongside business dictionary entries.
  • LINEAGE: New dataset branch at start of view-resolution section resolves the object as a dataset and uses dataset_name as the seed view for dependency graph construction. Returns resolved_dataset_id, resolved_dataset_features in result so graph builder can render dataset node with connected features.
  • FRESHNESS: New dedicated dataset branch (before feature catalog lookup) resolves the object, collects unique process_ids from dataset features, calls follow_up_report() for each, and aggregates per-entity freshness (entity name → feature count → last run date → status). Returns dataset-level reliability assessment.
  • AgentState multi-turn context (agent/graph.py) — added resolved_dataset_name: Optional[str] to persist the human-readable dataset name alongside resolved_dataset_id. Extended _skill_with_triplet() to propagate dataset context (resolved_dataset_id, resolved_dataset_features, resolved_dataset_name) from skill results to state, so all skills (not just SUMMARY) persist dataset context for follow-up questions. Updated node_skill_summary to also set resolved_dataset_name.
  • Dataset node rendering (agent/graph.py, _build_context_graph_pyvis) — dataset nodes now use resolved_dataset_name as the label (human-readable) instead of 8-char UUID prefix. Feature nodes referenced by resolved_dataset_features are now created in nodes_dict with proper edges. Entity nodes from dataset features are also added to the graph when available, showing the full entity→feature hierarchy within the dataset context.

Technical Details

  • No new temporal SQL introduced; all dataset and process catalog access reuses existing API functions (_resolve_dataset(), _get_dataset_with_features(), process_catalog(), follow_up_report()) whose CURRENT VALIDTIME semantics are baked into view DDLs.
  • Case sensitivity handled consistently: dataset name matching uses .upper() on both sides; seed SQL for lineage uses original catalog case (NOT CASESPECIFIC behavior).
  • Pyvis physics configuration: stabilization.iterations=150, barnesHut.gravitationalConstant=-8000, springLength=120, smooth dynamic edges, directional arrows.

[0.3.1.23] - 2026-05-19

Added

  • Mistral AI embedding provider (agent/embedding.py) — added support for mistralai provider via langchain_mistralai.MistralAIEmbeddings. Set EMBEDDING_MODEL_PROVIDER = "mistralai" to enable. Model defaults to "mistral-embed" but is configurable via EMBEDDING_MODEL_MODEL.

Changed

  • Vector index upsert semantics (agent/vector_index.py) — replaced _add_documents_batched with _upsert_documents_batched to enable update-if-exists behaviour. Re-running build_vector_index(force_rebuild=False) now replaces stale embeddings when documentation is updated, instead of silently skipping rows whose IDs already exist. Upsert is automatic and transparent to the user.
  • Configurable embedding batch delay (agent/vector_index.py, __init__.py, config.py) — added EMBEDDING_BATCH_DELAY global (default 0.0 seconds). Set to a positive value (e.g. 0.5) to throttle embedding batch submissions and prevent vLLM/TEI server overload / panic mode. Delay is applied between batches, never after the final batch. Override via TDFS4DS_EMBEDDING_BATCH_DELAY env var or .env file.

[0.3.1.22] - 2026-05-19

Fixed

  • Embedding model token limit handling (agent/vector_index.py, config.py, __init__.py) — fix chatbot startup error "Input validation error: inputs must have less than 512 tokens" that occurs when business dictionary sections exceed the TEI server's token cap. Added configurable global EMBEDDING_MODEL_MAX_TOKENS (default 450 = 12% safety margin below 512 for intfloat/multilingual-e5-large). New _truncate_text() helper in vector_index.py truncates all text (objects, sections, columns) to max_tokens × 4 chars before embedding. Users can override via TDFS4DS_EMBEDDING_MODEL_MAX_TOKENS env var or .env file for larger-context models. Rebuild the vector index with force_rebuild=True after changing the limit.

[0.3.1.21] - 2026-05-19

Changed

  • Comprehensive consumer agent documentation (docs_src/ai/consumer-agent.md) — expanded the AI capabilities guide with full LangGraph workflow diagram (10 intents + routing), graph singleton pattern explanation, AgentState as a named-entity knowledge graph (all 9 persistent fields documented), MCP integration architecture with governance guards, and hybrid retrieval strategy (structured SQL + vector fallback per skill). Documentation now covers business dictionary 3-level hierarchy (OBJECTS, SECTIONS, COLUMNS), Chroma collections, and per-skill retrieval patterns.
  • LaTeX rendering fix (agent/consumer_agent.py) — strip LLM-added $ delimiters before wrapping in fresh delimiters to prevent double-delimited $$...$$ $$...$$ that KaTeX rejects as red text. LLM now correctly renders EXPLAIN formulas despite sometimes ignoring the "no delimiters" instruction.
  • Chatbot UI improvements (agent/chatbot.py):
  • Added show_copy_button=True to enable copy-to-clipboard on all messages.
  • Fixed session context persistence: thread_id now rotates to a new UUID on "Clear conversation" so LangGraph MemorySaver checkpoint resets and cross-turn state (domain, entity, feature group) is forgotten as expected.
  • Added collapsible "Context knowledge graph" accordion showing interactive Plotly network visualization of accumulated context (domain → entity → features → views → datasets).
  • Context graph visualization (agent/graph.py) — new _build_context_graph_plotly(state) builds a NetworkX directed graph from all context fields and renders as color-coded Plotly network (blue diamond for domain, green square for entity, orange circles for features, grey squares for views, purple triangles for datasets, red nodes for column sources). Graph updates after each turn to reflect new resolutions and multi-object DEFINITION expansions. Returns None when state is empty (no context yet).

[0.3.1.20] - 2026-05-17

Changed

  • Gradio trace panel now renders the multi-object context graph (agent/graph.py, agent/consumer_agent.py) — skill_definition_multi() now includes a context_graph key in its result with per-feature resolution status (found/not found, entity, view, has docs, dataset membership), shared view names, and shared dataset names. The trace panel renders a "Context Graph (multi-object)" section showing exactly which nodes were resolved and which docs were fetched, making it easy to diagnose missing documentation or wrong entity disambiguation.

[0.3.1.19] - 2026-05-17

Changed

  • Multi-object DEFINITION now uses a graph-based shared context (agent/consumer_agent.py, agent/graph.py) — new _build_multi_object_context() helper resolves all requested features in a single catalog pass, then fetches shared documentation (process view sections, dataset overviews) once per unique node rather than once per feature. New skill_definition_multi() function assembles a unified context string and makes a single LLM call that sees entity, process, dataset, and per-feature descriptions simultaneously, enabling genuine cross-referencing. Features without business descriptions are clearly flagged; if nothing is documented the LLM is skipped entirely and a structured "not documented yet" response is returned. The post-hoc consolidation pass in synthesize_answer is removed — it is no longer needed.

[0.3.1.18] - 2026-05-17

Changed

  • skill_definition no longer speculates when no documentation exists (agent/consumer_agent.py) — when context_parts contains only structural metadata (entity name, process view name, catalog boilerplate) with no actual business descriptions, the LLM call is skipped and a clear "registered but not documented yet" message is returned instead. This eliminates hallucinated feature explanations generated from feature names and sibling column names alone.
  • documentation_not_available error replaced with clean message — the raw error key that surfaced in the chatbot UI is now a user-readable "No documentation found for X." explanation, integrating cleanly into multi-object section output.
  • Multi-object DEFINITION consolidation pass (agent/consumer_agent.py, agent/graph.py) — when multiple features are queried in a single DEFINITION request, node_skill_definition flags the result with _multi_object=True and synthesize_answer runs a single consolidation LLM call to merge the independent per-feature sections into one coherent, cross-referenced answer. Sections for undocumented features clearly state they are not documented; the LLM is instructed not to infer their meaning from other sections. Falls back to the concatenated sections if the consolidation LLM call fails.

[0.3.1.17] - 2026-05-17

Added

  • DATA_QUERY intent for MCP data retrieval (agent/graph.py, agent/consumer_agent.py, agent/chatbot.py, skills/ca-data-query/SKILL.md) — the consumer agent can now route sample and aggregation requests (e.g. "show me 20 rows from DS_FRAUD", "how many claims per region?") to the MCP server when mcp_enabled=True. The new intent replaces the vague MCP_TOOLS label with a specific, actionable description that the classifier reliably distinguishes from feature store metadata intents (DEFINITION, FRESHNESS, EXPLAIN). Feature store skills always take priority; MCP is the fallback for data retrieval only.
  • Automatic MCP fallback routing — _route_intent now falls back to DATA_QUERY (instead of the first registered skill) when mcp_enabled=True and the classifier returns an unrecognised intent.
  • Process view guard preserved — _get_forbidden_objects() already blocks direct queries against process views (FS_V_*) and feature storage tables; only registered datasets are queryable via MCP.
  • Step indicator — Gradio chatbot shows "Querying data — DS_X..." during MCP execution.

[0.3.1.16] - 2026-05-17

Fixed

  • Multi-object DEFINITION returns documentation_not_available (agent/graph.py) — when the user named multiple features, datasets, or a mix (e.g. "what is combined_score and risk_band?"), the classifier produced a comma-separated object_name that was passed as a single lookup key. _get_feature_catalog_with_docs did an exact match and found nothing, so all fallback paths failed. node_skill_definition now splits on commas, calls skill_definition once per name, and combines the individual explanations into one response with per-object section headers. Works for features, datasets, and mixed questions.

[0.3.1.15] - 2026-05-17

Added

  • Real-time step indicator in Gradio chatbot (agent/chatbot.py, agent/graph.py) — the chatbot now shows a live progress label as each graph node completes, instead of a frozen interface. Labels include the argument being processed (e.g. ⏳ Looking up definition — **combined_score**..., ⏳ Detecting data domain → **MY_DOMAIN**...).
  • consumer_agent_stream() (agent/graph.py) — new generator variant of consumer_agent that uses graph.stream(stream_mode="updates") to yield one ("step", node_name, partial_state) event per completed node, followed by a ("done", answer, trace_md) event. MemorySaver multi-turn memory is fully preserved.
  • Immediate user-message display — the first yield in _respond() pushes the user question to the chatbot and clears the input box before the agent starts, so the question no longer disappears on Enter.

[0.3.1.14] - 2026-05-16

Added

  • Entity recognition node (agent/graph.py) — new node_recognize_entity inserted between detect_domain and skill dispatch. For each named object (comma-separated), queries the feature catalog, dataset catalog, and process catalog to identify the object type (feature/dataset/ process/unknown). Stores results in recognized_entities state field.
  • Secondary DATASET intent — when a DEFINITION question contains dataset-related keywords (e.g. "same dataset?") and all named objects are features, node_recognize_entity sets secondary_intent = "DATASET". node_synthesize then also runs skill_dataset and appends a dataset-availability section after the definition answer.
  • _recognize_object_type(name) helper in consumer_agent.py — lightweight catalog lookup returning {type, entity_name, view_name, process_id}.

Fixed

  • Documentation lookup gap in skill_definition — when a feature description is stored in FS_BUSINESS_DICTIONARY_COLUMNS against a dataset view (not the process view), the view-scoped lookup in path 1 returned nothing. Path 1 now also runs a broad column-name-only search (COLUMN_NAME = feature_name, any TABLE_NAME) so docs are found regardless of which table name was used when they were written.
  • Catalog-only fallback — when a feature exists in the catalog but has absolutely no documentation (all BD paths return nothing, vector search returns nothing), the agent now returns what the catalog knows (entity, process view) with a note that docs are pending, instead of the generic documentation_not_available error that triggered the "reach out to data team" response.
  • No-doc error message — synthesize_answer documentation_not_available path now produces a factual "not found in feature store, verify spelling" message instead of the unhelpful "reach out to your data team" suggestion. This path only fires when the object is truly absent from every catalog.
  • MCP StructuredTool monkey-patch — _make_guarded_tools now creates proper StructuredTool wrappers instead of patching invoke/ainvoke (which raised "has no field 'invoke'" in Pydantic v2).
  • MCP context-manager — _skill_mcp_tools_async updated to the non-context-manager API (client = MultiServerMCPClient(...); tools = await client.get_tools()) required by langchain-mcp-adapters >= 0.1.0.

Updated skills

  • tdfs4ds/skills/ca-definition/SKILL.md — documents the new resolution chain (step 0 entity recognition, step 1b broad BD_COLUMNS fallback, step 5 catalog-only fallback, from_catalog_only return key). Adds multi-feature + dataset classifier example.
  • tdfs4ds/skills/ca-dataset/SKILL.md — notes secondary-intent triggering from DEFINITION.

[0.3.1.9] - 2026-05-16

Changed

  • MCP transport is now configurable (agent/consumer_agent.py, __init__.py, config.py) — The hardcoded "transport": "sse" has been replaced by a tdfs4ds.MCP_SERVER_TRANSPORT global (default 'streamable_http', the modern MCP standard) settable via TDFS4DS_MCP_SERVER_TRANSPORT in .env or OS env. Use 'sse' for legacy SSE servers.

[0.3.1.8] - 2026-05-16

Changed

  • Classifier examples moved to SKILL.md (agent/consumer_agent.py, tdfs4ds/skills/ca-*/SKILL.md, tdfs4ds/cli.py) — The hardcoded _INTENT_CLASSIFIER_EXAMPLES constant has been removed from Python. Each ca-*/SKILL.md file now contains a ## Classifier Examples section with its own few-shot examples. _read_skill_classifier_examples() in cli.py parses these sections; skill_catalog() exposes them under the "examples" key; _build_classifier_examples() in consumer_agent.py assembles the classifier prompt dynamically. Plugin skills installed via TDFS4DS_SKILLS_FOLDER are automatically included if they have a ## Classifier Examples section.

[0.3.1.7] - 2026-05-16

Changed

  • Multi-feature DATASET lookup (agent/consumer_agent.py, agent/graph.py) — When the user asks "is there a dataset containing feature1 and feature2?", the agent now returns three layers of information: which datasets contain feature1, which contain feature2, and which datasets contain both (intersection). The classifier recognises comma-separated feature names in object_name via two new few-shot examples. skill_dataset delegates to a single-feature path for one name and a new multi-feature path otherwise; the dataset creation flow (Step 1–3) is only triggered for single-feature no-match results.

[0.3.1.6] - 2026-05-16

Added

  • MCP server integration for the consumer agent (agent/consumer_agent.py, agent/graph.py, agent/chatbot.py) — The consumer agent can now delegate questions to an external MCP (Model Context Protocol) server when no feature store skill applies.
  • Configure the MCP endpoint via TDFS4DS_MCP_SERVER_URL in .env or as an OS env var (e.g. TDFS4DS_MCP_SERVER_URL=http://localhost:8000/sse); the variable is auto-loaded into tdfs4ds.MCP_SERVER_URL on import.
  • Toggle at runtime via the new mcp_enabled parameter on consumer_agent() and on the Gradio chatbot's "Enable MCP Tools" checkbox (hidden when no URL is configured).
  • Safety guardrails enforced at both the tool and LLM-prompt level:
    • Forbidden objects: SELECT statements against feature store process views or feature storage tables are refused outright; _get_forbidden_objects() fetches the live lists from the process and feature catalogs at call time.
    • Allowed data access: the sub-agent's system prompt directs it to only query datasets registered in the dataset catalog when feature data is needed.
    • Row cap: all SELECT queries are rewritten to SELECT TOP 20 by _rewrite_select_top20(); a ⚠ notice is appended to the answer when the 20-row limit is reached.
  • Uses langchain-mcp-adapters (MultiServerMCPClient + SSE transport) as a lazy import inside skill_mcp_tools(), so the dependency is optional.
  • Jupyter-compatible async execution: probes asyncio.get_running_loop() and falls back to a ThreadPoolExecutor when a loop is already running.

[0.3.1.3] - 2026-05-15

Changed

  • fs-upload skill (tdfs4ds/skills/fs-upload/SKILL.md) — added mandatory LOCKING ROW FOR ACCESS to the Pattern A SQL template, the partitioning= comment in the upload_features call, and two new rules: access-lock mandate and hyper-segmented ML partition column guidance.
  • fs-dataset skill (tdfs4ds/skills/fs-dataset/SKILL.md) — added rule explaining that a partition column must be registered as a feature via upload_features before it can appear in a dataset (hyper-segmented ML pattern).

[0.3.1.2] - 2026-05-15

Fixed

  • generate_partitioning_clause crashes on unparenthesised PARTITION BY (utils/info.py) — extract_partition_content uses a regex that requires outer parentheses (PARTITION BY (...)). When the user passes a bare string such as 'PARTITION BY RANGE_N(region BETWEEN 1 AND 10 EACH 1, NO RANGE OR UNKNOWN)', the regex returns None and the subsequent len(None) raised TypeError. Fixed by detecting the None result and falling back to extracting everything after PARTITION BY as the content, so FEATURE_ID partitioning is correctly prepended.

[0.3.0.7] - 2026-05-15

Added

  • drop_dataset module-level function (dataset/dataset_catalog.py) — expose DatasetCatalog.drop_dataset as a standalone importable function so callers can write from tdfs4ds.dataset.dataset_catalog import drop_dataset without instantiating the class directly. Accepts dataset_id and an optional schema_name (defaults to tdfs4ds.SCHEMA).

[0.3.0.6] - 2026-05-15

Fixed

  • No feature columns documented in dataset documentation (genai/documentation.py) — _extract_feature_process_map_from_ddl captured the full qualified column alias produced by build_dataset (e.g. provider_velocity_7d__demo_user___vw_feat_claims_) as the feature-name key. _propagate_docs_from_processes then tried to match that full alias against the short names returned by retrieve_documentation (e.g. provider_velocity_7d), which always failed, so every feature ended up in missing_features and the dataset documentation showed "No feature columns documented." Fixed by stripping the __{database}___{view}_ qualifier in _extract_feature_process_map_from_ddl (split on first __) so the map keys are always short business names. Duplicate short names across multiple processes are deduplicated by keeping the first encountered process (any documented process provides the same description).

[0.3.0.5] - 2026-05-15

Fixed

  • Entity columns silently dropped for intermediate views (genai/documentation.py) — _document_intermediate_view hardcoded entity_columns=[] when calling document_sql_query_columns, so intermediate views (e.g. vw_temp_aggregate) were always documented with 0 entity columns regardless of the actual entity definition. Fixed by adding an entity_columns parameter to _document_intermediate_view and threading it through from both call sites: document_process_incremental passes process_info.get('ENTITY_COLUMNS', []) and document_dataset_incremental passes its local entity_columns. Matching is case-insensitive (uppercased set lookup) while preserving the column casing from the view's own column list.

[0.3.0.3] - 2026-05-15

Fixed

  • Missing skills/*/prompts/*.md in installed package (setup.py) — the package_data glob only included skills/*/SKILL.md (top-level Claude Code skill descriptions). Nested LLM prompt files under skills/*/prompts/*.md were never packaged, so a non-editable pip install . produced a site-packages/tdfs4ds/skills/ tree with no prompts/ subdirectory. Every skill_prompt("qo-summary"), skill_prompt("qo-explain"), etc. raised FileNotFoundError: No bundled skill named 'qo-summary'. Fixed by adding 'skills/*/prompts/*.md' to package_data.

[0.3.0.2] - 2026-05-15

Fixed

  • _generate_dataset_description AIMessage bug (genai/documentation.py) — the vLLM and fallback chains piped the LLM output directly into _robust_json_parser, which expects a plain str. LangChain passes an AIMessage through the | operator, causing a ValueError: Expected string response, got AIMessage. Fixed by wrapping with RunnableLambda(_parse) that extracts .content first — the same pattern already used in build_sql_documentation_chain and build_explain_documentation_chain.

[0.3.0.1] - 2026-05-15

Changed

  • BUILD_DATASET entity clarification fallback (graph.py, consumer_agent.py) — when the feature catalog returns no features for the active domain, the agent no longer dead-ends immediately. Instead it enters a new awaiting_entity_clarification pending state and asks the user to name the entity or subject area (e.g. "Customer", "Transaction"). The user's answer is used to retry _get_feature_catalog_with_docs(entity_filter=…) and, if features are found, the flow continues normally to feature proposal. Only when the retry also returns nothing does the agent report no_features_found with a precise "no features matching X" message. The hard-stop message now includes the entity hint for better diagnostics.
  • _DATASET_PENDING_ACTIONS (graph.py) gains "awaiting_entity_clarification" so the LLM classifier short-circuit correctly routes the clarification reply back to node_skill_build_dataset.
  • ca-build-dataset/SKILL.md updated to document the new awaiting_entity_clarification pending state and the two-attempt strategy.

[0.3.0.0] - 2026-05-15

Added

  • BUILD_DATASET consumer-agent intent — a new 9th intent that lets a business user ask the agent to autonomously create a feature store dataset from a plain-language use-case description (e.g. "build me a dataset for churn analysis"). No feature name is required to start the flow.
  • tdfs4ds/skills/ca-build-dataset/SKILL.md — skill file for the new intent, auto-discovered by consumer_agent_skill_catalog() (uses the ca- prefix convention). Registers BUILD_DATASET into the dynamic intent classifier and graph without any hardwired lists.
  • resolve_features_for_dataset(entity_name, feature_list) (consumer_agent.py) — resolves a list of {name, description} feature dicts to the selected_features dict format required by build_dataset(). Calls get_feature_versions() and splits results into a clean {feature_name: uuid} dict (single-version features, ready for build_dataset()) and a conflicts list ([{feature_name, versions: [{process_id, process_view_name}]}]) for features with multiple registered process IDs.
  • skill_build_dataset(entity_name, selected_features, dataset_name, ...) (consumer_agent.py) — end-to-end dataset creation skill: calls build_dataset() then, when an LLM provider is configured, auto-calls document_dataset_incremental(). Returns status=dataset_created with a doc_status field (success / failed / skipped), or status=creation_failed on error. Accepts return_query=True for dry-run / unit-test use without a live connection.
  • node_skill_build_dataset(state) (graph.py) — LangGraph node implementing a 4-step autonomous creation flow:
  • Step 0 (no pending): resolves entity from the catalog (uses the feature-richest entity as a heuristic when no feature name is given), calls skill_propose_features() immediately to propose the top-20 relevant features for the use case.
  • Step 1 (awaiting_feature_confirmation): user replies "ok" or "all"; calls resolve_features_for_dataset() to detect version conflicts.
  • Step 2 (awaiting_version_selection): if conflicts exist, presents options (numbered) and waits for the user to pick a version per feature; parses replies by number or view-name substring.
  • Step 3 (awaiting_dataset_name): validates the proposed name with skill_check_dataset_name(); on success immediately calls skill_build_dataset() and creates the Teradata view + docs.

Changed

  • IntentResult Literal extended to include "BUILD_DATASET" (hardwired fallback model, consumer_agent.py).
  • Intent classifier examples gain three BUILD_DATASET examples (English and French) to disambiguate from the existing DATASET intent.
  • _DATASET_PENDING_ACTIONS (graph.py) gains "awaiting_version_selection" so the LLM classifier short-circuit also covers the version-conflict resolution turn.
  • _node_classify_impl short-circuit (graph.py) now preserves the active intent instead of forcing "DATASET" — BUILD_DATASET flows remain correctly routed to their own node across all pending turns.
  • synthesize_answer() (consumer_agent.py) gains a full BUILD_DATASET branch covering all status codes: features_proposed, features_confirmed, version_conflict, name_available, name_taken, dataset_created, creation_failed, no_features_found.
  • _INTENT_NODE_MAP and _SKILL_LABEL (graph.py) both include "BUILD_DATASET".

[0.2.9.9] - 2026-05-13

Added

  • 11 qo-* prompt files reorganized into nested prompts/ subfolders under their owning qo-skill-* directories (e.g. qo-skill-explain/prompts/qo-explain.md instead of standalone qo-explain/SKILL.md). Reduces skill namespace clutter while keeping related prompts together.

Changed

  • skill_path() in cli.py now searches nested prompt paths: When a flat SKILL.md is not found, it searches <skill>/prompts/<name>.md, allowing skill_prompt('qo-explain') to resolve transparently from the new nested location.
  • install_skills() in cli.py copies prompts/ subfolders: When installing skills with tdfs4ds-install-skills(), the prompts/ subfolder (if present) is copied alongside SKILL.md so relative links in skill bodies resolve correctly.
  • qo-skill-*/SKILL.md bodies rewritten with ## Execution sections: Each now documents its full execution pipeline, references prompt files with relative [prompt-name](prompts/prompt-name.md) links, and explains inputs/outputs, matching the discoverable-skill pattern from ca-* and fs-* skills.
  • Argument-hint bracket format fixed on qo-skill-* SKILL.md ([...] instead of <...> for consistency with other skills).
  • All fs-* SKILL.md files now have name: frontmatter field with human-readable Feature Store: <Action> naming (e.g. name: Feature Store: Upload), consistent with Consumer Agent: * and Query Optimizer: * conventions.
  • Query optimizer report naming changed to timestamp-first (tdfs4ds_optim_{timestamp}_{view}_{process_id}.md instead of tdfs4ds_optim_{view}_{process_id}_{timestamp}.md). Reports now sort naturally by date in file listings.

[0.2.9.8] - 2026-05-13

Added

  • 8 consumer-agent SKILL.md files (tdfs4ds/skills/ca-*/SKILL.md): Each of the 8 consumer-agent intents (SEARCH, DEFINITION, USAGE, FRESHNESS, SUMMARY, LINEAGE, EXPLAIN, DATASET) is now a first-class Claude Code skill in tdfs4ds/skills/ with compliant frontmatter (name, intent, description, argument-hint, allowed-tools: Read, Grep, Glob). Skills are identified by the ca- name prefix convention. They serve dual purpose: Claude Code users can invoke them as reference skills, and build_graph() reads them at runtime to configure the consumer agent dynamically. Bundled via package_data.
  • consumer_agent_skill_catalog() (cli.py / top-level API): Returns metadata for all ca-* skills (filtered by name prefix), keyed by skill name. Used by build_graph() to discover available intents at runtime without any hardwired lists.
  • reset_consumer_agent() (tdfs4ds/agent/graph.py): Clears the compiled-graph singleton cache so the next consumer_agent() call picks up any newly added or removed ca-* SKILL.md files.
  • _build_intent_model(intents) (tdfs4ds/agent/consumer_agent.py): Helper that builds a dynamic Pydantic model for structured intent classification output using Union[Literal["A"], Literal["B"], ...] (equivalent to Literal["A", "B", ...] per PEP 586) and pydantic.create_model.

Changed

  • consumer_agent() gains skills= parameter (tdfs4ds/agent/graph.py): Pass a list of intent names (e.g. ['SEARCH', 'SUMMARY']) to restrict the agent to a subset of skills. Different subsets use separate compiled-graph instances (and separate MemorySaver state), so thread_id namespacing is preserved per skill set.
  • build_graph() is now data-driven (tdfs4ds/agent/graph.py): Graph construction, routing, and the intent-classifier system prompt are all built dynamically from consumer_agent_skill_catalog() instead of being hardwired. Adding or removing a ca-* SKILL.md file (and calling reset_consumer_agent()) changes the available intents without any Python edits.
  • intent_classifier() accepts dynamic model and descriptions (tdfs4ds/agent/consumer_agent.py): New optional parameters intent_model and intent_descriptions allow the graph to inject a dynamically constructed Pydantic model and per-intent descriptions. Falls back to the hardwired IntentResult and static descriptions when called without arguments (backward compatibility).
  • install_skills() installs all 19 skills (cli.py): ca-* skills are valid Claude Code skills and are now installed alongside fs-* skills. The total bundled skill count increases from 11 to 19.
  • fs-agent/SKILL.md updated (tdfs4ds/skills/ and .claude/skills/): The hardwired intents table is replaced with a reference to the ca-* SKILL.md files and documentation for the new skills= parameter and consumer_agent_skill_catalog().

[0.2.9.7] - 2026-05-13

Added

  • build_dataset grouped query strategy (grouped=True) (dataset/builder.py): A new grouped parameter (default False) activates a MAX(CASE WHEN FEATURE_ID=… THEN FEATURE_VALUE END) … GROUP BY pivot strategy instead of the default one-sub-query- per-feature multi-join approach. Features sharing the same (table, process_id) pair are collapsed into a single sub-query, reducing the number of JOINs from N (one per feature) to one per unique source group. Single-feature groups emit a plain FEATURE_ID = N scan (no GROUP BY overhead). INNER JOIN semantics are preserved via a HAVING clause that rejects entities missing any feature in the group. Works with both snapshot (feature_store_time=False) and time-travel (feature_store_time=True) modes — the appropriate temporal qualifier (SEQUENCED VALIDTIME or VALIDTIME AS OF TIMESTAMP '…') is injected inside each sub-query.
  • Demo notebooks 20 and 21 (demos/notebooks/):
  • 20 - Build a Dataset with Grouped Queries (snapshot).ipynb — mirrors notebook 03 with grouped=True, SQL inspection section, sub-query count comparison, and row-count equivalence validation.
  • 21 - Build a Dataset in the past with Grouped Queries (snapshot).ipynb — combines grouped=True with feature_store_time=True, confirming both VALIDTIME AS OF TIMESTAMP and MAX(CASE WHEN) appear in the generated DDL.
  • fs-dataset skill updated (tdfs4ds/skills/fs-dataset/SKILL.md): The skill now covers three parameters that were previously missing: grouped (the MAX(CASE WHEN) pivot strategy added in this release), join_type ('INNER' vs 'FULL OUTER'), and return_query (inspect the generated DDL before executing). The code template shows all three as commented-out options; the Rules section explains when to prefer grouped=True and when join_type='FULL OUTER' is appropriate.
  • td-explain skill synced to .claude/skills/ — the skill existed in tdfs4ds/skills/td-explain/ but was missing from the project-level .claude/skills/ directory.
  • Claude Code Skills documentation (README.md, docs_src/getting-started/installation.md, docs_src/skills/overview.md): All three surfaces now document the Python API and CLI entry point for skill installation, explain the difference between user-level (~/.claude/skills/) and project-level (.claude/skills/) targets, and describe the two teammate sharing patterns (git-committed .claude/skills/ vs per-venv tdfs4ds-install-skills). The README gains a new ## Claude Code Skills section with the full skill catalogue table.

Changed

  • export_skills default overwrite changed from True to False (cli.py): export_skills is the "add missing skills" function; its default should skip files that already exist at the destination. Pass overwrite=True explicitly for unconditional replacement. install_skills retains overwrite=True as its default (upgrade semantics).

Fixed

  • build_dataset return_query=True was executing side effects (dataset/builder.py): The early-return for return_query=True was placed after tdml.execute_sql(query) and the dataset catalog registration calls. Every "inspect SQL" invocation silently re-created the view and re-registered catalog entries. Fixed by moving the early-return to immediately after query generation, before any DDL execution.

[0.2.9.6] - 2026-05-12

Fixed

  • build_dataset time-travel (feature_store_time=True) now works correctly (dataset/builder.py): Thorough MCP-validated investigation found two separate Teradata constraints:
  • VALIDTIME AS OF TIMESTAMP '…' cannot be placed before AS in REPLACE VIEW DDL (Teradata requires AS immediately after the view name — Error 3707).
  • An outer temporal qualifier placed after AS LOCK ROW FOR ACCESS fails with Error 9330 ("ValidTime qualifier requires at least one table with ValidTime") because the qualifier cannot reach temporal tables that are wrapped inside derived-table sub-queries. Fix: mirror the existing SEQUENCED VALIDTIME pattern exactly — place VALIDTIME AS OF TIMESTAMP '…' inside each sub-query directly against the temporal feature table, and emit no outer temporal qualifier (the outer view body is a plain SELECT). Verified against Teradata 20.0.0 via EXPLAIN before committing.

[0.2.9.5] - 2026-05-12

Fixed

  • build_dataset time-travel fix (superseded by 0.2.9.6) — incorrectly assumed outer qualifier propagates into derived-table sub-queries; replaced immediately.

[0.2.9.4] - 2026-05-12

Fixed

  • build_dataset time-travel fix (superseded by 0.2.9.5) — incorrect approach, replaced immediately.

[0.2.9.3] - 2026-05-12

Fixed

  • upload_features fails with AttributeError: 'NoneType' has no attribute 'fetchall' (feature_store/feature_store_management.py, utils/query_management.py): A refactor of execute_query added a fetch_results parameter (default False) and stopped returning the cursor on the modern teradataml path when fetch_results=False. GetTheLargestFeatureID (called during register_features inside upload_features) still chained .fetchall() directly on the return value, which is now None. Fixed by passing fetch_results=True to that call site.
  • execute_query error handler references undefined tdfs4ds.DEBUG — the attribute is DEBUG_MODE. Would raise AttributeError whenever a SQL error triggered the handler. Fixed to use tdfs4ds.DEBUG_MODE.

[0.2.9.2] - 2026-05-12

Fixed

  • Infinite recursion / Jupyter DeadKernelError when accessing tdfs4ds.genai (tdfs4ds/__init__.py): The __getattr__ lazy-loader used from . import genai, which triggers importlib._handle_fromlist. That function calls hasattr(tdfs4ds, 'genai') to check whether the attribute already exists; since it does not yet, Python re-enters __getattr__('genai'), which issues another from . import genai, producing unbounded recursion. With Python's default 1000-frame limit the resulting RecursionError can crash the Jupyter kernel process before the surrounding try/except in setup() can catch it, surfacing as DeadKernelError. Fixed by writing a None placeholder into tdfs4ds.__dict__['genai'] before the actual import starts. _handle_fromlist finds the attribute already set and skips the recursive lookup. After importlib.import_module('tdfs4ds.genai') returns, the placeholder is replaced with the real module; on failure it is removed and the exception re-raises normally.
  • The bug was masked in ≤ 0.2.9.0 by a typo: setup() called genai.documentations_tables_creation() (extra s), which raised AttributeError before the recursion loop could start. Once the typo was corrected in 0.2.9.1 the loop became reachable, causing the kernel crash.

[0.2.9.1] - 2026-04-30

Fixed

  • Silent VARCHAR truncation in documentation uploads (genai/documentation.py): tdml.copy_to_sql calls for staging volatile tables in upload_documentation, upload_business_dictionary_objects, upload_business_dictionary_columns, and upload_business_dictionary_sections did not pass explicit column types. teradataml inferred VARCHAR sizes from the dataframe and could default to a width far smaller than the destination's VARCHAR(32000), silently truncating long values (notably ENTITY_COLUMNS_JSON, BUSINESS_LOGIC_DESCRIPTION, ENTITY_DESCRIPTION, FEATURE_DESCRIPTION, BUSINESS_DESCRIPTION, SECTION_CONTENT) before the MERGE into the destination tables. The truncated value was then unparseable as JSON when read back, surfacing as "Failed to parse JSON from LLM response" in retrieve_documentation and a malformed ENTITY_COLUMNS_JSON warning.
  • All five staging-table copy_to_sql calls now pass explicit types={...} with tdml.VARCHAR(length=32000, charset='LATIN') for description/JSON columns and matching widths for identifier columns.
  • Backward compatibility: no destination DDL change required — destination tables were always created with VARCHAR(32000). The bug was strictly at the staging layer (volatile, recreated on every call), so no schema migration is needed. Rows previously written with truncated values must be repaired by re-running the upload, e.g. document_process_incremental(process_id=..., force_update=True).

[0.2.9.0] - 2026-04-17

Added

  • EXPLAIN execution metrics tracked as first-class signals (documentation.py): every EXPLAIN analysis now extracts two deterministic counters from the raw Teradata EXPLAIN text and surfaces them alongside the 1–5 scores.
  • n_steps — number of numbered execution steps in the plan (e.g. 1) First, ...).
  • n_spool_objects — count of distinct Spool N objects referenced in the plan (intermediate materialisations).
  • Parsed by _parse_explain_metrics(); both values returned by document_sql_query_explain() and run_explain_documentation().
  • Stored in FS_PROCESS_DOCUMENTATION_EXPLAIN via two new columns N_STEPS and N_SPOOL_OBJECTS (auto-migrated on first upgrade by documentation_tables_creation() / migrate_documentation_explain_schema() — older rows show NULL until re-documented).
  • Returned by retrieve_explain_documentation() and propagated through get_process_info() as EXPLAIN_N_STEPS / EXPLAIN_N_SPOOL_OBJECTS.
  • Rendered in the HTML documentation panel (_print_documentation) as two new badges next to the User/Overall scores.
  • Score Summary table in optimizer reports now reports four metrics (query_optimizer.py): User score, Global score, EXPLAIN steps, Spool objects — shown as Before → After (or Input → Simplified → Optimized when simplification applied). Negative deltas on steps/spools are highlighted as improvements.
  • Score-vs-cost trade-off note in reports: when a rewrite improves the user score but adds steps or spool objects, the report now appends an explanatory note so reviewers understand the optimizer traded a small amount of intermediate materialisation (typically single-row CTEs) for a genuine structural SQL fix.
  • Per-round metric tracking in skill_generate_candidates: every refinement round records n_steps and n_spool_objects in addition to user_score and remaining_you_items — visible in the per-candidate breakdown of the report.
  • Ground-truth EXPLAIN signal parser _parse_explain_signals() (documentation.py): extracts deterministic signals from the raw EXPLAIN (presence of real product joins, per-table scan counts, data redistribution) used to cross-validate LLM-generated [You] items before scoring.
  • Semantic [You] item classification _classify_you_items() (documentation.py): categorises each [You] item (e.g. real product join vs scalar-subquery false positive) in a single LLM call; items in known false-positive categories are filtered from the deterministic score computation. Regex fallback retained for when the LLM is not configured.
  • classify parameter added to document_sql_query_explain(), run_explain_documentation(), and _compute_user_score_from_you_items(): intermediate scoring in the optimizer uses classify=False to keep the raw [You] signal; the final public-facing score (via document_process_incremental) uses classify=True to filter noise.
  • Deterministic tiebreaker in plan comparison (query_optimizer.py): when two candidates have equal user scores, the optimizer now deterministically picks the one with fewer EXPLAIN steps + Spool objects. Logged as "Tiebreaker: …" with full metric context in the report's reasoning section.
  • Refinement acceptance rule: a refinement round is rejected when the user score is unchanged but steps/spools regressed — ensures the structurally lightest SQL at each score tier is retained.
  • Plan comparison prompt criteria extended: the selection prompt now tells the LLM to prefer candidates with fewer EXPLAIN steps and Spool objects as a tiebreaker when scores are equal (before falling back to readability).
  • Simplification baseline preserved (skill_simplify_query): the full baseline EXPLAIN analysis is now stored in the result dict under _baseline_analysis, enabling the optimizer report's 3-stage table to show pre-simplification steps/spools alongside the user score.

Fixed

  • get_process_info fails when documentation tables return no row (process_store/process_store_catalog_management.py): direct key access (documentation['DOCUMENTED_SQL'], etc.) raised KeyError when retrieve_documentation / retrieve_explain_documentation returned an empty dict. Replaced with .get(...) with safe defaults; the success log line is now gated on a non-empty result so empty-lookup cases no longer claim success.
  • retrieve_explain_documentation schema compatibility: the query now probes the live DDL via SHOW TABLE and only selects N_STEPS / N_SPOOL_OBJECTS when those columns are present — older installs that haven't run the schema migration continue to work.

Changed

  • migrate_documentation_explain_schema() now multi-column (documentation.py): migrates USER_SCORE, N_STEPS, and N_SPOOL_OBJECTS in one DDL-probe pass. Idempotent — only runs ALTER TABLE ADD <col> for columns actually missing from SHOW TABLE output.
  • upload_documentation_explain NULL-safe: all integer metric columns use a shared _int_or_null helper so new EXPLAIN rows write NULL cleanly when a score or counter could not be computed.

[0.2.8.1] - 2026-04-16

Added

  • SQL simplification step (Step 0.5) in skill_optimize_query (query_optimizer.py): before any EXPLAIN analysis the optimizer now runs a structural compaction pass that merges unnecessary nesting layers (multiple derived-table wrappers → CTE + single outer SELECT) and rewrites verbose subquery structures into more readable forms — giving the LLM a cleaner baseline to work with and reducing candidate tokens.
  • skill_simplify_query(sql_query, process_id) — new standalone public function. Returns {"simplified_sql", "original_sql", "simplified", "original_score", "simplified_score"}. Accepts raw SQL or a process_id; runs deterministic sanitisation, baseline EXPLAIN, LLM compaction, then accepts the result only when the simplified score ≥ original score.
  • simplify_query(sql_query, process_id) — synchronous public wrapper (no LangGraph) exported from tdfs4ds and tdfs4ds.agent.
  • _simplify_sql_sync() — LLM helper; adapted from the retired _compact_sql_sync with a new CRITICAL SEMANTIC RULE: scalar subqueries must preserve their exact scope (uncorrelated global aggregates must not gain a correlated WHERE clause).
  • Step 0.5 is recorded in steps as {"step": "simplification", "simplified": bool, ...}.
  • When simplification changes the SQL, the stored EXPLAIN analysis is considered stale and a fresh EXPLAIN is run on the simplified query as the optimizer's new baseline.
  • Report ## Simplification section: present in every report showing whether compaction was applied, the original vs simplified score, and a collapsible block with the simplified SQL when it differs.
  • 3-stage query comparison in reports: Input → After Simplification → After Optimization. The "Input Query" section now always shows the original (pre-simplification) SQL; "After Simplification" and "After Optimization" sections follow.
  • ALWAYS CHECK FOR REDUNDANT TABLE SCANS instruction in the EXPLAIN prompt (documentation.py): mandatory whole-query structural check. If the same source table is scanned more than once with the same GROUP BY key — as separate CTEs, subqueries, or JOIN branches — the LLM must flag it as a [You] warning regardless of per-step efficiency. Example detected: two CTEs both scanning transactions GROUP BY CustomerID that can be merged into a single conditional-aggregation pass.
  • singlepass strategy CTE structure rule (query_optimizer.py): the singlepass technique constraint now explicitly instructs the LLM to place the aggregation in a CTE and compute derived columns (ratios, date arithmetic, alias-referencing expressions) in the outer SELECT — preventing expression duplication when SQL aliases cannot be referenced within the same SELECT level.

Fixed

  • "Query (Unchanged)" label after simplification (query_optimizer.py): when a simplification pass had already improved the SQL but no further optimization candidate was accepted, the report incorrectly showed "Query (Unchanged)". The report now shows "Input Query / After Simplification" with the correct per-stage scores.
  • Missing "Remaining Issues" section at selected_idx = -1 (query_optimizer.py): when no optimization candidate was selected (all candidates regressed or only one strategy was tried and failed), the Remaining Issues section was absent. It is now populated from original_analysis when selected_idx = -1.
  • Non-actionable [You] items inflating scores (documentation.py): EXPLAIN sometimes generates [You] items that confirm the query is already correct ("no action needed", "already efficient", "approach is correct", etc.) — these items inflate the issue count and lower the deterministic user score unfairly. A _non_actionable regex filter now removes them before score computation. A matching instruction was added to the EXPLAIN prompt: [You] items must always describe something to fix, never confirm a pattern is already fine.
  • Duplicate uncorrelated scalar subquery flagged as false positive (documentation.py): the EXPLAIN scorer was generating [You] items for scalar subqueries appearing more than once in the same SELECT list (e.g. high_value_transaction_count and high_value_ratio both containing the same SELECT AVG(...) FROM t expression). In Teradata the optimizer evaluates identical uncorrelated scalar subqueries once and reuses the result — this is a SQL-text artifact, not a performance problem. Added to "Do NOT lower the score for" in the EXPLAIN prompt and to the _non_actionable filter.
  • Teradata reserved word DAY causing Error 3706 (query_optimizer.py): LLM-generated rewrites using DATEADD(DAY, n, date) or (date1 - date2) DAY caused "expected something between '(' and the string 'DAY'" because DAY is a Teradata reserved word and putting (DAY together is always illegal. Added explicit entries to _teradata_sql_guide(): a "DATE minus DATE" section (Teradata subtraction already returns integer days — no DAY qualifier needed) and explicit ✗ DATEADD(DAY, ...) / ✗ DATEDIFF(DAY, ...) examples noting these are SQL Server functions that do not exist in Teradata.
  • Error 5480 / Error 3504 alternating fix loop (query_optimizer.py): when a window function was nested inside another (Error 5480), the LLM fix removed the nesting but placed an aggregate and a window function in the same SELECT + GROUP BY (Error 3504). Added both error patterns with correct separation examples to _teradata_sql_guide().
  • INSTRUCT_MODEL_URL=None rejected for OpenAI provider (documentation.py): the URL presence check if not INSTRUCT_MODEL_URL was triggered even when the provider was openai (which uses a fixed endpoint and does not require a URL). Fixed by checking _url_required = INSTRUCT_MODEL_PROVIDER not in ('openai', 'bedrock') first.

Removed

  • _compact_sql_sync() and _try_compaction() (query_optimizer.py): retired. The compaction logic is now the pre-optimization simplification step (Step 0.5), which runs on every query rather than only after a perfect 5/5 EXPLAIN. Compaction call sites in skill_generate_candidates have been removed accordingly.

[0.2.8.0] - 2026-04-14

Added

  • Query Optimizer Agent (tdfs4ds/agent/query_optimizer.py): LLM-powered SQL rewrite pipeline for Teradata feature-engineering queries. Process-aware — when a process_id is supplied, SQL and stored EXPLAIN documentation are pulled directly from the process catalog, avoiding redundant LLM calls.
  • 8-step pipeline inside skill_optimize_query:
    1. Process context — get_process_info fetch (skipped when no process_id)
    2. Structured EXPLAIN analysis — document_sql_query_explain scores the plan 1–5 with [You]-prefixed author-actionable warnings and recommendations
    3. Lineage graph — Primary Index + partition columns collected per underlying object via build_teradata_dependency_graph
    4. DDL fetch — SHOW TABLE / SHOW VIEW for every referenced object
    5. Candidate generation loop — LLM proposes ≤ N rewrites focused on [You]-actionable items; near-duplicate guard (SequenceMatcher ≥ 0.97) discards cosmetic reformats; deterministic _try_flatten_identity_wrapper applied to every candidate before EXPLAIN
    6. Candidate EXPLAIN — document_sql_query_explain run per surviving candidate
    7. Plan comparison — LLM selects the best plan; deterministic safeguard overrides a regressing LLM choice and promotes the highest-scoring valid candidate
    8. FilterManager check — partitioned-but-unfiltered objects flagged; auto-generates a ready-to-run FilterManager loop snippet
  • query_optimizer() — synchronous LangGraph entry point; multi-turn via MemorySaver singleton (_get_checkpointer()); retrieves SQL from history on follow-up questions that do not supply new SQL
  • aquery_optimizer() — async wrapper for Jupyter notebooks (avoids background-thread overhead from asyncio.run)
  • skill_generate_candidates() — exported candidate generation loop; two independent counters (valid_count, failure_count); early-stop only at perfect 5/5 score; QUERY_OPTIMIZER_MAX_FAILURES caps syntax/EXPLAIN failures separately from the candidate budget
  • display_optimization_result() — Jupyter display helper: renders a before/after score comparison widget (colour-coded Δ Change cells, strategy and business-logic-preservation footer) followed by the full Markdown report
  • score_delta result key: optimized, best_strategy, original_user_score, original_global_score, best_user_score, best_global_score, user_score_delta, global_score_delta, business_logic_preserved
  • Native Teradata tools (no MCP server required): seven read-only metadata functions over the active teradataml connection — _td_list_databases, _td_list_tables, _td_list_views, _td_get_object_type, _td_get_table_ddl, _td_get_view_ddl, _td_get_query_explain
  • _teradata_sql_guide(): single source of Teradata syntax rules injected into LLM prompts (date arithmetic, percentile/ordered-set aggregates, INTERVAL literals, NULL/conditional, common Error 3706/3707 patterns)
  • _fix_answer_fences(): post-processes LLM output to close fenced code blocks that the model forgot to close before a Markdown section heading
  • Three new globals (tdfs4ds/__init__.py):
  • QUERY_OPTIMIZER_MAX_CANDIDATES (default 5) — max valid rewrites to evaluate
  • QUERY_OPTIMIZER_MAX_FAILURES (default 5) — max EXPLAIN/syntax failures before the candidate loop stops; failures do not count toward MAX_CANDIDATES
  • QUERY_OPTIMIZER_MAX_CANDIDATE_TOKENS (default None) — hard cap on completion tokens per candidate call; set to e.g. 4096 for small-context models
  • Notebook 11 - Query Optimizer Agent with tdfs4ds.ipynb (notebook dev/genai/): end-to-end walkthrough — process-aware entry, multi-turn follow-ups, score comparison widget, and FilterManager recommendation.
  • README: added Query Optimizer Agent section with pipeline table, quick start, all result keys, score_delta structure, display_optimization_result usage, and FilterManager integration snippet; added consumer agent notebook references (09, 10); added EXPLAIN→DEFINITION drill-down paragraph and resolved_column_sources state field to the consumer agent section.

[0.2.7.5] - 2026-04-14

Added

  • EXPLAIN→DEFINITION drill-down (consumer_agent.py, graph.py): after an EXPLAIN turn the agent now remembers every source column mentioned in the feature formula.
  • SourceColumnRef Pydantic model captures column_name and source_table from the LLM response.
  • ExplainOutput gained a source_columns field; the LLM is asked to resolve all distinct column references (resolving through subquery aliases to the real base table/view wherever possible).
  • synthesize_answer appends a "Variables involved" list after the explanation (- \col` (from `table`)`) so users can immediately ask "what is X?" about any input variable.
  • node_skill_explain writes the column→source-table mapping into the new resolved_column_sources field in AgentState (persisted via MemorySaver).
  • BD_COLUMNS secondary definition path (consumer_agent.py): skill_definition gained a new lookup path that queries FS_BUSINESS_DICTIONARY_COLUMNS for columns that are not registered features (dimension keys, derived columns, etc.).
  • New optional parameter hint_table_name: when set (from a prior EXPLAIN turn via resolved_column_sources), the search is narrowed to the correct table immediately.
  • node_skill_definition reads resolved_column_sources and passes the matched table as hint_table_name.
  • Fuzzy follow-up resolution for DEFINITION (graph.py): new _fuzzy_resolve_from_column_sources(question, col_sources) helper. When the intent classifier extracts no explicit object_name for a DEFINITION question, the helper tokenises the user's question (with a bilingual EN/FR stop-word list) and checks whether any token is a substring of a known column name or source table. A match is used as the resolved object name — no need for the user to quote the exact identifier.
  • resolved_column_sources in AgentState (graph.py): new optional dict field (keys: uppercase column names; values: bare source table/view names). Displayed in the Gradio trace panel under "Feature Context (remembered)".

Changed

  • connect() round-trip reduction (lifecycle.py): replaced two db_list_tables() and two DataFrame(in_schema(...)).columns / get_ddl() calls with a single targeted DBC.TablesV query and a combined DBC.ColumnsV query. Falls back to the original approach automatically when DBC access is restricted.
  • DatasetCatalog._exists() accepts pre-computed object set (dataset_catalog.py): new optional existing_objects parameter. When supplied by connect(), the db_list_tables() round-trip inside the constructor is skipped entirely.
  • Lazy LangChain / sqlparse imports (genai/documentation.py): langchain_openai, langchain_core, and sqlparse are now imported inside the functions that need them. This removes the 30–40 s startup cost incurred on every plain import tdfs4ds when LangChain is installed.

[0.2.7.2] - 2026-04-10

Fixed

  • Multi-turn memory loss (graph.py): consumer_agent() was calling build_graph() on every invocation, creating a fresh MemorySaver each time. All cross-turn state (resolved_object_name, resolved_feature_triplet, resolved_data_domain, etc.) was wiped after every question, so follow-up questions like "when has this feature been updated?" could not resolve back to the feature discussed in the previous turn. Fixed by introducing a module-level _graph_instance singleton via _get_graph() — the compiled graph and its MemorySaver are now built once and reused for the lifetime of the process.
  • Ambiguous-feature prompt on contextual follow-ups (consumer_agent.py): when a feature name exists across multiple entity types and the user asked a follow-up question without repeating the entity, skill_freshness and skill_definition returned ambiguous_feature and asked for clarification even though the entity was already established from a previous turn. Fixed by passing hint_entity_name from resolved_entity_name state: if the hint uniquely identifies one row in the catalog, disambiguation is silent and the skill proceeds normally.

Added

  • Entity and feature-group context in AgentState (graph.py): two new persisted fields carried by MemorySaver across turns:
  • resolved_entity_name — entity currently in focus (e.g. "CustomerID"); updated whenever a feature triplet is resolved.
  • resolved_feature_list — list of feature names currently in focus; holds a single name after a single-feature resolution, or the full entity feature list after a SUMMARY turn covering one entity.
  • skill_freshness / skill_definition accept hint_entity_name (consumer_agent.py): optional parameter used for silent disambiguation (see fix above).
  • skill_summary returns feature group context (consumer_agent.py): the result dict now includes feature_names (flat list) and feature_names_by_entity (entity → list of feature names). node_skill_summary uses these to update resolved_feature_list and, when exactly one entity is covered, resolved_entity_name.
  • Trace panel shows feature-group size (graph.py): when resolved_feature_list holds more than one feature, the Feature Context section of the agent trace displays the count and names of all features in focus.

[0.2.7.1] - 2026-04-10

Fixed

  • Intent classifier — verbatim extraction of feature name and domain (consumer_agent.py): the intent_classifier system prompt now explicitly instructs the LLM to copy both the feature/dataset name and the domain name exactly as typed (no spelling correction, no pluralisation change, no case normalisation). Previously the LLM would silently "fix" names it considered misspelled (e.g. nb_days_since_last_transactions → nb_days_since_last_transaction), causing catalog lookups to fail on perfectly valid feature names. The same risk applied to domain abbreviations (e.g. ATR expanded to a full word), which would cause skill_summary to summarise under the wrong label.
  • process_view_not_found error response (synthesize_answer): the synthesizer is now explicitly told not to suggest alternative spellings or guess at correct names — it can only see what the skill returned, so any suggestions it invented were hallucinations.

Added

  • Fuzzy feature-name matching (consumer_agent.py): new helper _fuzzy_match_feature(object_name) uses difflib.get_close_matches (stdlib, no new dependency) with a similarity cutoff of 0.75 against the full feature catalog across all data domains. Called automatically when an exact lookup fails in skill_explain, skill_lineage, and skill_freshness.
  • Single close match → the skill auto-corrects to the catalog name, runs normally, and includes a correction_note in the result dict.
  • Multiple close matches → returns a new feature_not_found_did_you_mean error with the candidate list (no answer is attempted).
  • feature_not_found_did_you_mean error handler (synthesize_answer): new error path that tells the user the exact name was not found and lists the similar catalog names found — without guessing what those features do.
  • Auto-correction notice in answers: when a single-candidate auto-correction is applied, synthesize_answer prepends an italic note to the direct answer, e.g. 'nb_days_since_last_transactons' was not found — answered for the closest match 'nb_days_since_last_transactions'.

[0.2.7.0] - 2026-04-09

Added

  • Consumer agent module (tdfs4ds/agent/): conversational chatbot for business consumers of the feature store. Non-technical users can ask natural-language questions about features, datasets, definitions, data freshness, usage guidance, data lineage, and calculation logic — in English or French.
  • Intent classifier (consumer_agent.py): uses Pydantic structured output via llm.with_structured_output() to route questions to one of 7 skills: SEARCH, DEFINITION, USAGE, FRESHNESS, SUMMARY, LINEAGE, EXPLAIN.
  • 7 skills (consumer_agent.py): skill_search (semantic vector search), skill_definition (process view → column docs), skill_usage (audience, granularity, regulatory guidance), skill_freshness (latest execution timestamps), skill_summary (full feature list with descriptions), skill_lineage (upstream dependency graph with LLM summary), skill_explain (SHOW VIEW DDL → plain- language business explanation).
  • LangGraph orchestration (graph.py): StateGraph with conditional routing (classify → detect_domain → skill → synthesize → END) and MemorySaver checkpointer for multi-turn conversations.
  • DATA_DOMAIN auto-detection (graph.py): node_detect_domain node scans all DATA_DOMAINs via feature_catalog(all_data_domains=True) to find which domain owns the requested object. Stores resolved_data_domain in LangGraph state; persisted across turns via MemorySaver so follow-up questions reuse the last resolved domain. All skill nodes set tdfs4ds.DATA_DOMAIN via a context manager. New public functions: get_data_domain(), detect_data_domain_for_object().
  • Feature description resolution chain (consumer_agent.py): _get_feature_catalog_with_docs() resolves feature descriptions via get_list_entity → get_list_features → get_feature_versions → VIEW_NAME → FS_BUSINESS_DICTIONARY_COLUMNS. Used by DEFINITION and SUMMARY.
  • Embedding factory (embedding.py): get_embeddings() returns a LangChain Embeddings instance; supports openai/vllm/azure/bedrock providers. Falls back from EMBEDDING_MODEL_* to INSTRUCT_MODEL_* if not set. list_embedding_models(sub_paths=...) queries available models on the embedding endpoint.
  • Chroma vector index (vector_index.py): build_vector_index(force_rebuild=False) builds three collections (objects, sections, columns) from the business dictionary. Incremental by default. Documents are embedded in batches of 8 to respect TEI server limits. search_vector_index(query, sources, top_k) searches with optional section_name filtering and deduplicates on OBJECT_NAME.
  • Gradio chatbot (chatbot.py): launch_chatbot() starts a Gradio Blocks UI with JupyterHub proxy support. launch_chatbot_with_index(port, force_rebuild, ...) builds/refreshes the Chroma vector index then launches the Gradio UI — single command for end-to-end startup.
  • Agent trace panel (chatbot.py): collapsible gr.Accordion showing Intent Classification, DATA_DOMAIN Detection (available / active / source), Skill Executed, and Skill Result. Errors include per-step diagnostic messages. consumer_agent(return_trace=True) returns (answer, trace_md).
  • Model listing: list_instruct_models() and list_embedding_models() query the configured API endpoints for available models.
  • Demo notebooks: 09 - Consumer Agent Chatbot with tdfs4ds.ipynb (architecture, 7-intent test suite, 4-turn multi-turn demo) and 10 - Launch Consumer Agent Chatbot.ipynb (minimal 4-cell one-command launch).
  • Unit tests: tests/test_consumer_agent.py — 17 tests covering intent classification, skills, embeddings, vector index, graph routing, and multi-turn.
  • document_feature_store_incremental() (tdfs4ds/genai/documentation.py): single-pass, topo-sorted documentation of every registered process and dataset. Builds a combined lineage graph seeded from all dataset DDLs (expand_datasets_via_process_catalog=True) and any orphan processes; topologically sorts it (leaves first, roots last); then walks the order calling document_process_incremental and document_dataset_incremental with a shared pair of visited-sets so each process view is documented at most once regardless of how many datasets reference it. Returns a summary dict with processes_documented, datasets_documented, processes_skipped, datasets_skipped. Supports force_update, all_data_domains, and fallback documentation for objects absent from the graph.
  • Business dictionary mirroring (upload_documentation): after persisting process documentation to the process-documentation tables, the function now also writes:
  • The entity description to FS_BUSINESS_DICTIONARY_OBJECTS (type 'V')
  • Per-feature descriptions to FS_BUSINESS_DICTIONARY_COLUMNS (keyed by view name + feature name) Both writes are best-effort — failures are logged as warnings without aborting.
  • Shared visited-sets in document_dataset_incremental: new optional private parameters _visited_processes and _visited_views allow an outer orchestration call (e.g. document_feature_store_incremental) to pass a shared recursion guard, preventing duplicate documentation when multiple datasets share upstream process views.
  • Embedding model configuration variables (tdfs4ds/__init__.py, config.py): EMBEDDING_MODEL_PROVIDER, EMBEDDING_MODEL_URL, EMBEDDING_MODEL_API_KEY, EMBEDDING_MODEL_MODEL, EMBEDDING_MODEL_DIM — mirror the INSTRUCT_MODEL_* pattern. All loadable from .env / tdfs4ds.json / TDFS4DS_* env vars.
  • Chroma configuration variables: CHROMA_MODE ('local'/'server'), CHROMA_PATH, CHROMA_HOST, CHROMA_PORT.
  • Claude Code skill: fs-agent — generates boilerplate for the consumer agent chatbot workflow.

Changed

  • skill_definition: primary path now resolves via the process documentation chain (process view → column docs) instead of querying the feature name directly in the objects table. Falls back to dataset-level lookup, then vector search.
  • skill_summary: replaced bare feature_catalog() call (no descriptions) with _get_feature_catalog_with_docs() so each feature line includes its business description from process documentation.
  • synthesize_answer: DEFINITION, USAGE, FRESHNESS, SUMMARY, LINEAGE, EXPLAIN now short-circuit to their skill's final answer key (no second LLM pass). Only SEARCH still uses the generic synthesis path.
  • skill_freshness diagnostics: all three failure points (feature_catalog miss, process_catalog miss, empty followup) now return a diagnostic key with per-step detail instead of collapsing to the same opaque no_execution_history error.
  • _get_process_view: primary lookup is now feature_catalog → PROCESS_ID → process_catalog → VIEW_NAME. The FEATURE_NAMES comma-list search is kept as a fallback only.
  • setup.py: added langchain_chroma, langchain_teradata, langgraph, chromadb, pydantic, gradio to install_requires.
  • fs-setup skill: added commented embedding model and Chroma configuration blocks.
  • fs-document skill: added rule to rebuild vector index after documenting.

[0.2.6.5] - 2026-04-07

Added

  • Dataset documentation (tdfs4ds/genai/documentation.py): document_dataset_incremental(dataset_id) generates documentation for a dataset by walking its full lineage bottom-up.
  • Source tables: uses business dictionary if available, warns if missing.
  • Intermediate views: auto-documented via LLM if undocumented.
  • Process views: actively calls document_process_incremental if undocumented.
  • Feature/entity column descriptions are propagated from process documentation (no redundant LLM call for 1:1 feature mappings).
  • A single JSON-constrained LLM call generates five structured sections for the dataset itself: OVERVIEW, ENTITY, FEATURE_THEMES, BUSINESS_QUESTIONS, INTENDED_AUDIENCE.

  • Notebook 08 - Dataset documentation with tdfs4ds.ipynb (notebook dev/genai/).

  • Business dictionary sections table (FS_BUSINESS_DICTIONARY_SECTIONS): new temporal table keyed on (DATABASE_NAME, OBJECT_NAME, SECTION_NAME). Each row stores one section of an object's documentation as an independent RAG document. Together with the existing objects and columns tables, this forms a 3-level hierarchy designed for chunking-free hierarchical retrieval:

  • Level 0: FS_BUSINESS_DICTIONARY_OBJECTS — object summary (OVERVIEW)
  • Level 1: FS_BUSINESS_DICTIONARY_SECTIONS — one row per section
  • Level 2: FS_BUSINESS_DICTIONARY_COLUMNS — one row per column
  • New upload function: upload_business_dictionary_sections(df).
  • New helper: _fetch_sections_for_node(db, obj).
  • Table created automatically on next setup() or connect(create_if_missing=True).

Changed

  • _format_upstream_context: type label for 'F' changed from 'Feature Store Dataset' to 'Feature Store Process' (datasets use type 'D').

[0.2.6.4] - 2026-04-03

Added

  • Dataset lineage via process catalog (tdfs4ds/lineage/network.py): new expand_datasets_via_process_catalog parameter on build_teradata_dependency_graph (default True). When a dataset node is detected the graph parses the dataset view DDL to extract FEATURE_VERSION UUIDs (each is a PROCESS_ID), looks them up in FS_V_PROCESS_CATALOG, and adds edges directly to the registered feature-engineering views — bypassing the internal FS_T_... storage tables.
    Set expand_datasets_via_process_catalog=False to restore the previous leaf behaviour (dataset node connected directly to its raw storage tables).
    Expected lineage chain: dataset → process view(s) → source table(s).
  • graph_to_migration_manifest(graph, *, filter_database=None) (tdfs4ds/lineage/network.py, exported from tdfs4ds.lineage): converts a lineage graph into a JSON-serialisable migration manifest with three keys:
  • "views" — {database, name, type} entries for every view/dataset node
  • "tables" — {database, name, type, primary_index, partition_columns} entries
  • "edges" — {from, to} dependency pairs
    Pass filter_database to scope output to a single feature store schema; cross-boundary edges are excluded automatically.
  • Notebook 07 - Dataset Lineage via Process Catalog.ipynb (notebook dev/lineage/): demonstrates both behaviours side-by-side and the migration manifest export.

Fixed

  • Orphaned FS_T_... storage-table nodes are pruned from the graph result after their edges are replaced by process-view edges.
  • Pre-added edges from a dataset root to its raw storage tables are purged only when expand_datasets_via_process_catalog=True; when False those edges are preserved and the old output is unchanged.

[0.2.6.3] - 2026-04-03

Added

  • Business dictionary (tdfs4ds/genai/documentation.py): two new temporal tables and corresponding upload functions for storing business-oriented descriptions independently of the process documentation workflow. No LLM required — descriptions are plain text supplied by the user in a pandas DataFrame.
  • FS_BUSINESS_DICTIONARY_OBJECTS — object-level descriptions (DATABASE_NAME, OBJECT_NAME, OBJECT_TYPE 'T'/'V', BUSINESS_DESCRIPTION). Temporal VALIDTIME table.
  • FS_BUSINESS_DICTIONARY_COLUMNS — column-level descriptions (DATABASE_NAME, TABLE_NAME, COLUMN_NAME, BUSINESS_DESCRIPTION). Temporal VALIDTIME table.
  • upload_business_dictionary_objects(df) — inserts or updates object descriptions via CURRENT VALIDTIME MERGE; raises ValueError on missing columns.
  • upload_business_dictionary_columns(df) — inserts or updates column descriptions via CURRENT VALIDTIME MERGE; raises ValueError on missing columns.
  • Both tables are provisioned automatically by documentation_tables_creation(), called from setup() and connect(create_if_missing=True).
  • connect() accepts two new optional keyword arguments — documentation_business_dictionary_objects and documentation_business_dictionary_columns — to override the default table names, consistent with all other catalog parameters.
  • External configuration support (tdfs4ds/config.py): global variables can now be set from three external sources, eliminating the boilerplate setup cell that every notebook previously required.
  • JSON config file (tdfs4ds.json / ~/.tdfs4ds/config.json) — project defaults for non-sensitive settings. Keys are case-insensitive.
  • .env file (./.env / ~/.tdfs4ds/.env) — local overrides and credentials. Only TDFS4DS_* variables are read; the file is parsed without modifying os.environ. Add .env to .gitignore to keep secrets out of source control. Supports KEY=VALUE, quoted values, and export KEY=VALUE.
  • OS environment variables — prefix TDFS4DS_, e.g. TDFS4DS_SCHEMA, TDFS4DS_DATA_DOMAIN. Intended for CI/CD pipelines.
  • INSTRUCT_MODEL_API_KEY is rejected from JSON config files (credential safety); use .env or an OS env var instead.
  • tdfs4ds.load_config(path=None, dotenv_path=None) — public function to (re-)load config at any point, or to load from non-default file paths.
  • Priority chain (highest first): programmatic → OS env var → .env file → JSON config → teradataml auto-detection (SCHEMA) → built-in defaults.

Changed

  • SCHEMA auto-detection from the teradataml context now only runs when SCHEMA is still None after config/env loading, so env-var or file settings are not silently overwritten on import.

[0.2.6.2] - 2026-03-26

Fixed

  • tqdm rendering in VSCode Jupyter: switched all import tqdm / tqdm.tqdm(...) usages in entity_management.py, feature_store_management.py, and utils/lineage.py to from tqdm.auto import tqdm. tqdm.auto selects the widget renderer in JupyterLab/VSCode Jupyter and falls back to the standard terminal bar elsewhere.

[0.2.6.1] - 2026-03-26

Fixed

  • Teradata Error 6706 when uploading LLM-generated documentation to LATIN-charset columns. Added _sanitize_for_teradata() in genai/documentation.py that replaces common Unicode characters produced by LLMs (smart quotes, em/en dashes, ellipsis, bullet points, non-breaking spaces) with ASCII equivalents, then encodes the remainder as ISO-8859-1. Applied in both upload_documentation and upload_documentation_explain.

[0.2.6.0] - 2026-03-26

Changed

  • Reshape: refactored tdfs4ds/__init__.py into dedicated submodules for better maintainability (catalog.py, data_domain.py, execution.py, lifecycle.py, dataset/builder.py). Public API surface is unchanged.
  • Added Claude Code skill definitions (.claude/skills/, skills/) for feature-store workflows: fs-setup, fs-upload, fs-inspect, fs-filter, fs-rollout, fs-dataset, fs-document, fs-lineage, fs-analyze.
  • Extended demo notebooks and added pre-executed notebook exports under demos/notebooks_executed/.

[0.2.5.4] - 2025

Added

  • Lineage module: SQL query parsing, Teradata DDL analysis, dependency graph construction, and Sankey diagram visualisation.
  • Primary-index and partition-column inspection for source tables.
  • AWS Bedrock support in the GenAI documentation module.

[0.2.5.3] - 2025

Added

  • Retrieve and update stored documentation from documentation tables.
  • Pretty-print display of process documentation (display_process_info).
  • Manual documentation entry via chatbot output (feed_process_info_with_prompt_result).

Fixed

  • Bug fixes in documentation table creation and volatile-table drop logic.

[0.2.4.45] - 2025

Added

  • GenAI module (genai/documentation.py): LLM-powered auto-documentation of SQL processes using LangChain with OpenAI or AWS Bedrock.
  • document_process(): generates business-logic description, entity description, entity-column annotations, and feature-column descriptions from a process view.
  • get_the_explain() + document_sql_query_explain(): EXPLAIN-plan retrieval and LLM-powered optimisation analysis with warnings and recommendations.
  • upload_documentation() / upload_documentation_explain(): persist generated documentation into temporal documentation tables.
  • documentation_tables_creation(): provisions the documentation catalog tables.
  • Support for json_constraint mode (structured JSON output) and prompt_only mode (returns prompt without calling the LLM).