Consumer agent

A conversational chatbot built on LangGraph + Gradio with a pluggable vector store (ChromaDB by default, Teradata Enterprise Vector Store as an opt-in). Business users ask natural-language questions; the agent answers from the business dictionary.

Architecture

The agent executes a multi-stage LangGraph StateGraph with full cross-turn context persistence via a process-level MemorySaver singleton:

START
  ↓
classify
  (LLM: intent classification + object name extraction via Pydantic structured output)
  ↓
detect_domain
  (Scan all DATA_DOMAINs; resolve owning domain; persisted across turns)
  ↓
recognize_entity
  (Type-tag the object: feature, dataset, or process view)
  ↓
[conditional routing by intent]
  ├─ SEARCH         → skill_search        → synthesize → END
  ├─ DEFINITION     → skill_definition    → synthesize → END
  ├─ USAGE          → skill_usage         → synthesize → END
  ├─ FRESHNESS      → skill_freshness     → synthesize → END
  ├─ SUMMARY        → skill_summary       → synthesize → END
  ├─ LINEAGE        → skill_lineage       → synthesize → END
  ├─ EXPLAIN        → skill_explain       → synthesize → END
  ├─ ALGORITHM      → skill_algorithm     → synthesize → END (MCP only; dual-intent runs EXPLAIN first)
  ├─ DATASET        → skill_dataset       → synthesize → END
  ├─ BUILD_DATASET  → skill_build_dataset → synthesize → END
  ├─ RECAP          → skill_recap         → synthesize → END (memory only)
  ├─ OUT_OF_SCOPE   → (polite redirect)   → synthesize → END
  └─ DATA_QUERY     → skill_mcp_tools     → synthesize → END (MCP only)

RECAP and OUT_OF_SCOPE are built-in intents — they have no ca-* SKILL.md file. build_graph() injects their descriptions into the dynamic Pydantic intent model and the classifier prompt, and _build_classifier_examples() seeds explicit few-shot examples so they fire reliably. RECAP reads only the accumulated conversation memory (no Teradata query) to summarise the features, entities, and datasets discussed so far; OUT_OF_SCOPE politely redirects off-topic questions (weather, general coding, etc.).

Graph Singleton & State Persistence

The graph instance is cached in a module-level dict _graph_instances, keyed by (frozenset(skills), plugin_key, mcp_enabled). This singleton pattern ensures:

• One MemorySaver checkpoint per (skills, MCP) combo — cross-turn state (resolved_data_domain, resolved_feature_triplet, etc.) persists in LangGraph's built-in checkpointer without an external database.
• Automatic graph recompilation when skills change or MCP is toggled.
• Clean reset via reset_consumer_agent() which clears the entire cache.

Context as a Named-Entity Knowledge Graph

The AgentState TypedDict is a stateful knowledge graph whose fields are persisted across conversation turns. Users can refer back without repeating context:

State field	What it represents
resolved_data_domain	Active DATA_DOMAIN (project namespace)
resolved_object_name	Last explicitly named feature or dataset (remembered)
resolved_feature_triplet	{feature_name, entity_name, process_id, view_name} — the "current node" in the feature graph
resolved_entity_name	Entity type currently in scope (e.g. "CustomerID")
resolved_feature_list	One or many features currently discussed
resolved_column_sources	{COLUMN_UPPER: source_table} — populated by EXPLAIN, enables drill-down DEFINITION
resolved_dataset_id	UUID from the last SUMMARY turn on a dataset
resolved_dataset_features	Features loaded from that dataset
resolved_entity_name	Entity picked during an entity-clarification turn; drives skill_definition's entity hint
pending_action / pending_action_context	State machine for multi-step flows (dataset creation, entity clarification, ambiguous-query clarification)

The effective current object is resolved by _effective_object(state) with this priority: explicit this-turn → resolved_feature_triplet.feature_name → resolved_object_name.

Accumulated-context feature keying

accumulated_context_graph.features is keyed by FEATURE_NAME::ENTITY_NAME (both uppercased) — not by the bare feature name — so the same feature name registered under two entities never collides in the cross-turn cache. The features, datasets, entities, and source_views sections all use uppercase keys for stable lookup (processes and freshness_timeline stay keyed by their process-id UUIDs). Skills resolve entries through _feat_cache_lookup(), which tries the exact composite key, then a FEATURE:: prefix scan, then the legacy plain-FEATURE key (so older snapshots still load).

MCP / DATA_QUERY integration

When tdfs4ds.MCP_SERVER_URL is configured, users can enable the "Enable MCP Tools" checkbox in the chatbot to route questions to remote data sources via the MCP (Model Context Protocol).

How it works:

1. The graph registers an additional DATA_QUERY intent and corresponding skill_mcp_tools node.
2. When MCP is enabled and no feature-store skill matches the question, the classifier may assign DATA_QUERY intent.
3. _skill_mcp_tools_async() uses langchain_mcp_adapters.MultiServerMCPClient to wrap the remote tools (e.g. SQL query execution, file upload, database operations).
4. A ReAct agent (langgraph.prebuilt.create_react_agent) chains tool calls until the query is resolved.

Data governance & safety:

• _make_guarded_tools() wraps all MCP tools with rejection logic: queries referencing internal feature-store objects (FS_T_*, FS_V_*, process view names) are blocked.
• SELECT queries automatically get TOP 20 injected; if results hit the limit, a warning is appended so the user knows data was truncated.
• Event loop compatibility: uses ThreadPoolExecutor inside a running Jupyter loop, asyncio.run() in other contexts.

Editions: Community vs Enterprise

The two Teradata MCP products authenticate differently and expose different tool sets, so tdfs4ds.MCP_SERVER_EDITION tells the agent which one it is holding — it is added to the ReAct system prompt, and it decides whether credentials are sent at all.

Edition	Setting	Credentials
Community (default)	nothing to set	The server embeds its own — tdfs4ds sends no header
Enterprise	MCP_SERVER_EDITION='enterprise'	HTTP Basic, defaulting to TDFS4DS_TD_USERNAME / TDFS4DS_TD_PASSWORD

TDFS4DS_MCP_SERVER_URL=https://your-enterprise-mcp/one-td/mcp
TDFS4DS_MCP_SERVER_EDITION=enterprise

# only when the MCP account differs from the database one
# TDFS4DS_MCP_USERNAME=... / TDFS4DS_MCP_PASSWORD=...
# any other scheme, wins over the above
# TDFS4DS_MCP_SERVER_HEADERS={"Authorization": "Bearer ..."}
# internal CA — verification is never disabled, only re-anchored
# TDFS4DS_MCP_SERVER_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem

The connection is assembled in agent/mcp_connection.py (build_mcp_connection), shared by the agent and check_configuration() so a passing diagnostic describes the connection the agent actually makes. With none of these variables set the connection dict is exactly {"url", "transport"} — identical to the behaviour that predates the edition concept.

Why credentials are gated on the edition

TDFS4DS_TD_USERNAME/TD_PASSWORD are set in nearly every deployment. If Basic auth were derived from them unconditionally, every existing Community install would begin sending the Teradata password to its MCP endpoint on upgrade. Defaulting the edition to community makes that impossible rather than merely unlikely.

How many tools reach the model

Every tool's JSON schema is sent to the model on every turn, so a large tool set costs context before your question is even read. A live Enterprise server exposes around 132 tools — enough to overflow a model outright, which surfaces as a ContextWindowExceededError behind an HTTP 400.

tdfs4ds.MCP_MAX_TOOLS (default 32, 0 disables) bounds it:

TDFS4DS_MCP_MAX_TOOLS=32

At or below that count nothing is dropped — a Community-edition install binds exactly the tools it always did. Above it, the tools whose names and descriptions best match the question are kept, ranked deterministically with no extra model call; the remaining slots are filled with the rest, so an unusual phrasing degrades rather than leaving the agent with no tools at all.

Raise it for a large-context model. If the agent reports that it lacks a tool you know the server exposes, raising it is the first thing to try.

When even that does not fit

Choosing MCP_MAX_TOOLS is a guess — too high and the schemas still overflow, too low and the model loses tools it needed. Rather than predict it, tdfs4ds recovers from the failure: on a context-window error the model is shown a compact index of every tool (one line each, name — description), asked which the question needs, and only those are bound for a single retry.

The index costs roughly a tenth of what the schemas cost, so around 130 tools fit in it comfortably. The extra model call happens only after a failure — a setup whose tools already fit never pays for it.

tdfs4ds.MCP_TOOL_DISCOVERY     = True   # False to fail fast instead
tdfs4ds.MCP_FALLBACK_MAX_TOOLS = 8      # tools bound on the retry

The retry happens once. A second overflow means even the focused set does not fit — raise the model's context or lower MCP_MAX_TOOLS. It also never re-runs once an answer has been produced, so a failure late in a multi-step task cannot answer you twice.

This is recovery, not mid-task discovery

The retry re-selects from the same question, so the model still cannot reach for a tool it discovers it needs halfway through a task. Genuinely progressive discovery needs tools to be bound dynamically as the reasoning unfolds, which tdfs4ds does not do.

Intents

Intent	Trigger example	What happens
SEARCH	"What features analyse customer spending?"	Semantic search over objects / sections / columns collections (Chroma or Teradata Vector Store)
DEFINITION	"What does total_amount measure?"	Resolves feature → process view → column doc in FS_BUSINESS_DICTIONARY_COLUMNS
USAGE	"How do I use avg_amount in Tableau?"	Audience, granularity, regulatory guidance
FRESHNESS	"Is total_amount up to date?" / "When was DS_CLAIMS last refreshed?"	Latest run from follow_up_report(). Works for individual features (resolves single process) and datasets (aggregates across all processes feeding the dataset; returns last_successful_run, days_since_update, has_recent_failure). Returns "not found" only when the named object is neither a registered feature nor a dataset.
SUMMARY	"List all features for customers" / "What features belong to the provider_id entity?"	Full feature list with entity + description (an overview past AGENT_LIST_MAX_ITEMS); scoped to one entity when entity_filter resolves
LINEAGE	"Where does total_amount come from?"	build_teradata_dependency_graph upstream
EXPLAIN	"How is total_amount calculated?"	SHOW VIEW DDL → LLM plain-language explanation + per-column source map (also accumulates into the context graph)
ALGORITHM	"What algorithm is behind ml_risk_score?" (MCP only)	Walks the tdfs4ds→tdstone2 STO metadata chain to explain the ML model: algorithm, hyperparameters, training configuration, Python packages, partition count. Returns answer directly (no second LLM pass). Disabled when MCP is off — algorithm-shaped questions get a "please enable MCP" message instead.
VALUES	"Give me 5 CustomerID values" / "Top 5 customers by sum_amount" / "sum_amount for customer 33357" / "The features of TICKER NVDA"	Read in the database, answered by code: a sample of an entity's values; the statistics, top, bottom and outliers of a feature; the values of features for given entity values (primary-index access), with the valid time read and each feature's process version
RECAP	"What are the features we are talking about?"	Reads accumulated conversation memory only — no Teradata query
OUT_OF_SCOPE	"What's the weather today?"	Polite redirect; the question is unrelated to the feature store

DATASET, BUILD_DATASET, and DATA_QUERY (MCP only) are additional routable intents; see the routing diagram above.

Documented data outside the feature store

The business dictionary documents more than the feature store: source tables and other views can be described too (written by hand, imported, or generated for the views a process reads). The agent searches them, but always says which side of the line an object is on:

• In the feature store: process views (their columns are features), datasets, Time/FilterManagers. These have a data domain, an entity, a producing process and a refresh history.
• Outside the feature store: every other documented table or view. Documented data, not governed features: no data domain, no entity, no refresh history, and the relationships between their columns are not certified.

Question	Answer
"Which table contains data about stock prices?"	The documented tables and views that match, under Outside the feature store, each with its matching columns and the feature processes that already read it (lineage). When no feature matches, the answer says so first.
"Is there any data dealing with customer transactions?"	Two headed parts: In the feature store (features and datasets), then Outside the feature store.
"Which features describe spending per category?"	The features only, plus one line naming the documented tables outside the store that also cover the subject.
"What is this table?" / "What is the lineage of this table?" after such a search	The best match: its documentation, or its lineage.
"What is MARKET_PRICES?" (a documented source table)	Its documentation, opened by a fixed sentence placing it outside the feature store.

The answer about outside data is written by code from the business dictionary, never by the LLM, and the feature-matching LLM never sees it, so a source column is never presented as a feature. A question shaped like "which table holds…" goes to this search whatever the classifier said, unless it names a documented table or column. To list everything documented on both sides, use export_documentation.

Datasets from the chat

• "Where can I find max_amount?" lists the datasets exposing that feature, with their entity and data domain.
• "What are the available datasets?" lists every dataset with its entities and kind (current, as of a date, TimeManager, history).
• When the feature is in no dataset, the agent offers to create one, and creates, registers and documents it. Your database permissions apply: a refused CREATE VIEW is reported with the database's own message.
• Before building it asks the kind of dataset and its parameters:

Reply	Dataset
1 / "current"	current snapshot
2 2024-03-31 14:30 +02:00 / "as of 31 March 2024"	snapshot at that valid time
3 last 90 days / "les 6 derniers mois" / 3 from 2024-01-01 to 2024-06-30 / 3 all	history over the window

• Times: a time without an offset is read as UTC, and the answer says so. Slash dates such as 03/04/2026 are refused as ambiguous: write 2026-04-03 or "3 April 2026". A date given here sets the dataset's time only, not the conversation's.
• The name is checked against the dataset catalog and existing objects in the view database.
• The new dataset reaches semantic search at the next vector-index refresh (tdfs4ds.rebuild_vector_indexing()).

Compound EXPLAIN + ALGORITHM questions

A compound question "How is ml_risk_score computed? What algorithm is behind it?" carries both EXPLAIN signals ("how is … computed") and ALGORITHM signals ("algorithm"). With MCP on:

1. Routing — _node_classify_impl re-routes EXPLAIN → ALGORITHM when both signal sets match.
2. Dual-intent execution — node_skill_algorithm runs skill_explain first (SQL formula), prepends its answer with a --- divider above the ALGORITHM answer (ML model details), and persists the EXPLAIN column sources so a follow-up "what is <column>?" still resolves.

With MCP off, the algorithm half is unavailable and the question routes to EXPLAIN only (the _looks_like_algorithm_question guard fires the "enable MCP" message in that case).

Entity-scoped SUMMARY

When the named object resolves to a registered entity (not a feature or dataset), or when entity_filter is supplied, skill_summary lists only the features that belong to that entity — answering "what features belong to entity X?". _recognize_object_type now type-tags entity and data_domain objects in addition to feature / dataset / process.

Naming a data domain in the question

On a store with several data domains, a question that names one ("what are the features of the Governed Demand Forecasting M5 data domain?") switches the conversation to that domain. Case, quotes and a small typo are tolerated; a word that is not a data domain (a theme such as "pharmacy") does not switch anything. The listing and the context knowledge graph then show that domain's features, and the domain stays active for follow-up questions until another one is named. A domain without registered features is reported as such. A question that names no domain gets a short "which data domain?" choice when there are several.

Feature inventories: rendered by code

An inventory (SUMMARY) is rendered by code, so the same question on the same store gets the same answer (0.3.1.240). It starts with the counts, computed with the Report's definition (a feature is the triplet data domain / entity / name; a process is a distinct PROCESS_ID), and the notes the Report shows: features without a producer, features with several producers (versions), processes never run.

• Up to tdfs4ds.AGENT_LIST_MAX_ITEMS (default 40): one table per entity, Business meaning | Technical name. The meaning is the first sentence of the documented description; an undocumented feature says so.

• Above it: semantic groups, each with a label, one sentence and its technical names. The grouping is proposed once by the instruct model, checked against the catalog (no invented or missing name), stored in the business dictionary (section FEATURE_GROUPS of the reserved object __SCOPE__<domain>[__<entity>], excluded from search and from documentation coverage) and reused until the feature set changes. rebuild_documentation() regenerates it. Without an instruct model, the features are grouped by the process that computes them:

• an overview by producing process (fallback): the total, then one block per process with its stored documentation (the first sentence of what it computes and of what its entity is), a few feature names, and the count;

• a subject narrows the list: "Which features of entity STORE are about returns?" selects the features whose documented description matches, every name checked against the scope, and lists them all with their descriptions; a subject with no match says so;
• a process scope: "Which features does process V_SALES compute?" lists everything that process computes;
• follow-up suggestions name real processes.

Listings below the cap are unchanged. The data-domain list shows the largest domains and says how many more exist. The full feature list is still returned in the skill result, so the context graph and the follow-up cache are unchanged; after a subject, follow-ups ("are these up to date?") apply to the selection.

A past valid time

A question that names a date ("which features existed on 30 June 2023?", "au 31 octobre 2023") is answered from the catalogs and the business dictionary as they were then (VALIDTIME AS OF on the base tables). The date is kept for the following turns until the user says "now", and every answer at a past time starts with the date it was read at. Inventories, definitions, datasets, usage, search and freshness honour it; the other answers say they use the current state. With read-only stores attached, a past date is read from the primary store only, and the answer says so. The catalogs' valid time is when an object was registered: it is not a business date.

Freshness in business dates

FRESHNESS answers in business dates, rendered by code: computed up to 2023-10-31 (business date), last run 2026-10-09 11:00, compared with the process's TimeManager (up to date, or N business date(s) behind), naming the manager and how many other processes share it. At a past valid time, only the runs started by then count.

Two times: the store's and each dataset's

The agent answers about what is registered and documented at the current valid time, unless the question names a date (above). Each dataset has its own time, fixed when it was built, and the agent states it when it is not a current snapshot:

Dataset	What the agent says
current snapshot	nothing (the common case; every dataset of a store older than 0.3.1.216)
snapshot at a fixed date (as_of=)	values as of that date; process runs do not change them
snapshot at a TimeManager's business date	the manager and its current date; "stale" means the producing processes have not run for that date
historical (build_historical_dataset)	one row per entity and period, the window; the USAGE snippet shows how to read one instant

tdfs4ds.AGENT_LIST_MAX_ITEMS = 40   # 0 = always list everything (former behaviour)

State remembered across turns

State field	What is remembered
resolved_data_domain	Domain owning the current feature
resolved_object_name	Last explicitly named feature / dataset
resolved_feature_triplet	(feature_name, entity_name, process_id, view_name)
resolved_entity_name	Entity in focus (e.g. CustomerID)
resolved_feature_list	One or many feature names currently discussed
resolved_column_sources	Column → source-table map from the last EXPLAIN turn
resolved_entity_name	Entity picked during an entity-clarification turn

Follow-up questions that omit the feature name reuse the previous one automatically. When the same feature name exists across multiple entities, the remembered entity silently disambiguates.

Ambiguous-feature entity clarification

When a feature name exists under several entities (different granularities), the agent first tries to resolve the entity without asking:

1. Classifier extraction — the intent models carry an entity_name field; "What is total_amount for pharmacy_id?" extracts pharmacy_id directly and it becomes the entity hint for the skill (top priority in the hint chain).
2. Question scan — _entity_from_question() is a deterministic safety net at every ambiguous_feature site (DEFINITION, FRESHNESS, LINEAGE, EXPLAIN): the raw question is scanned for the candidate entity names (case-insensitive, _/space-normalised, multi-key entities match on any key component) and auto-resolves when exactly one matches.

Only when neither layer resolves it does skill_definition return error="ambiguous_feature" with the valid entity list. Rather than re-classifying the user's clarifying reply from scratch — which used to spiral into nonsense clarification loops — the agent resolves it deterministically:

1. node_skill_definition raises a tracked awaiting_feature_entity_clarification pending action (via _ambiguous_feature_pending) carrying the entity list, the original feature, and the origin intent. synthesize_answer renders a numbered menu — a deterministic EN/FR template, not an LLM call (an LLM-rendered clarification was observed embellishing the question with a guessed generic definition of the feature).
2. On the next turn, _node_classify_impl short-circuits the LLM classifier and resolves the reply through _resolve_clarification_choice (digit / ordinal / fuzzy label).
3. Role separation + intent inheritance — the entity comes from the reply, while the feature (object_name) and intent (DEFINITION) are inherited from context. So both a bare "1" and a mixed-concept reply like "the total_amount feature of the provider_id entity" resolve correctly. The chosen entity is stored in resolved_entity_name and used as skill_definition's entity hint — the answer comes back scoped, with no second menu.

Typo recovery

The classifier pins intent to question shape, not name recognition: "what is ml_rsk_scre?" stays DEFINITION even though the token matches nothing (it may be a typo), instead of falling into SEARCH's speculative synthesis. When the documentation lookup then misses, _fuzzy_match_feature (difflib, cutoff 0.75) kicks in: a single close candidate is answered directly with a visible correction note ("'ml_rsk_scre' was not found — answered for the closest match 'ml_risk_score'"); multiple candidates produce a did-you-mean list. DEFINITION and FRESHNESS auto-correct this way, matching the existing LINEAGE / EXPLAIN behaviour.

Lineage grounding

Lineage answers only ever mention objects that exist in a dependency graph. The cross-turn cache is consulted only when a previous LINEAGE turn actually traced upstream objects — the bare feature → process_view association written by a DEFINITION turn is treated as a cache miss and triggers a real SHOW VIEW/SHOW TABLE walk. The summarisation prompt forbids inventing object names; when no upstream is found the agent says so explicitly instead of calling the LLM.

At a past date ("where did X come from on 30 June 2023?") the agent reads the persisted graph VALIDTIME AS OF that date (load_lineage_digraph(as_of=...)): lineage writes are diff-based, so the graph's history is real. The cache and the SHOW VIEW walk describe today and are not used then; an object absent from the graph at that date is answered as "no lineage recorded at that date". TimeManager and FilterManager objects are named as drivers (they set the business date or the segment), not as data sources.

How EXPLAIN states the grain and the filters

The formula alone does not say what a value means: SUM(Amount) per customer and month, over completed transactions only, at the current business date, is a different feature from the same expression without the WHERE. The agent reads the process view's SQL by code (agent/sql_context.py, no LLM) and appends a fixed block to every EXPLAIN answer:

Computed at / on which rows
- aggregated per `t.CustomerID`
- only rows where `t.Status = 'OK'`
- only rows with a match in `DB_LAB.customers` (inner join on `c.CustomerID = t.CustomerID`)
- at the current business date of the TimeManager `DB_LAB.BUSINESS_DATE`

It follows the expression through the derived tables and CTEs it reads, and reports the GROUP BY keys or the window (PARTITION BY / ORDER BY), WHERE, HAVING, QUALIFY, DISTINCT, inner joins (rows without a match are dropped), semi-joins (IN (SELECT ...), EXISTS), outer joins (rows kept, columns NULL), a WHERE on the optional side of a LEFT join (which makes it an inner join), and the TimeManager / FilterManager the view reads. The LLM is given the same block and asked to say it in business words; the columns used to filter or join are listed so they can be asked about next.

Evaluating the agent

tdfs4ds.evaluate_agent(scenario_file) (or the admin app's Evaluation panel) runs the agent over a scenario set and has an LLM judge score each answer on five metrics (faithfulness, completeness, relevance, robustness, efficiency). The judge is grounded on facts read from the store, not on gold prose: the expected facts are what the answer is checked against, not the answer itself.

from tdfs4ds.agent import evaluation as ev
ss = ev.generate_scenarios(seed=0)            # questions and facts from the store
ev.save_scenarios(ss, 'scenarios.json')
res = ev.run_evaluation(ss, refresh_facts='check')
open('report.html', 'w', encoding='utf-8').write(ev.build_report(res, 'html'))

Questions come from data. A feature is picked because it fits the skill: documented for DEFINITION, EXPLAIN and SEARCH; produced by a process that ran for FRESHNESS; read by a dataset for USAGE and DATASET; registered under several entities for the disambiguation case. The choice is deterministic for a seed. Every intent is generated, or listed in meta["coverage"] with the reason it was not (BUILD_DATASET creates an object: include_writes=True; ALGORITHM needs MCP).

Facts come from the same authorities as the Report. Each turn keeps a fact_spec; eval_facts.build_facts rebuilds its facts from the store:

Skill	Facts
SUMMARY, DATA_DOMAINS	feature and process counts (catalog_counts, the Report's definition), per entity, domains
DEFINITION	feature, entity, process view, first sentence of its documentation, entity variants
EXPLAIN	the expression, its grain and its filters (read from the view's SQL)
FRESHNESS	last completed business date, last run, failure, TimeManager date, dates behind, peers
LINEAGE	upstream objects, consumers, drivers (persisted graph)
USAGE, DATASET	datasets reading the feature (by FEATURE_ID, so per entity) and their entity keys
SEARCH, FEASIBILITY	features whose documentation carries the query's words

The agent is challenged on time (category temporal), with dates taken from the store's history: the catalog at a date between two registrations, a date before anything existed, a date kept by a follow-up question then cleared by "now", a feature that did not exist yet, freshness and lineage at a past date, a dataset's own time, and a year alone (which is not a date).

The judge applies the agent's time rules. Each turn carries time_context: the valid time it is about, computed with the agent's own _resolve_as_of. The agent's trace says which valid time it answered at (**As of**), and the two are compared. Names that exist today but not at the turn's date are absent_names: mentioning one is a violation, even though the name is real. The judge is told the run's date and when the store was first populated, and that business date, run time and catalog time are three different things.

Checks done by code. Before the LLM reads the answer, the code checks the required names, counts (digits or words, EN/FR) and dates (ISO or textual), and the violations. Faithfulness is capped at 2 on a violation, completeness at 3 when a dated answer does not state its date, robustness at 2 when the agent answered at another valid time than the conversation set.

Facts that change. run_evaluation(refresh_facts='check') (default) rebuilds every turn's facts from its fact_spec and flags turns whose facts no longer match the store (STALE in the report); 'refresh' judges on the rebuilt facts. The scenario hash covers the facts too.

Quick start

import tdfs4ds
from tdfs4ds.agent import launch_chatbot_with_index

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'vllm'
tdfs4ds.INSTRUCT_MODEL_URL      = 'https://api.example.com/v1'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = 'my-key'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'mistral-7b-instruct'
tdfs4ds.EMBEDDING_MODEL_URL     = 'https://api.example.com/v1/e5'
tdfs4ds.EMBEDDING_MODEL_MODEL   = 'bge-m3'

# One call: build the vector index (incremental) then launch the chatbot
demo = launch_chatbot_with_index(port=7860, force_rebuild=False)

Programmatic API

from tdfs4ds.agent import consumer_agent

answer = consumer_agent("What features are available?", thread_id="session-1")
answer = consumer_agent("How is nb_days_since_last_transactions calculated?", thread_id="session-1")
answer = consumer_agent("When was it last updated?", thread_id="session-1")
answer = consumer_agent("And for CustomerID?", thread_id="session-1")   # entity remembered, new feature group

consumer_agent returns just the answer string by default. Two opt-in flags change the return shape:

answer, trace_md = consumer_agent(q, thread_id="s1", return_trace=True)
answer, trace_md, acc_ctx = consumer_agent(q, thread_id="s1", return_context=True)

• return_trace=True → (answer, trace_md) — the markdown agent trace.
• return_context=True → (answer, trace_md, acc_ctx) — a 3-tuple that also returns the accumulated_context_graph dict (the cross-turn knowledge graph), so callers always get the trace alongside the context.

EXPLAIN → DEFINITION drill-down

After an EXPLAIN turn, the agent lists every variable involved in the formula with its source table. Asking "what is <column>?" resolves the column through FS_BUSINESS_DICTIONARY_COLUMNS — even when the column is not a registered feature.

USER: How is total_amount calculated?
AGENT: total_amount sums AMOUNT_NET grouped by CUSTOMER_ID.
       Variables involved:
         AMOUNT_NET  ←  SOURCE_DB.TRANSACTIONS
         CUSTOMER_ID ←  SOURCE_DB.TRANSACTIONS

USER: what is AMOUNT_NET?
AGENT: AMOUNT_NET is the net transaction amount in EUR, post-discount …

Chatbot UI

The chat in the admin app (0.3.2.0)

python -m tdfs4ds.serve opens the admin app on the Chatbot panel. The chat is part of the app (it used to be a Gradio page in an iframe):

• answers stream in. A status line under the question says what the agent is doing ("Classifying intent → FRESHNESS", "Checking data freshness");
• answers render as formatted text, with tables and lists. Charts (MCP tools) appear inside the answer;
• each answer has a Copy button (the Markdown). Its Sources line is folded;
• Copy chat in the header copies the whole conversation as Markdown;
• suggestion chips start a conversation. They name real objects of the connected store;
• Clear empties the screen and the assistant keeps the conversation. New chat starts a new one;
• the conversation is per browser. Turns from several browsers are answered one at a time, as in the Gradio chat;
• light and dark themes follow the system, or the ◐ button in the top bar;
• the layout works on a phone.

The Gradio chat is still served at /chatbot and on its own port (TDFS4DS_SERVE_CHATBOT_PORT), so existing links keep working. In admin view, the panel's Classic chat ↗ link opens it. It is also what launch_chatbot() shows in a notebook. It has the same layout as the app chat: the header buttons (Context graph, Copy chat, Clear, New chat), suggestions, and the context graph in a side column.

User view and admin view

A User / Admin switch sits in the top bar. The browser remembers the choice. TDFS4DS_SERVE_DEFAULT_VIEW=user sets the first view for new visitors (default admin, so an existing deployment looks the same).

	User view	Admin view
Chat	question, answer, charts, sources	the same, plus under each answer: intent, skill, data domain, object, time to answer, tokens and LLM calls, the timed steps, MCP tool calls, and the full trace
Context graph	hidden	Context graph button: the conversation's knowledge graph in a side panel. Drag its left edge to widen it (double-click restores the default)
Navigation	Chatbot, Report, Lineage, Objects	every panel

The switch only hides panels: it is not access control. Anyone who can open the app can switch to the admin view.

Token counts cover every model call of the turn, the intent classifier included, labelled by graph node. Before 0.3.2.0 the trace's token figures were always 0.

Tracing with Opik

The agents can send their runs to Opik:

pip install tdfs4ds[opik]
export TDFS4DS_TRACING=opik                 # or tdfs4ds.TRACING = 'opik'
export TDFS4DS_OPIK_PROJECT_NAME=tdfs4ds    # optional, the default

• What each turn records. A turn of the consumer agent (and each query-optimizer run) becomes one trace, with a span per graph node (classify, detect_domain, the skill, synthesize) and per LLM or tool call. Token usage is on the LLM spans.
• Grouping and tags. Traces are grouped into an Opik thread by the conversation's thread_id. They are tagged tdfs4ds and consumer_agent / query_optimizer, with the data domain, the MCP flag and the tdfs4ds version as metadata.
• Where traces go is Opik's own configuration, never a tdfs4ds setting: opik configure, or Opik's OPIK_URL_OVERRIDE / OPIK_API_KEY / OPIK_WORKSPACE variables. tdfs4ds reads none of them.
• Opik's own reporting is off. Left at its defaults, the Opik SDK sends product analytics to stats.comet.com and error reports to Sentry. When tdfs4ds starts tracing it switches both off, so traces go only to your Opik. Set TDFS4DS_OPIK_VENDOR_TELEMETRY=true to keep Opik's defaults.
• Tracing never breaks a turn. A missing package, a bad configuration or an unreachable server logs one warning, and the agent answers untraced.
• Checking it. tdfs4ds.check_configuration() shows an Agent tracing row: off, or installed and configured, and the kind of target (Opik cloud or self-hosted).

On Teradata AI Studio, the AgentOps package turns this on (see AgentOps deployment).

Agent trace panel

The chatbot includes a collapsible Agent trace accordion showing full execution:

• Intent Classification — detected intent, object name (this turn vs. remembered)
• Feature Context — triplet (feature, entity, process ID, view), feature list, column sources
• Dataset Context — dataset ID and feature count (if applicable)
• Pending Action — multi-step flow state (e.g. awaiting dataset name)
• DATA_DOMAIN — available domains, active domain, resolution source
• Skill Executed — name + input object
• Skill Result — structured output (for single features) or context graph (for multi-object)
• Errors include per-step diagnostic messages

Context knowledge graph panel

A second collapsible accordion displays an interactive Plotly network graph visualising the agent's accumulated context:

• Nodes (colour-coded by type):
• Domain (blue diamond)
• Entity (green square)
• Features (orange circles)
• Process views (grey squares)
• Datasets (purple triangles)
• Column sources (red nodes, when EXPLAIN was used)
• Edges show relationships: domain → entity → features → views → datasets

The graph is rebuilt after each turn to reflect new resolutions, multi-object DEFINITION expansions, and EXPLAIN column mappings.

Admin parameters panel (launch_chatbot_app)

launch_chatbot_app() is an additive wrapper around the chatbot: same chat UI, plus a collapsible ⚙️ Parameters panel for changing configuration live — without restarting the process. (launch_chatbot and launch_chatbot_with_index are unchanged; use them when you don't need the panel.)

from tdfs4ds.agent import launch_chatbot_app

# Same prerequisites as launch_chatbot (a connected context + built index).
demo = launch_chatbot_app(port=7860)

The panel edits the same globals as the agent

The admin panel does not keep its own configuration. Every field maps directly to the same tdfs4ds.* global that the consumer agent already reads (tdfs4ds.SCHEMA, INSTRUCT_MODEL_*, EMBEDDING_MODEL_*, VECTOR_STORE_BACKEND, CHROMA_PATH, …). Applying a change in the panel is exactly equivalent to setting that global in Python, so it also affects any consumer_agent(...) / query_optimizer_agent(...) calls made from the same process afterwards. Values are pre-filled from the current globals when the app starts — including whatever was loaded from env vars / .env / JSON config (see Configuration).

The panel has four groups; each has an Apply button and a status line:

Group	Fields	What Apply does
Feature store database	database name	tdfs4ds.connect(database=…, create_if_missing=False). Validated: a database that does not contain a tdfs4ds feature store is rejected and the active store is not changed. On success: sets tdfs4ds.SCHEMA, rebuilds the index, resets the agent, clears the chat.
Vector store	backend (chroma/teradata), Chroma path; Refresh / rebuild button	Sets VECTOR_STORE_BACKEND / CHROMA_PATH and rebuilds. Refresh alone re-runs build_vector_index(force_rebuild=True) — use it after the business dictionary changes, since the local Chroma index does not auto-update.
Instruct model	provider, model, URL, API key, proxy	Sets INSTRUCT_MODEL_* (incl. INSTRUCT_MODEL_PROXY) and resets the agent. The next LLM call builds a fresh client from these globals.
Embedding model	provider, model, URL, API key, proxy, dimension	Sets EMBEDDING_MODEL_*, then rebuilds the index (embeddings changed) and resets the agent.

Any change that affects the data or the model rebuilds the index when needed, resets the agent (reset_consumer_agent()), rotates the conversation thread, and clears the chat so the session starts clean against the new configuration.

When run via the API server (python -m tdfs4ds.serve), this admin app is served automatically on a second port — see Deployment → OpenAI-compatible API server.

Database choice is process-wide — isolate per user with separate processes

Because the panel mutates process-wide globals, switching the feature-store database changes it for everyone served by that process. To give each user their own database, run a separate process / container per user (one container per user, each with its own TDFS4DS_SCHEMA) rather than sharing one — see Per-user database choice. There is no per-session database isolation within a single process.

Business dictionary & retrieval architecture

The agent answers from a 3-level documentation hierarchy designed for chunk-free hierarchical RAG, using both structured SQL and vector search.

Documentation tables (all VALIDTIME temporal)

Level	Table	What it stores	Populated by
0	FS_BUSINESS_DICTIONARY_OBJECTS	Object overview (BUSINESS_DESCRIPTION)	document_process, document_dataset_incremental
1	FS_BUSINESS_DICTIONARY_SECTIONS	Named sections per object (OVERVIEW, ENTITY, FEATURE_THEMES, BUSINESS_QUESTIONS, INTENDED_AUDIENCE)	document_dataset_incremental
2	FS_BUSINESS_DICTIONARY_COLUMNS	Per-column descriptions (feature name → view name → description)	document_process, document_dataset_incremental

What gets vectorised

The vector store maintains 3 collections, populated by build_vector_index():

• {prefix}_objects ← BUSINESS_DESCRIPTION from OBJECTS
• {prefix}_sections ← SECTION_CONTENT from SECTIONS
• {prefix}_columns ← BUSINESS_DESCRIPTION from COLUMNS

The prefix is tdfs4ds for Chroma (cosine-distance HNSW collections) and tdfs4ds.TERADATA_VECTOR_STORE_PREFIX for the Teradata backend.

Documents are embedded in batches of 8 to respect rate limits (OpenAI, TEI, etc.).

Backend selection

tdfs4ds.VECTOR_STORE_BACKEND = 'chroma'    # default — local or remote ChromaDB
# or
tdfs4ds.VECTOR_STORE_BACKEND = 'teradata'  # Teradata Enterprise Vector Store

build_vector_index() and search_vector_index() dispatch on this setting; all skill code remains unchanged. The Teradata backend persists the three collections as named TeradataVectorStores directly in Vantage and accepts either a LangChain Embeddings object (default — keeps your vLLM / TEI / OpenAI-compatible setup) or a teradatagenai.TeradataAI object (set TERADATA_AI_API_TYPE to opt in). See LLM & embedding models for the full configuration.

Hybrid retrieval strategy

Each skill uses a mix of structured SQL (for precision) and vector search (for recall when names don't match exactly):

Skill	Structured lookups	Vector search
SEARCH	BD_SECTIONS (FEATURE_THEMES, BUSINESS_QUESTIONS), BD_OBJECTS, catalog	Yes — semantic priority, results prepended
DEFINITION	Feature catalog → process view → BD_COLUMNS; BD_OBJECTS + BD_SECTIONS for datasets	Fallback only (when SQL returns nothing)
USAGE	BD_SECTIONS (INTENDED_AUDIENCE, OVERVIEW)	Fallback only
FRESHNESS	Feature catalog → process catalog → follow_up_report (single feature); dataset catalog → all process_ids for the dataset's features → aggregated follow_up_report (dataset)	None
SUMMARY	Feature catalog + BD_SECTIONS (FEATURE_THEMES, OVERVIEW)	None
LINEAGE	build_teradata_dependency_graph (DDL walk)	None
EXPLAIN	SHOW VIEW DDL	None

For multi-object DEFINITION queries, the skill also builds a context_graph dict (features, shared views, shared datasets) stored in skill_result and visualised in the chatbot's context panel. EXPLAIN does the same — _skill_explain_multi returns a context_graph (carrying each feature's process_id + view_name) and a resolved_triplet focus pointer, which node_skill_explain merges into accumulated_context_graph, so a follow-up turn after an EXPLAIN can resolve from cache.

Vector index management

from tdfs4ds.agent import build_vector_index, search_vector_index

build_vector_index(force_rebuild=False)    # incremental: indexes only new rows (Chroma)
build_vector_index(force_rebuild=True)     # full rebuild: wipe collections and reindex
search_vector_index('total_amount', k=5)   # raw vector hits (not used directly by skills)

Listing available models

from tdfs4ds.agent import list_instruct_models, list_embedding_models

list_instruct_models()
list_embedding_models(sub_paths=['e5', 'code'])

Notebooks

• notebook dev/genai/09 - Consumer Agent Chatbot with tdfs4ds.ipynb — architecture walkthrough, 7-intent test suite, 4-turn demo.
• notebook dev/genai/10 - Launch Consumer Agent Chatbot.ipynb — minimal 4-cell launch.