tdfs4ds.agent.consumer_agent¶
Consumer agent skills, intent classifier, and answer synthesizer.
Business consumers (data managers, study managers, regulatory affairs) interact
with the feature store in natural language. This module provides:
- intent_classifier — routes questions to the right skill
- Five skills: search, definition, usage, freshness, summary
- synthesize_answer — converts structured skill output to plain language
module-attribute
¶
_STYLE_DIRECTIVES = {'concise': 'Style directive: Be terse. 3-6 short bullets or two compact sentences. No analogies, no preamble, no closing summary. Skip narrative framing.', 'standard': None, 'verbose': 'Style directive: Long-form is allowed. Use sectioned bullets, analogies, and worked examples when they help the reader.'}
module-attribute
¶
_GROUNDING_DIRECTIVE = 'GROUNDING RULE: Ground all claims strictly in the documentation and catalog information provided. You MAY volunteer a helpful interpretation or idea beyond what is documented, but you MUST explicitly label it — prefix it with "Not documented — interpretation:" (in English) or "Idée non documentée :" (in French) — so the reader can tell a stored fact from an inference. NEVER present an inference as a documented fact. Omit sections (e.g. Inclusion/Exclusion Criteria, Interpretation) entirely when the provided documentation contains no content for them — do not write placeholder text like "not documented" for every structural section.'
module-attribute
¶
module-attribute
¶
_MONTHS = {'january': 1, 'february': 2, 'march': 3, 'april': 4, 'may': 5, 'june': 6, 'july': 7, 'august': 8, 'september': 9, 'october': 10, 'november': 11, 'december': 12, 'janvier': 1, 'fevrier': 2, 'février': 2, 'mars': 3, 'avril': 4, 'mai': 5, 'juin': 6, 'juillet': 7, 'aout': 8, 'août': 8, 'septembre': 9, 'octobre': 10, 'novembre': 11, 'decembre': 12, 'décembre': 12}
module-attribute
¶
_NOW_RE = compile("\\b(now|today|currently|current state|at present|right now|maintenant|aujourd'hui|actuellement|à présent|en ce moment)\\b", I)
module-attribute
¶
_REL_UNITS = {'hour': 'hour', 'hours': 'hour', 'heure': 'hour', 'heures': 'hour', 'day': 'day', 'days': 'day', 'jour': 'day', 'jours': 'day', 'week': 'week', 'weeks': 'week', 'semaine': 'week', 'semaines': 'week', 'month': 'month', 'months': 'month', 'mois': 'month', 'year': 'year', 'years': 'year', 'an': 'year', 'ans': 'year', 'annee': 'year', 'année': 'year', 'annees': 'year', 'années': 'year'}
module-attribute
¶
_REL_NUMBERS = {'a': 1, 'an': 1, 'one': 1, 'un': 1, 'une': 1, 'two': 2, 'deux': 2, 'three': 3, 'trois': 3, 'four': 4, 'quatre': 4, 'five': 5, 'cinq': 5, 'six': 6, 'seven': 7, 'sept': 7, 'eight': 8, 'huit': 8, 'nine': 9, 'neuf': 9, 'ten': 10, 'dix': 10, 'twelve': 12, 'douze': 12}
module-attribute
¶
module-attribute
¶
_REL_PATTERNS = (compile(f'\b(?:last|latest|lastest|past|previous|recent|preceding)\s+(?:({_REL_NUM_RE})\s+)?({_REL_UNIT_RE})\b', I), compile(f'(?:\b({_REL_NUM_RE})\s+)?\bderni[eè]re?s?\s+({_REL_UNIT_RE})\b', I), compile(f'\b()({_REL_UNIT_RE})\s+derni[eè]re?\b', I), compile(f'\bdepuis\s+({_REL_NUM_RE})\s+({_REL_UNIT_RE})\b', I))
module-attribute
¶
module-attribute
¶
_SOURCE_DATA_RE = compile('\\b(?:which|what|where|any|is\\s+there|are\\s+there|find|look(?:ing)?\\s+for|search)\\b[^?]*\\b(?:tables?|columns?|fields?|data|source\\s+tables?|views?)\\b|\\b(?:quelles?|quels?|o[uù]|y\\s+a-t-il|existe-t-il|trouver|cherche\\w*)\\b[^?]*\\b(?:tables?|colonnes?|champs?|donn[ée]es|vues?)\\b', IGNORECASE)
module-attribute
¶
_SOURCE_TOPIC_RE = compile('\\b(?:about|on|regarding|related|relating|concerning|deal(?:s|ing)?|contain(?:s|ing)?|hold(?:s|ing)?|has|have|having|gives?|records?|per|store[sd]?|describ\\w*|with|for|sur|concernant|li[ée]e?s?|contien\\w*|avec|pour|traitant|parlant|a|ont|donne\\w*|enregistre\\w*|par)\\b', IGNORECASE)
module-attribute
¶
_FEATURE_STORE_WORDS_RE = compile('\\b(?:features?|datasets?|caract[ée]ristiques?|jeux?\\s+de\\s+donn[ée]es|feature\\s+store|data[\\s_-]?domains?|domaines?\\s+de\\s+donn[ée]es|process(?:es|us)?)\\b', IGNORECASE)
module-attribute
¶
_SOURCE_WORDS_RE = compile('\\b(?:tables?|columns?|fields?|source|raw|colonnes?|champs?|brutes?|any\\s+data|des\\s+donn[ée]es)\\b', IGNORECASE)
module-attribute
¶
_SEARCH_STOP_WORDS = frozenset({'a', 'an', 'the', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could', 'should', 'may', 'might', 'shall', 'can', 'what', 'which', 'who', 'how', 'when', 'where', 'why', 'i', 'me', 'my', 'we', 'our', 'you', 'your', 'it', 'its', 'this', 'that', 'these', 'those', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'about', 'from', 'by', 'as', 'into', 'through', 'during', 'there', 'than', 'so', 'if', 'any', 'all', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'not', 'only', 'own', 'same', 'then', 'than', 'too', 'very', 'just', 'show', 'me', 'give', 'list', 'find', 'get', 'tell', 'want', 'need', 'look', 'looking'})
module-attribute
¶
_RISK_HIGHER_CLAIM_PATTERN = 'higher\\s+(?:values?\\s+)?(?:means?|=|is|indicate[sd]?|signal[s]?|suggest[s]?)?\\s*(?:a\\s+)?(?:greater|more|higher)?\\s*(?:likelihood\\s+of\\s+|risk|anomal|fraud|danger)'
module-attribute
¶
_OWNER_STOP = {'THE', 'OF', 'AND', 'WITH', 'FROM', 'TABLE', 'VIEW', 'COLUMN', 'SIMILAR', 'SAME', 'THAN', 'DATA', 'VALUE', 'VALUES', 'DANS', 'TABLE', 'COLONNE', 'DE', 'DU', 'LA', 'LE', 'LES', 'DES'}
module-attribute
¶
_DATASET_FILLER_WORDS = frozenset({'the', 'a', 'an', 'my', 'our', 'this', 'that', 'dataset', 'datasets', 'data', 'set', 'table', 'view', 'le', 'la', 'les', 'un', 'une', 'du', 'des', 'de', 'd', 'jeu', 'jeux', 'donnees', 'données'})
module-attribute
¶
_CONCEPT_FILLER_WORDS = frozenset({'feature', 'features', 'indicator', 'indicators', 'variable', 'variables', 'column', 'columns', 'metric', 'metrics', 'kpi', 'kpis', 'attribute', 'attributes', 'caracteristique', 'caracteristiques', 'caractéristique', 'caractéristiques', 'indicateur', 'indicateurs', 'colonne', 'colonnes', 'métrique', 'métriques'})
module-attribute
¶
_MEANING_FILLER_WORDS = frozenset({'the', 'feature', 'features', 'column', 'variable', 'indicator', 'la', 'le', 'les', 'variable', 'indicateur', 'colonne'})
module-attribute
¶
_FRENCH_MARKERS = {'est', 'sont', 'quoi', 'quel', 'quelle', 'quels', 'quelles', 'comment', 'pourquoi', 'les', 'des', 'une', 'cette', 'ces', 'dans', 'pour', 'avec', 'que', 'qui', 'où', 'donnée', 'données', 'derrière', 'signifie'}
module-attribute
¶
_DOWNSTREAM_RE = compile('\\b(depend(s|ed)?\\s+on\\s+(it|this|that|the|x|\\w+)|what\\s+(uses|consumes|depends|reads|relies)|used\\s+by|consumed\\s+by|consumers?|downstream|impact(ed|s)?\\b|affect(s|ed)?|who\\s+uses|qui\\s+(utilise|dépend)|en\\s+aval|impacte)\\b', I)
module-attribute
¶
_UPSTREAM_RE = compile("\\b(where\\s+(does|do).*(come|comes)\\s+from|comes?\\s+from|sources?|upstream|depends?\\s+on\\s+what|built\\s+from|derived\\s+from|d['e]?où|en\\s+amont|provient|feeds?|fed\\s+by|alimente)\\b", I)
module-attribute
¶
module-attribute
¶
_STO_SCORING_UUID_RE = compile('TDS_STO_SCORING(?:_LIVE)?_([0-9a-fA-F]{8}_[0-9a-fA-F]{4}_[0-9a-fA-F]{4}_[0-9a-fA-F]{4}_[0-9a-fA-F]{12})', IGNORECASE)
module-attribute
¶
_ALGO_CODE_KEYWORDS = {'code', 'source', 'package', 'packages', 'library', 'libraries', 'import', 'algorithm', 'algorithms', 'sklearn', 'scikit', 'python', 'paquet', 'paquets', 'librairie', 'bibliothèque', 'algorithme', 'modèle'}
module-attribute
¶
_PY_STDLIB = {'json', 'os', 'sys', 're', 'math', 'time', 'datetime', 'io', 'base64', 'pickle', 'collections', 'itertools', 'functools', 'typing', 'logging', 'warnings', 'abc', 'copy', 'random', 'string', 'csv', 'decimal'}
module-attribute
¶
_NUMERIC_TYPE_TOKENS = ('FLOAT', 'DECIMAL', 'NUMERIC', 'NUMBER', 'INT', 'BYTEINT', 'SMALLINT', 'REAL', 'DOUBLE')
module-attribute
¶
module-attribute
¶
_SCORE_NAME_TOKENS = ('combined_score', 'ml_risk_score', 'score', 'zscore', 'z_score', 'risk', 'anomaly')
module-attribute
¶
_NUMERIC_NAME_HINTS = ('score', 'zscore', 'z_score', 'amount', 'count', 'ratio', 'rate', 'velocity', 'avg', 'stddev')
module-attribute
¶
module-attribute
¶
_KIND_PROBLEMS = {'unrecognised': ('I did not recognise the kind of dataset.', "Je n'ai pas reconnu le type de dataset."), 'as_of_needs_date': ('A snapshot at a date needs the date (and optionally the time).', "Un instantané à une date demande la date (et éventuellement l'heure)."), 'history_needs_window': ("A history needs its window: two dates, a duration, or 'all'.", 'Un historique demande sa fenêtre : deux dates, une durée, ou « tout ».'), 'ambiguous_date': ('A date like 03/04/2026 can be read two ways: please write it as YYYY-MM-DD.', 'Une date comme 03/04/2026 se lit de deux façons : écrivez-la AAAA-MM-JJ.'), 'unreadable_date': ('I could not read that date.', "Je n'ai pas pu lire cette date."), 'unreadable_duration': ('I could not read that duration.', "Je n'ai pas pu lire cette durée.")}
module-attribute
¶
_KIND_TS_RE = compile('\\b(\\d{4}-\\d{2}-\\d{2})(?:[ T](\\d{1,2}:\\d{2}(?::\\d{2})?)\\s*(Z|UTC|GMT|[+-]\\d{2}:?\\d{2})?)?(?![\\d:])', IGNORECASE)
module-attribute
¶
module-attribute
¶
_DURATION_TERM_RE = compile('(\\d+)\\s*(?:derni[eè]re?s?\\s+|last\\s+|past\\s+)?(years?|yrs?|months?|weeks?|days?|hours?|minutes?|ann[ée]es?|ans?|mois|semaines?|jours?|heures?)\\b', IGNORECASE)
module-attribute
¶
_FR_UNITS = {'an': 'years', 'ans': 'years', 'annee': 'years', 'annees': 'years', 'année': 'years', 'années': 'years', 'mois': 'months', 'semaine': 'weeks', 'semaines': 'weeks', 'jour': 'days', 'jours': 'days', 'heure': 'hours', 'heures': 'hours'}
module-attribute
¶
_HISTORY_WORDS = compile('\\b(histor\\w*|over time|window|p[ée]riode|between|entre|from|since|depuis|last|past|derni[eè]re?s?|[ée]volution|until|jusqu)\\b', IGNORECASE)
module-attribute
¶
_CURRENT_WORDS = compile("\\b(current|now|latest|today|actuel\\w*|courant\\w*|maintenant|aujourd'hui|default|d[ée]faut)\\b", IGNORECASE)
module-attribute
¶
_FULL_WORDS = compile('\\b(all|full|entire|whole|complete|tout|toute|complet|compl[èe]te|enti[èe]re?)\\b', IGNORECASE)
module-attribute
¶
_NAME_STOPWORDS = {'call', 'it', 'name', 'named', 'the', 'dataset', 'please', 'use', 'as', 'create', 'build', 'appelle', 'le', 'la', 'nom', 'nomme', 'jeu', 'de', 'données', 'donnees', 'merci', 'avec'}
Bases: BaseModel
One sub-question of a COMPOUND (multi-purpose) question.
Populated only when a single user message carries MORE THAN ONE intent over
interrelated subjects (e.g. "what is provider_z_score AND where does it come
from?" = DEFINITION + LINEAGE). Single-purpose questions leave the parent
sub_intents field null.
class-attribute
instance-attribute
¶
intent: str = Field(description='Intent for this sub-question (e.g. DEFINITION, LINEAGE, FRESHNESS, EXPLAIN)')
class-attribute
instance-attribute
¶
object_name: Optional[str] = Field(default=None, description='The object/feature/column this sub-question is about')
class-attribute
instance-attribute
¶
entity_name: Optional[str] = Field(default=None, description='Entity/granularity for this sub-question, if named')
class-attribute
instance-attribute
¶
depends_on_prev: bool = Field(default=False, description="True when this sub-question needs the PREVIOUS sub-question's answer to resolve (sequential — e.g. resolve the owning object in DEFINITION, then LINEAGE-position it). False when it is independent (parallel).")
Bases: BaseModel
Structured output for intent classification (hardwired fallback — all 10 intents).
instance-attribute
¶
intent: Literal['SEARCH', 'DEFINITION', 'USAGE', 'FRESHNESS', 'SUMMARY', 'DATA_DOMAINS', 'FEASIBILITY', 'LINEAGE', 'EXPLAIN', 'DATASET', 'BUILD_DATASET', 'RECAP', 'HELP', 'OUT_OF_SCOPE']
class-attribute
instance-attribute
¶
object_name: Optional[str] = Field(None, description='Name of any documented item mentioned: a feature, a documented column, a table, a view, or a dataset')
class-attribute
instance-attribute
¶
domain: Optional[str] = Field(None, description='DATA_DOMAIN — the feature-store project/namespace that isolates features; NOT a feature theme. Copy verbatim if the user names one, else null.')
class-attribute
instance-attribute
¶
entity_name: Optional[str] = Field(None, description='Entity / granularity level if the user names one (the key a feature is computed for), copied verbatim')
class-attribute
instance-attribute
¶
user_preferences: Optional[dict] = Field(None, description="User preferences or constraints (e.g. {'currency': 'USD', 'output_format': 'JSON'})")
class-attribute
instance-attribute
¶
sub_intents: Optional[List['_SubIntent']] = Field(None, description='Ordered sub-questions when ONE message carries MORE THAN ONE intent over interrelated subjects; null for single-purpose questions')
class-attribute
instance-attribute
¶
as_of: Optional[str] = Field(None, description="The date the question asks about the feature store AT (e.g. 'on 30 June 2023', 'as of 2023-06-30', 'at the end of June 2023'), copied as written. Null when the question is about the current state.")
Bases: BaseModel
One candidate interpretation of an ambiguous query.
class-attribute
instance-attribute
¶
label: str = Field(description="One-line description shown to the user (e.g. 'Define <the named feature>')")
class-attribute
instance-attribute
¶
intent: str = Field(description='Intent that best matches this interpretation (e.g. DEFINITION, EXPLAIN)')
class-attribute
instance-attribute
¶
object_name: Optional[str] = Field(default=None, description='Feature or dataset name for this interpretation')
Bases: BaseModel
A column referenced in a feature formula and its source table/view.
class-attribute
instance-attribute
¶
column_name: str = Field(description='Column or alias name as it appears in the SQL expression (bare name, no schema)')
class-attribute
instance-attribute
¶
source_table: str = Field(description='Source table or view the column comes from (bare name, no schema prefix)')
Bases: BaseModel
Structured output for the EXPLAIN skill.
class-attribute
instance-attribute
¶
explanation: str = Field(description='Plain business language explanation of the feature: data sources (by purpose, not table name), filters in business terms, how the value is computed, and what it represents. Maximum 6 sentences. No SQL syntax, no column names, no table names.')
class-attribute
instance-attribute
¶
calculation_sql: Optional[str] = Field(None, description="The SQL expression that computes this specific feature, extracted verbatim from the SELECT list. If the outer SELECT merely passes through an alias defined in a subquery, drill down into that subquery to find the real computation — repeat for every nesting level until the actual arithmetic or aggregation is found. Include only the expression itself (e.g. 'CAST(CURRENT_DATE - MAX(transaction_date) AS INT)'), never the full query. Return null only when the expression genuinely cannot be resolved at any nesting level.")
class-attribute
instance-attribute
¶
calculation_latex: Optional[str] = Field(None, description="LaTeX representation of the calculation in display-math notation, WITHOUT surrounding delimiters. Populate ONLY when the user explicitly requested a formula, equation, or LaTeX. Use \\text{} for business names and standard LaTeX for operators (\\max, \\sum, \\frac, \\cdot, etc.). Example: r'\\text{nb\\_days} = \\text{TODAY} - \\max(\\text{transaction\\_date})'. Return null when not requested or when the expression is too complex to render meaningfully as a single formula.")
class-attribute
instance-attribute
¶
source_columns: Optional[list[SourceColumnRef]] = Field(None, description="For every distinct column or alias referenced in the feature's calculation expression, provide its bare name and the source table or view it originates from (resolve through subquery aliases to the real base table/view wherever possible). Bare names only — no schema prefix. Omit entity-key columns (e.g. primary-key columns). Return null if the formula contains no identifiable column references.")
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Return a follow-up report as a tdml.DataFrame, optionally filtered by
process_id and/or a filtermanager's applied filter.
Behavior by arguments
- process_id is None and filtermanager is None: Return all rows from SCHEMA.FOLLOW_UP_NAME, sorted by START_DATETIME desc.
- process_id is not None and filtermanager is None: Return rows for the given PROCESS_ID.
- process_id is not None and filtermanager is not None:
Return rows for the given PROCESS_ID whose APPLIED_FILTER matches the
JSON_AGG of
filtermanager's columns coming from its schema/view. - process_id is None and filtermanager is not None:
Return rows whose APPLIED_FILTER matches the JSON_AGG of
filtermanager(no PROCESS_ID constraint).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filtermanager
|
Optional[object]
|
An object exposing |
None
|
process_id
|
Optional[str]
|
Optional process identifier used to filter by PROCESS_ID. |
None
|
Returns:
| Type | Description |
|---|---|
|
tdml.DataFrame: The resulting dataframe sorted by START_DATETIME (descending). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If the query fails. |
Return follow-up rows for several process_ids in a single query.
Batched counterpart to follow_up_report(process_id=…): issues one
WHERE PROCESS_ID IN (…) query instead of one round-trip per process.
Callers group the result by PROCESS_ID in pandas to reproduce the
per-process aggregation. Returns a tdml.DataFrame sorted by
START_DATETIME descending.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process_ids
|
Iterable of process identifier strings. Empty/None entries are dropped; duplicates are collapsed. |
required |
Returns:
| Type | Description |
|---|---|
|
tdml.DataFrame with the same columns as the follow-up table. When the |
|
|
id list is empty, an empty result ( |
|
|
caller's downstream pandas logic still works. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the query fails. |
llm.with_structured_output(schema), resilient to a gateway that does
not enforce the schema it was asked for.
The single seam every structured-output call site goes through — one in the classifier, one in EXPLAIN, seven in the query optimizer, all of which fail identically on a relaying gateway. A new call site added directly against the model would silently miss this (there is a test that forbids it).
An explicit method= from the caller always wins, and a configured
STRUCTURED_OUTPUT_METHOD short-circuits the probe.
(input, output) tokens of an LLMResult, whatever the provider's shape.
A LangChain callback that records every LLM call of a turn in the tracker.
Only _llm_call recorded usage, so the intent classifier, EXPLAIN's
structured output and the JSON calls were never counted and a turn could
show 0 tokens. Put in the turn's run config (graph._run_config), this
sees every model call made inside the graph, labelled by its node; the runs
_llm_call already recorded carry :data:_TOKENS_RECORDED_TAG and are
skipped. Never raises.
List available instruct/chat models from the configured provider.
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns: MODEL_ID, OWNED_BY (OpenAI-compatible) or |
DataFrame
|
MODEL_ID, PROVIDER, MODALITY (Bedrock). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the provider is not recognized or not configured. |
Return a LangChain chat model based on the active provider.
Injects tdfs4ds config variables into environment so that boto3, Azure, and Google Cloud clients pick up credentials and endpoints automatically.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_tokens
|
Optional[int]
|
Optional cap on completion tokens. Useful when the model has a small total context window and the caller knows the expected output is short (e.g. structured candidate generation). |
None
|
callbacks
|
Optional list of LangChain callback handlers (e.g. for token usage tracking). Passed through to the model constructor. |
None
|
Returns:
| Type | Description |
|---|---|
|
A LangChain BaseChatModel instance. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the provider is not recognized. |
Return the style directive line(s) for the active AGENT_ANSWER_STYLE.
Returns an empty string when the style is unset, unknown, or 'standard'.
Grounding directive prepended to every _llm_call system prompt.
Instructs every skill to stay grounded in the provided docs while explicitly labelling non-grounded interpretations (so the judge does not penalise them as hallucinations). Also suppresses empty boilerplate sections in DEFINITION answers.
Name the answer language explicitly, decided once by _question_is_french.
"Answer in the same language as the question" was not enough: an English question was answered in French (seen live, 0.3.1.224), the prompt itself quoting French text and the documentation possibly being French.
Execute a single LLM chat completion with token tracking.
The active tdfs4ds.AGENT_ANSWER_STYLE is prepended to the system prompt
so every answer (search, definition, explain, freshness, ...) inherits the
same verbosity preference. The grounding directive is also prepended to
instruct the model to label non-grounded interpretations explicitly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
system_prompt
|
str
|
System message content. |
required |
user_prompt
|
str
|
User message content. |
required |
skill_name
|
str
|
Name of the skill calling this (for token tracking). |
'unknown'
|
Returns:
| Type | Description |
|---|---|
str
|
The assistant's response text. |
_llm_call_json(system_prompt: str, user_prompt: str, *, schema=None, skill_name: str = 'unknown', default=None, expect=None)
Ask for JSON through whatever constraint this endpoint actually honours.
:func:_llm_call is a prose call — an unconstrained invoke that
returns text. Five callers were nevertheless asking it for JSON and
hand-stripping markdown fences afterwards, which works on an endpoint that
enforces a response format and degrades silently on one that does not.
Two of them wrote json.loads(x) if x.startswith("[") else [], so a model
answering in prose produced an empty result with no error and no
warning — on the SEARCH path, among others.
That is not hypothetical. Measured on a live LiteLLM gateway, eight of nine
constraint syntaxes were accepted and ignored, and the model answered
**Classification:** ... to a request for JSON. These calls also prepend
the grounding and style prefixes, which push towards prose.
So the same prompt now goes through the provider dispatcher — the one the
documentation generator uses, which honours STRUCTURED_OUTPUT_METHOD
(tool calling included) and falls back once to an unconstrained call — and
is parsed by _robust_json_parser rather than a local regex. On failure
it returns default and says so in the log: a silent empty is the thing
being fixed, so it must not be reintroduced here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema
|
plain JSON Schema for the reply, or |
None
|
|
expect
|
|
None
|
|
default
|
returned on any failure. Callers pass their own fallback. |
None
|
Get current token usage as markdown for trace display.
Shows the usage of the current turn (since the last mark_turn()),
not the cumulative session totals — per-turn numbers are what the trace
panel and LLM-as-judge evaluations need. The session total is appended
as a single summary line.
List all DATA_DOMAINs available in the feature store.
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns |
Return the real DATA_DOMAIN that name designates, or None.
The classifier's domain field carries what the user typed ("the
"governed demand forecasting m5" data domain"): quoted, re-cased, sometimes
with a typo -- or a theme ("pharmacy") that is not a DATA_DOMAIN at all. The
catalog filters on the exact stored value, so the name must be mapped to it:
exact match ignoring case/quotes first, then a high-cutoff fuzzy match
(FUZZY_NAME_MATCH_CUTOFF). A theme maps to None.
detect_data_domain_for_object(object_name: str, prefer_domain: Optional[str] = None) -> Optional[str]
Return the DATA_DOMAIN that contains object_name as a feature name.
Searches the full feature catalog (all data domains). Falls back to the process catalog (VIEW_NAME) if the feature is not found by name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_name
|
str
|
Feature or view name to locate. |
required |
prefer_domain
|
Optional[str]
|
When the object exists in multiple domains, return this one if it is among the matches. Used to prevent silent context switches on ambiguous feature names (e.g. "Claims" appearing in multiple feature stores). |
None
|
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Matching DATA_DOMAIN string, or |
Cross-store federation limit (v0.3.1.148): the catalog scanned here is the
federated UNION across the primary + read-only stores, but resolution returns a
bare DATA_DOMAIN with NO store tag (and _with_domain only switches
tdfs4ds.DATA_DOMAIN, not the active store). With DISJOINT stores a domain
uniquely identifies its store, so this is correct. If two stores share a domain
name, _pick returns the first match (primary rows sort first → primary
wins); making resolution store-aware is the documented follow-up.
Answer questions about the feature store's DATA_DOMAINs.
Two modes:
- list (
object_nameempty) — inventory every registered DATA_DOMAIN with its distinct feature count. - detail (one or more names in
object_name, comma-separated) — for each named domain report its feature count and per-entity breakdown. Names are matched case-insensitively against the registered domains, with typo-tolerant fuzzy fallback (_best_name_similaritygated bytdfs4ds.FUZZY_NAME_MATCH_CUTOFF) mirroring the DEFINITION/FRESHNESSdid_you_meancontract.
Returns a dict carrying its own answer (direct-answer skill — no second
LLM pass), or a did_you_mean / not_found error payload.
Return {entity_name: distinct_feature_count} for one DATA_DOMAIN.
Render the domain_not_found_did_you_mean answer string.
Build a dynamic Pydantic model for intent classification from a list of intent names.
Uses Union[Literal["A"], Literal["B"], ...] which equals Literal["A", "B", ...]
per PEP 586, constructed at runtime via pydantic.create_model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
intents
|
list
|
List of uppercase intent names (e.g. |
required |
Returns:
| Type | Description |
|---|---|
|
A Pydantic BaseModel subclass with an |
|
|
the provided values, plus |
|
|
and |
Collect the string values out of a Literal/Union[Literal, ...].
_build_intent_model builds Union[Literal["A"], Literal["B"], ...], so
the values are one level down from the union. Recursive and total: anything
unrecognised yields nothing rather than raising, because the only caller uses
this to narrow a prompt and must degrade to "no filtering".
The intents the classifier is actually allowed to return, or None.
Read from the model first: that is the authority that will reject a value, so narrowing the prompt from any other source could drift from it. Falls back to the description keys, then to None (= no filtering).
Real names from the connected store for the classifier examples.
Read from the turn's catalog snapshot (already paid for by the turn), so it
costs nothing extra; deterministic (sorted) so the prompt is stable between
turns. Never raises: an unreadable catalog gives {} and every example
with a placeholder is dropped.
Fill the placeholders of one example, or None when one cannot be filled.
Braces in a store name are doubled: the examples go through a LangChain prompt template.
Build the few-shot classifier examples string from ca-* SKILL.md files.
Examples are read from the '## Classifier Examples' section of each skill's SKILL.md file, making the SKILL.md files the single source of truth. Plugin skills loaded from TDFS4DS_SKILLS_FOLDER are included automatically.
Also injects a low-confidence counter-example to teach the LLM to assign below-0.5 confidence and populate interpretations for bare topic words.
available_intents restricts the examples to intents the classifier can
actually return. This is correctness, not tidiness. build_graph drops
an unavailable intent from the descriptions and from the dynamic Pydantic
Literal, but this function used to walk the catalog unfiltered — so
with MCP disabled the prompt still carried five -> DATA_QUERY examples,
one of them literally "show me a sample", while the schema forbade that value.
The model obeyed the examples, Pydantic rejected the answer, and the turn died
with 14 literal_errors. The same leak existed for ALGORITHM, masked only
by the deterministic _looks_like_algorithm_question pre-guard.
None (the default) keeps every example, so an existing caller that passes
nothing is unchanged.
intent_classifier(question: str, intent_model=None, intent_descriptions: dict = None, context_hint: Optional[str] = None)
Classify a user question into one of the available intents.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question in natural language (French or English). |
required |
intent_model
|
Pydantic model for structured output. When |
None
|
|
intent_descriptions
|
dict
|
|
None
|
context_hint
|
Optional[str]
|
Optional context about recently mentioned features/datasets (e.g., "Last feature: provider_z_score in entity CLAIM_ID"). Added to the system prompt to help resolve vague follow-up questions. |
None
|
Returns:
| Type | Description |
|---|---|
|
An instance of |
|
|
|
First date written in text, as YYYY-MM-DD[ HH:MM:SS], or None.
ISO (2023-06-30, with an optional time), 30 June 2023 / 30 juin 2023,
June 30, 2023, and end of June 2023 / fin juin 2023 (last day).
A bare year or month is not a date: it is ignored.
A span back from now written in text: (n, unit) with unit one of
hour, day, week, month, year, else None.
"the latest week" -> (1, 'week'), "last 7 days" -> (7, 'day'),
"les 3 derniers mois" -> (3, 'month'), "depuis 2 jours" -> (2, 'day').
A unit word alone ("weekly", "per day") is not a span.
The valid time a turn asks for: (literal, clear, unreadable).
literal: a normalised TIMESTAMP literal (store_consistency.normalise_as_of)
when the question (or the classifier's as_of) names a date; clear: the
question asks about now ("now", "today", "actuellement"), which ends a date set
on an earlier turn; unreadable: the classifier named a date nobody could read
(the answer says so). Neither: the turn keeps the conversation's date.
The first line of an answer read at a past valid time ("" when current).
Context manager: read the metadata at as_of (a normalised literal or None).
Clear the per-turn memo. Call once at the start of each user turn.
Return the cached materialized frame for key or compute+store it.
fn must return the already-materialized object (e.g. a pandas DataFrame);
caching the lazy tdml.DataFrame would save nothing because re-materializing
re-runs the query. key must encode the request scope (id/name set + the
active DATA_DOMAIN) so a domain switch mid-graph cannot return a wrong slice.
On failure the exception propagates and nothing is cached.
Escape a single value for use inside a SQL string literal (double quotes).
Build an UPPER(col) IN ('A','B',…) predicate from a str or list.
Returns None when values is falsy / empty so callers can skip the clause.
De-duplicates (case-insensitively) and escapes single quotes. A single string
still produces a one-element IN list, which is equivalent to =.
(bare names, {BARE: {DB, ...}} | None per bare name) for business
dictionary lookups: DB_SOURCE.MARKET_PRICES is stored as OBJECT_NAME
MARKET_PRICES in database DB_SOURCE, and a qualified name matched
nothing (0.3.3.11: "tell me about DB_SOURCE.MARKET_PRICES" read as
undocumented). A bare name keeps every database (None).
Rows of df whose (database, name) was asked for (see _split_object_values).
_query_bd_sections(section_names: Optional[list[str]] = None, object_name: Optional[str] = None, object_names: Optional[list[str]] = None) -> pd.DataFrame
Query FS_BUSINESS_DICTIONARY_SECTIONS with optional filters.
Uses execute_sql (not from_query) because CURRENT VALIDTIME must be at the outermost level — from_query wraps in a subquery which breaks it.
object_name (singular, back-compat) and object_names (list, batched)
are merged into a single UPPER(OBJECT_NAME) IN (…) predicate so callers
can fetch many objects in one round-trip instead of looping.
_query_bd_objects(object_name: Optional[str] = None, object_names: Optional[list[str]] = None) -> pd.DataFrame
Query FS_BUSINESS_DICTIONARY_OBJECTS with optional filter.
object_name (singular, back-compat) and object_names (list, batched)
merge into one UPPER(OBJECT_NAME) IN (…) predicate.
_query_bd_columns(table_name: Optional[str] = None, column_name: Optional[str] = None, table_names: Optional[list[str]] = None, column_names: Optional[list[str]] = None) -> pd.DataFrame
Query FS_BUSINESS_DICTIONARY_COLUMNS with optional filter.
Singular kwargs (table_name / column_name, back-compat) and list
kwargs (table_names / column_names, batched) merge into
UPPER(TABLE_NAME) IN (…) AND UPPER(COLUMN_NAME) IN (…) so all features
for a set of views can be fetched in one round-trip; the caller slices the
result per (view, column) in pandas.
All statements in one request (one round trip); one by one when the request fails (an optional table missing on an older store).
The turn's catalog snapshot, or None (read federation attached, no
SCHEMA, or the core catalogs unreadable) - callers then use the accessors.
The public catalog accessor tdfs4ds.<name> (the snapshot's fallback),
looked up at call time so tests can stub it.
feature_catalog(all_data_domains).to_pandas(), from the turn snapshot.
process_catalog(all_data_domains).to_pandas(), from the turn snapshot.
dataset_catalog().to_pandas(), from the turn snapshot.
get_dataset_features(dataset_id).to_pandas(), from the turn snapshot.
get_dataset_entity(dataset_id).to_pandas(), from the turn snapshot.
At a past valid time, only the runs started by then ("what was known then"). FS_FOLLOW_UP is an event log, not a temporal table: filtered on its run time.
Run history (FS_FOLLOW_UP) of the given processes, newest first, as one
raw-SQL read memoized per turn. follow_up_report_many(...).to_pandas()
returned the same rows but cost ~3 s of teradataml metadata calls.
Every process of every domain, for whole-store scans (find-object-by- name, the allow/forbid set). Served from the turn snapshot.
Every feature of the current state, whatever the turn's valid time (to name a domain the store has today in an answer about a past date).
The feature catalog for whole-store scans, served from the turn snapshot.
Return PROCESS_ID → VIEW_NAME (+ ENTITY_ID, FEATURE_NAMES) for given ids.
Scoped, memoized replacement for process_catalog(all_data_domains=True)
.to_pandas() used only to map a handful of process_ids to their view name.
Issues one WHERE PROCESS_ID IN (…) query against the public process
catalog view (which already applies VALIDTIME), so it materializes only the
turn's rows. Cached per-turn on the id-set. Fail-open: returns an empty frame
on error (callers already tolerate a missing view name).
Count distinct (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME) feature identities.
A feature's identity in tdfs4ds is that triplet: the same name registered under two entities, or in two data domains, is two distinct features (the domain is used when the frame carries a DATA_DOMAIN column). This helper is robust to duplicate rows (e.g. a feature appearing in several datasets) and to frames that expose ENTITY_ID instead of ENTITY_NAME.
Cross-store federation limit (v0.3.1.148): the identity key does NOT include a store dimension. With read-only stores holding DISJOINT domains/entities/names (the common case) this is exact. Only when two stores independently reuse the SAME (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME) do their rows collapse to one — an undercount. Threading the FEATURE_STORE provenance column into the key here (and into domain resolution / the context-graph keys) is the documented follow-up for full collision-proof multi-store counting.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
pandas DataFrame containing at least FEATURE_NAME and an entity column (ENTITY_NAME preferred, ENTITY_ID accepted). |
required | |
group_by
|
Optional[str]
|
When given, return a |
None
|
Returns:
| Type | Description |
|---|---|
|
int when |
_get_feature_catalog_with_docs from the turn snapshot, or None.
Same resolution as the per-entity path: a feature's process is a current
process of the same data domain whose ENTITY_ID equals the feature's
ENTITY_NAME and whose FEATURE_NAMES lists it (get_feature_versions's
join, case-insensitive like the NOT CASESPECIFIC columns, entity columns in any
order), picked deterministically when several (versions); its description is the business-dictionary column of that view.
_get_feature_catalog_with_docs(feature_filter: Optional[str] = None, entity_filter: Optional[str] = None) -> pd.DataFrame
Return a DataFrame of features enriched with their process-level documentation.
Resolution chain (batched — see below):
1. Feature catalog (active domain) → all (ENTITY_NAME, FEATURE_NAME) pairs in
one materialization; filters applied in pandas.
2. get_feature_versions(entity, features) once per entity (each call already
batches all the entity's features via FEATURE_NAME IN (…)) → process_id.
3. _processes_by_id(all_process_ids) — one scoped PROCESS_ID IN (…) query
→ VIEW_NAME (replaces the old full all-domains process_catalog load).
4. _query_bd_columns(table_names=all_views, column_names=all_features) — one
scoped query → BUSINESS_DESCRIPTION, sliced per (VIEW_NAME, FEATURE_NAME) in
pandas.
The previous implementation issued one get_list_features per entity and one
_query_bd_columns per feature (plus a whole all-domains process_catalog
materialization), i.e. tens of round-trips for a multi-entity domain. This
version is ~3 scoped round-trips total.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_filter
|
Optional[str]
|
If given, only return rows for this exact feature name (case-insensitive). |
None
|
entity_filter
|
Optional[str]
|
If given, only process this exact entity name (case-insensitive). Speeds up entity-scoped lookups. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns: |
DataFrame
|
FEATURE_NAME, ENTITY_NAME, PROCESS_ID, VIEW_NAME, BUSINESS_DESCRIPTION |
_build_multi_object_context(feature_names: list, hint_entity_name: Optional[str] = None, accumulated_context: Optional[dict] = None) -> dict
Build a shared context graph for multiple features in one pass.
Calls _get_feature_catalog_with_docs() once (no filter) then: - Resolves each name to (entity, process_id, view_name, description) - Fetches BD_SECTIONS + all BD_COLUMNS for each unique VIEW_NAME (once each) - Fetches dataset membership for each feature via get_dataset_features() - Fetches BD_OBJECTS + BD_SECTIONS for each unique dataset (once each)
Returns:
| Type | Description |
|---|---|
dict
|
{ "features": [{name, entity_name, process_id, view_name, description, found_in_catalog, has_description, dataset_names}], "shared_views": {view_name: {"sections": [...], "columns": [...]}}, "shared_datasets": {ds_name: {"database", "bd_description", "sections"}}, "entity_names": [str], |
dict
|
} |
{process_id, view_name, entity, domain, feature_names} of the current
process whose view is name (bare or db.view, quotes ignored), else None.
Identify what type of object name is by querying catalogs.
Returns a dict with keys: type ("feature"|"dataset"|"entity"|"data_domain"|"process"|"unknown"), entity_name, view_name, process_id.
concepts: when no catalog name matches exactly, also try what the phrase
refers to (:func:_recognize_concept: a dataset by its name's words or its
documentation, or features by their description). concepts="datasets":
datasets only, no feature-documentation step (no vector search). Exact names
always win.
Return the union of registered feature, dataset and BD object names (uppercase).
Used to filter vector-search hits so that fabricated names (embedding-similar but not actually registered) never reach the LLM context. Empty set on failure means no filtering is applied (fail-open).
Set data_domain on each feature match whose name belongs to one data domain (in place).
0.3.5.14: features of two domains were listed together, as if one use case
(seen live). A dataset's lag / lead column (N_EVENTS_LAG_1MO) takes its
base feature's domain; an ambiguous or unknown name stays unlabelled.
Upper-case names of the datasets' offset columns (N_EVENTS_LAG_1MO); empty when unreadable.
Build LLM context from vector search results when SQL exact match fails.
Vector hits whose OBJECT_NAME is not present in the feature / dataset /
process catalogs are dropped — embedding similarity alone is not enough
proof that an object exists, and surfacing unverified names leads to
fabricated feature suggestions. Documented data outside the feature store
is reported separately (:func:_outside_store_hits), never mixed in here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
str
|
Free-text search query (object name or natural language question). |
required |
top_k
|
int
|
Maximum results per collection. |
5
|
results
|
Hits already read by |
None
|
Returns:
| Type | Description |
|---|---|
tuple[str, Optional[str]]
|
Tuple of (context_string, resolved_object_name). Both may be empty/None. |
A discovery question about documented DATA ("which table holds X?"), with no feature-store word in it. Deterministic; used to route such a question to SEARCH whatever the classifier said.
The question names feature-store content (features, datasets) and no source data.
documentation_export.feature_store_index over the turn's catalogs (fail-open: empty).
{(DB, OBJ): 'V' | 'T'} of the attached objects, once per turn (0.3.5.2). Fail-open.
(scope, kind): 'feature store' or 'outside the feature store', and the kind.
Process views reading db.name (lineage), [] when unknown (fail-open).
Vector hits on documented objects OUTSIDE the feature store, grouped by object.
[{database, object, kind, description, columns: [{name, description}],
sections: [{name, content}], read_by: [...], store}], best match first.
The object's own description is read from the business dictionary when only
a column or a section matched. Fail-open: [].
Without a vector index: documented objects outside the store whose name, description or column descriptions share a word with the question.
The fixed EN/FR block naming documented data outside the feature store.
The fixed EN/FR block for attached views / tables found by a search (0.3.5.2).
One line naming outside objects, for a question about the feature store's content.
A fixed sentence for an object (or one of its columns) OUTSIDE the store, else ''.
'' too when the catalogs could not be read (inside and outside cannot be told apart).
['DB.OBJ'] read by (out) or reading (in) db.name in the lineage graph
(edges go consumer -> source). Fail-open: [].
The fixed sentence for an object INSIDE the store (0.3.5.2): an attached object names what it reads (feature-store objects marked); a dataset or process view read by attached views names them. '' otherwise.
Return (by_entity, by_name) feature→dataset maps for the current domain.
Reverse of the dataset catalog: maps every registered feature to the datasets that expose it, so the knowledge graph can draw feature→dataset edges (otherwise features render as isolated nodes after a SUMMARY).
A feature's identity is the pair (FEATURE_NAME, ENTITY_NAME) — the same name
registered under two entities is two distinct features. The dataset _FEATURES
table carries FEATURE_ID (the feature's identity), which the feature catalog
resolves to the authoritative ENTITY_NAME. We therefore key the primary map on
(ENTITY_UPPER, FEATURE_NAME_UPPER) so a dataset is linked only to the entity it
actually exposes — never to a same-named feature under a different entity.
Returns a tuple
by_entity—{(ENTITY_UPPER, FEATURE_NAME_UPPER): [dataset_name, ...]}by_name—{FEATURE_NAME_UPPER: [dataset_name, ...]}fallback, used only for features whoseFEATURE_IDhas no catalog match (entity-less / cross-domain).
Three bounded materializations (dataset features, feature catalog, dataset catalog),
joined on FEATURE_ID in pandas (no per-row DB query); memoized per turn; fail-open
to empty maps (no dataset linkage).
The data time of one dataset: {kind, as_of, tm_view, business_date,
window_text, window_start, window_end, source}. kind is current,
as_of, time_manager, history or unknown. Memoized per turn,
never raises.
Sources, in order: the build/window rows read straight from their PUBLIC
views (never through DatasetCatalog, whose constructor creates missing
objects - a legacy store has no build table and must stay untouched); then
the view's own DDL (parse_dataset_ddl), which is how a dataset
registered before the build table existed is read; else unknown.
One sentence stating a dataset's data time; '' for a current snapshot.
Short tag for dataset listings: current / as of
Look up object_name in the dataset catalog.
Accepts "database.name" or "name" (case-insensitive).
Returns {dataset_id, dataset_name, dataset_database} or None.
A dataset's features resolved by identity, or None to use the name fallback.
A dataset feature row carries its FEATURE_ID, which identifies the feature (data domain, entity, name) in the catalog, whatever the active domain. Matching on the name alone took the first feature of that name in the active domain: a dataset of another domain whose feature names also exist elsewhere got the wrong entity, process and description (seen live: a one-entity dataset reported as two entities, with processes of another domain).
The process is the one of that domain and entity whose feature list names the feature; when several do, the one the dataset view names (FEATURE_VERSION).
Return a list of feature dicts for dataset_id, enriched with process view and docs.
Each item: {feature_name, entity_name, view_name, feature_type, description}.
Return lowercase non-stop-word tokens from the question (used to pre-filter catalog).
Cheap, dependency-free English depluralization for catalog token matching.
'pharmacies'→'pharmacy', 'providers'→'provider', 'boxes'→'box', 'claims'→'claim'. Leaves 'ss' words (e.g. 'class') and short words unchanged.
Return the tokens plus their depluralized singular forms (both kept).
Split a catalog row's name/domain/entity into lowercase word tokens.
Names use _ separators (pharmacy_z_score → {pharmacy, z, score}); the
full joined name is kept too so substring/prefix checks still work.
Number of distinct normalized question tokens that match a row's tokens.
A token matches when it equals a row token, or is a prefix of one / has one as
a prefix (bidirectional, e.g. provider ↔ provider_id). Tokens of length
≤2 only count on exact equality to avoid noise.
Truncate text to ~max_chars, cutting on a word boundary and appending '…'.
Returns the text unchanged when it already fits or max_chars <= 0 (disabled).
Find features or datasets matching a business need expressed in natural language.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question describing the business need. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Dict with key |
True when the description asserts that higher values mean more risk.
Deterministic regex over the stored business description; used to decide whether an inverted-score formula contradicts the prose.
One sentence saying name is a tag of a deep (historian) table, or None.
None when the table has no deep layout or name is one of its physical columns, so every other column answer is unchanged.
_direction_correction_note(business_description: str, calculation_sql: Optional[str]) -> Optional[str]
Return a clarification note when the stored description's risk direction
contradicts an inverted calculation formula, else None.
The DEFINITION skill trusts the stored business_description; when the
actual SQL inverts the score (1 - (...), so a lower value means higher
risk) but the prose claims "higher = riskier", we append an authoritative
correction rather than silently surfacing the contradiction (which EXPLAIN,
reading the live SQL, would expose).
skill_definition(question: str, object_name: str, hint_entity_name: Optional[str] = None, hint_table_name: Optional[str] = None, accumulated_context: Optional[dict] = None, _fuzzy_corrected: bool = False) -> dict
Explain what a feature, column, or dataset is in plain business language.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
object_name
|
str
|
Name of the feature, column, or dataset to define. |
required |
hint_entity_name
|
Optional[str]
|
Entity name remembered from a previous turn — used to auto-disambiguate when the same feature name exists across entities. |
None
|
hint_table_name
|
Optional[str]
|
Source table/view remembered from a prior EXPLAIN turn — used to narrow column-level BD lookups to the right table when the column is not a registered feature. |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
Meaningful words of a table name: HCF_FIBER_VW -> {HCF, FIBER}.
_pick_column_owners(rows: list, question: str, hint_table: Optional[str] = None, asked: Optional[list] = None) -> list
The documented owners of a column the question means (pure).
rows: [{db, table, description}] of one column. An owner named in the
question (HCF_ACCESS.HCF_FIBER_VW, or a word of it: "the Historian") wins,
then the table remembered from an earlier turn, else every owner. Words of the
qualified names and of the asked column names do not count as owner words
(HCF of HCF_ACCESS.HCF_FIBER_VW is not a hint for another HCF table).
Owners with the same description are kept once (a view exposing its table repeats it).
The SQL text comparing col_a and col_b (x.A = y.B or the reverse), else None (pure).
_documented_columns_context(names: list, question: str, column_sources: Optional[dict] = None) -> dict
Documentation of names that are columns of documented tables / views (not features).
Returns {"found": {NAME_UPPER: [owner rows]}, "lines": [...], "scope": [...],
"joins": [...], "columns": [...]}. One business-dictionary read for all names,
one for the owners' descriptions; join evidence reads the SQL of the processes
reading both owners. Fail-open: an empty result when nothing can be read.
skill_definition_multi(question: str, feature_names: list, hint_entity_name: Optional[str] = None, accumulated_context: Optional[dict] = None, column_sources: Optional[dict] = None) -> dict
Unified multi-feature definition using a shared context graph.
Calls _build_multi_object_context() once to resolve all features and fetch shared documentation (process views, datasets, entities), then makes a single LLM call with the full unified context. No consolidation pass needed.
skill_usage(question: str, object_name: str, accumulated_context: Optional[dict] = None, entity_name: Optional[str] = None) -> dict
Advise a consumer on how to safely use a feature in their work.
Enumerates ALL datasets that expose the feature (not just one) and emits a paste-ready SELECT snippet for each so the consumer can pick the right view for their dashboard.
Implements fast-path caching: queries accumulated_context_graph first (cache hit → return immediately), falls back to catalog scans (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question about usage (Tableau, Excel, regulatory, etc.). |
required |
object_name
|
str
|
Name of the feature or dataset. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state (used for cache). |
None
|
entity_name
|
Optional[str]
|
Entity name for disambiguation (used in cache lookups). |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
dict
|
|
What 'up to date' means for this dataset's own time ('' for a current one).
A dataset frozen at a date does not change when its processes run again; one following a TimeManager is stale when its producers have not completed a run for the manager's current business date; a history ends at its window end.
Paste-ready SELECT on a dataset. A historical dataset returns one row per entity and period, so its snippet shows the period and how to read one instant.
skill_freshness_for_feature_list(question: str, feature_names: list, entity_name: Optional[str] = None) -> dict
Aggregate freshness across a list of feature names referenced by a group pronoun.
Used when the user says "are these features up to date?" without naming any of them explicitly. Looks each feature up in the catalog, collects its process_id, pulls follow_up_report for each, and returns a combined last-successful-run / days-since / failure verdict.
Where a process stands against its TimeManager (0.3.1.240).
{"manager": "db.view", "business_date": "YYYY-MM-DD", "behind": n} - behind
is the number of the manager's dates after the last one the process completed,
up to the current one (0 = up to date). None when the process has no
recorded TimeManager or it cannot be read. Two small reads, memoized per turn.
{PROCESS_ID: bare view name} from the turn's process catalog (never raises).
Deterministic freshness lines, one per process, in business dates.
Returns (lines, facts); facts maps PROCESS_ID to
{"last_business_date", "last_run_at", "last_failure_at", "time_manager"}.
Per-segment freshness when a process runs segmented (FilterManager).
Groups the follow-up rows by their APPLIED_FILTER segment and returns one
freshness record per distinct segment. Business dates (VALIDTIME_DATE) are
not segments: they are reported by _business_date_lines (0.3.1.240).
Returns:
[{segment, last_successful_run, days_since_update, has_recent_failure}],
sorted stalest-first. Returns [] when there is a single segment / no
FilterManager (caller keeps the existing global aggregation unchanged).
skill_freshness(question: str, object_name: str, hint_entity_name: Optional[str] = None, accumulated_context: Optional[dict] = None, _fuzzy_corrected: bool = False) -> dict
Tell the consumer whether a feature or dataset is up to date and reliable.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question about freshness / reliability. |
required |
object_name
|
str
|
Name of the feature or dataset. |
required |
hint_entity_name
|
Optional[str]
|
Entity name remembered from a previous turn — used to auto-disambiguate when the same feature name exists across entities. |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
_summary_cache_hit(acc: Optional[dict], domain: Optional[str], entity_filter: Optional[str], object_name: Optional[str]) -> Optional[dict]
Return the prior turn's cached summary payload when it covers THIS request.
A hit requires: no explicit object named this turn (a named dataset/entity must
re-resolve), and the prior _last_summary was taken for the same
(domain, entity_filter) scope. Returns the cached payload dict or None.
tdfs4ds.AGENT_LIST_MAX_ITEMS as a non-negative int (0 = no cap).
First sentence of text, cut on a word boundary at limit characters.
{PROCESS_ID: {"logic": ..., "entity": ...}} from the stored process
documentation (business logic + entity description), one batched read,
memoized per turn. Fail-open: {}.
Does the question restrict the list to a subject? If so, which features of
frame match it? {"topic", "selected": [names], "dropped": [names]}.
The LLM sees every feature of the scope (name, entity, description) in
batches of _TOPIC_BATCH; the first batch decides whether there is a
subject at all (one call when there is none). Names it returns that are not
in the frame are dropped and reported, never shown. Fail-open: no subject.
French 'de' + label with the contraction: de le -> du, de les -> des.
[{view, process_id, entity, names}] largest first (ties by view name).
Features with no producer are not a process: they form the group whose
view is None (rendered "without a producer", never counted).
The Report's counts (store_consistency.catalog_counts) for a scope, from
the turn snapshot. process_ids restricts the processes (a dataset scope).
None when the catalogs cannot be read: the caller counts from its frame.
_large_listing_answer(question: str, scope_label: str, frame: DataFrame, *, full_list: bool = False, documented_themes: str = '', scope_is_process: bool = False, counts: Optional[dict] = None, group_scope: Optional[tuple] = None) -> dict
The feature inventory of a scope, rendered by code (0.3.1.240).
Same store, same question: the same answer. The LLM never writes the list nor a
number; it only (a) decides whether the question narrows the list to a subject
(_select_topic_features, names checked) and (b) proposes semantic groups,
once per feature set, stored and reused (feature_groups).
- up to
AGENT_LIST_MAX_ITEMSfeatures (orfull_list): a table per entity, business meaning (first sentence of the documented description) and technical name; - above: semantic groups (label, one sentence, technical names) for the scope
group_scope = (domain, entity); without them, the producing processes.
counts are catalog_counts for the scope (the Report's numbers); without
them the processes are counted from frame.
Returns {"answer", "topic", "topic_dropped", "groups", "selected_frame"}.
Merge a _large_listing_answer into a skill_summary result: the answer,
and - after a subject narrowed the list - the selected features as the turn's
feature list (so a follow-up 'are these up to date?' is about them).
(entity as stored, [data domains having features for it]). Fail-open: (entity, []).
The one entity of the store the question names (_best_entity_match), or None.
_summary_entity_across_domains(question: str, entity: str, accumulated_context: dict = None) -> Optional[dict]
The feature list of entity in each data domain that has it, or None.
One domain: that domain's entity summary. Several: one section per domain (largest first), since the same entity in two domains is two sets of features. None when no domain has the entity (the caller keeps its path).
skill_summary(question: str, domain: str = None, object_name: str = None, entity_filter: str = None, accumulated_context: dict = None, _entity_fanned: bool = False) -> dict
Give a high-level summary of available features for a domain or a specific dataset.
When object_name resolves to a registered dataset, the response is scoped to that dataset's feature list and the resolved dataset context is returned in the result so the caller can persist it in agent state for follow-up EXPLAIN turns.
When entity_filter is given (or object_name resolves to an entity type), the feature list is scoped to that single entity — answering "what features belong to X?".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
domain
|
str
|
Therapeutic domain to filter by (optional). |
None
|
object_name
|
str
|
Dataset, entity, or feature name mentioned by the user (optional). |
None
|
entity_filter
|
str
|
Entity name to scope the feature list (optional). |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
domain (default: the session's DATA_DOMAIN) holds features of entity (its exact key set).
A dataset's features as the listing frame. _get_dataset_with_features
has no PROCESS_ID: it is taken from the domain catalog by (entity, name).
Every feature a process computes, listed deterministically (a process is the finest group, so the whole list is shown even above the cap). A subject in the question narrows it, as for any scope.
Explain the feature store's own conceptual model (direct-answer, no DB query).
For a confused user asking "what is a feature vs a non-feature?", "what is a data domain?", "how is the feature store organized?", "what can I ask you?". Returns a fixed, grounded explanation (EN/FR) anchored on the certainty distinction between a feature (rich, governed column with a known entity key) and a non-feature column (free; relationships only documentation-inferable, never certain).
Recap the features, entities, and datasets from the current conversation.
Reads directly from the accumulated_context_graph (MemorySaver-persisted state) instead of querying Teradata. Returns a structured dict grouping features by domain and entity for synthesis into markdown.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
accumulated_context
|
dict
|
The accumulated_context_graph dict from agent state. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
Return the process VIEW_NAME registered for object_name (feature or view).
Resolution order: 1. _get_feature_catalog_with_docs chain (join feature → process catalog) → VIEW_NAME 2. process_catalog (all domains) → FEATURE_NAMES comma list match → VIEW_NAME 3. process_catalog (all domains) → exact VIEW_NAME match
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_name
|
str
|
Feature name or view name. |
required |
Returns:
| Type | Description |
|---|---|
|
|
|
|
|
Extract distinct column/alias identifiers from a SQL expression.
Strips literals, SQL keywords, function names and numeric tokens to return only identifiers that look like column or alias references. Used to cross-check the LLM's source_columns list for completeness.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
formula_sql
|
str
|
SQL expression string (e.g. the body of a CASE WHEN). |
required |
Returns:
| Type | Description |
|---|---|
list
|
Sorted list of distinct identifier strings (case-preserved as in formula). |
Extract all base table/view identifiers referenced from FROM/JOIN clauses in ddl.
Returns a set of bare (unquoted, uppercase) identifiers — used as a cheap sanity check that the LLM's source_table claims correspond to a real relation the view actually reads from.
Map {alias_lower: REAL_TABLE_UPPER} for FROM/JOIN <table> <alias> clauses.
ONLY records an alias when the relation after FROM/JOIN is a real (schema-qualified
or bare) table — NOT a subquery. A subquery wrapper (FROM ( … ) alias) has a
( after FROM/JOIN, so it is skipped and its alias (e.g. ps, phs, and
the outer base_data/flags wrappers) is deliberately NOT mapped to any table.
Used to resolve an LLM-claimed source_table that is actually a table alias back
to the real underlying table, and to detect subquery aliases (absent here) so a
bogus "(from
Resolve an LLM-claimed source_table to the real base table, or signal "drop".
Returns:
| Type | Description |
|---|---|
Optional[str]
|
|
Optional[str]
|
|
Optional[str]
|
|
Optional[str]
|
|
Extract identifiers that look like real columns referenced in ddl.
Keeps the column in alias.col (and drops the bare alias), then returns
the set of remaining whole-word identifiers (uppercased). Used to decide
whether a formula identifier missing from the LLM's source_columns is a
genuine base/output column (worth surfacing) or a subquery-derived temporary
/ alias that only exists in an intermediate scope (drop it).
_validate_source_columns(source_columns: list, formula_sql: Optional[str], ddl: Optional[str]) -> tuple[list, list]
Validate LLM-produced source_columns against the formula and DDL.
Returns (augmented, missing_identifiers) where:
* augmented = source_columns annotated with source_confidence ('high'
when the source_table appears in the DDL's FROM/JOIN set, 'low' otherwise)
* missing_identifiers = formula identifiers that are NOT present in
source_columns AND plausibly are real columns (so the caller can surface
them as UNKNOWN). Single/double-character tokens (SQL aliases like p,
s) and subquery-derived temporaries that never appear in the DDL body
(e.g. mn/mx computed inside an inner SELECT) are dropped so they are
not mis-reported as drillable source variables.
Split a SELECT list into items at top-level commas (paren-depth aware).
Nested CASE/window/function expressions contain commas; only commas at paren
depth 0 separate select items. Used to build the <expr> AS <alias> map for
the formula-column closure.
Identifiers from the columns that select WHICH ROWS / ORDERING a derived
feature is computed over — i.e. window PARTITION BY / ORDER BY keys and
GROUP BY keys.
These are the inputs that are NOT visible from the feature's arithmetic yet
materially change its value (e.g. a duplicate-detection flag defined via
ROW_NUMBER() OVER (PARTITION BY patient_id, drug_code ORDER BY claim_date) —
the grain keys decide what counts as a duplicate). It deliberately does NOT pull
CASE/WHERE threshold operands: those re-reference the same arithmetic as the
derived feature itself (already listed as a variable), so surfacing them would
re-introduce the over-listing this prune exists to remove.
Map {alias_lower: defining_expression} for every <expr> AS <alias> in ddl.
Scans all SELECT segments (paren-aware) so derived columns at any nesting level are captured. Used by the formula-column closure to walk from a derived feature to the governing columns of its definition.
Transitive closure of the columns that materially feed calculation_sql.
Starts from the identifiers in the formula, then for each identifier that is a
DERIVED column (<expr> AS <alias> in the DDL) pulls in the identifiers of its
GOVERNING clauses (window/CASE/WHERE/GROUP BY/HAVING) — iterating to a fixpoint.
This keeps columns that legitimately drive the formula through clauses (e.g.
patient_id, drug_code, claim_date behind a ROW_NUMBER() OVER (PARTITION BY
patient_id, drug_code ORDER BY claim_date) flag) while NOT pulling in the plain
arithmetic operands of an already-listed derived feature.
Returns a set of lowercased column names, or None (= "do not prune") when
either input is falsy — fail-open so a missing formula/DDL never hides variables.
Keep only source_columns whose name is in closure (None ⇒ keep all).
Fail-open: when closure is None (no formula/DDL) the list is returned
unchanged, so pruning never removes a legitimately-listed variable.
Return {token: (best_score, best_name)} — the closest catalog name and
its similarity score for each token.
Size-adaptive backend (the candidate set drives the choice, per design):
- small set (len(known_names) <= tdfs4ds.FUZZY_INDB_THRESHOLD): Python
difflib.SequenceMatcher ratio — no DB round-trip, ideal for the small
per-turn-cached catalogs the consumer agent uses (e.g. 52 names);
- large set: Teradata in-database StringSimilarity (jaro_winkler), scored
set-based in one query — scales to thousands of names.
known_names is an iterable of catalog names (any case). Matching is
case-insensitive. Fail-open: on any error returns {} (caller treats as no
fuzzy match, i.e. exact-only).
Python-difflib backend for _best_name_similarity (small candidate sets).
Teradata StringSimilarity (jaro_winkler) backend for large candidate sets.
Builds an inline (token, name) cross-product as the table-operator input
and keeps the max jaro_winkler score per token. Falls back to difflib on any
SQL error (fail-open via the caller's try/except).
Return the distinct ENTITY_NAME values across all data domains (cache-backed).
Reads the per-turn-cached feature catalog (_all_features_cached) and returns
the unique non-empty ENTITY_NAME values. Fail-open: empty list on any error.
Return the canonical catalog spelling of name, correcting a single typo.
Conservative, single-high-confidence typo correction used at the classify seam so a misspelled name never propagates into the context graph (mints a duplicate node). The name pool depends on kind:
"domain"→ registered DATA_DOMAIN names (get_data_domain())"entity"→ distinct ENTITY_NAME values (_entity_name_pool())"object"/ anything else → the union of feature / dataset / process-view names (_known_catalog_names()) PLUS domain names (anobject_namecan itself be a typo'd domain).
Resolution order: blank → returned as-is; exact case-insensitive hit → the
canonical-cased pool name; else a single fuzzy match with score
>= tdfs4ds.FUZZY_NAME_MATCH_CUTOFF (default 0.90) → the matched name;
otherwise name is returned unchanged so the skills' did_you_mean
clarification still owns genuine multi-candidate ambiguity.
Fail-open: never raises; any error returns name unchanged. Memoized per turn.
Return catalog feature names that are close (fuzzy) matches for object_name.
Uses :func:difflib.get_close_matches against the full feature catalog
(all data domains). Only names with a similarity score >= 0.75 are returned,
which catches single-character differences (missing/extra letter, transposition)
without being too permissive.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_name
|
str
|
The name as typed by the user (may contain a typo). |
required |
Returns:
| Type | Description |
|---|---|
list
|
List of up to 3 feature names from the catalog in their original case, |
list
|
ordered by similarity. Empty list when no close match is found. |
Return dataset names that are close (fuzzy) matches for object_name.
Mirrors :func:_fuzzy_match_feature against the dataset catalog so a
near-miss like 'ds_investigation_que' resolves instead of silently failing.
Returns:
| Type | Description |
|---|---|
list
|
List of up to 3 dataset names in their original case, ordered by |
list
|
similarity. Empty list when no close match is found. |
The words of a dataset name or phrase, lower case, filler and ds_ prefix removed.
The one dataset whose name is phrase's words, in order ([] otherwise).
"customer RFM dataset" -> ds_customer_rfm. Deterministic, and only a
unique match counts: two datasets with the same words is a question to ask,
not a guess to make.
The words of phrase that carry its meaning: dataset and feature filler removed.
{"type": "dataset", "dataset_name", "matched_by": "name"} when name is
a registered dataset's own name (any case, with or without its database).
The classifier sometimes returns the dataset's name itself for "which dataset
gives me the RFM features?" (DS_CUSTOMER_RFM, seen live); without this the
dataset skill looked it up as a feature and answered that no feature has it.
The one dataset whose name holds every word of phrase ("RFM features" ->
ds_customer_rfm), [] otherwise. Only a unique match counts.
The one dataset whose own documentation (description, sections) holds every
word of phrase as a whole word, [] otherwise. Fail-open.
What a phrase that names no catalog object refers to.
In order: the one dataset whose name holds its words, the one dataset whose
documentation does, then the features whose own description matches it
(:func:_features_matching_meaning). type is "dataset" (with
dataset_name), "concept" (with features) or "unknown";
matched_by says how. Reads the turn's cached catalogs; no LLM call.
Registered features whose documentation matches phrase: [(name, description)].
Only for a name that matched nothing ("recency" for DAYS_SINCE_LAST_TXN). A
feature is offered only when its own description contains the stem of a word
of the phrase; the vector index (columns) only orders them, the features it
did not return come after. At most limit. Fail-open: [] when the
catalog cannot be read.
'No feature is named X; these documented features match its meaning', EN/FR.
'
Return all dataset names registered in the dataset catalog (original case).
Cheap token-based French detection for deterministic (non-LLM) replies.
Only used where the answer is templated rather than LLM-generated — the LLM paths handle language matching natively via the system prompt.
Deterministically resolve an ambiguous feature's entity from the question text.
When a feature exists under several entities the user may have already named
the one they mean ("total_amount for pharmacy_id") — asking a clarification in
that case is a dead turn. Scans the question for the candidate entity names
(case-insensitive, _/space-normalised; multi-key entities like
claim_id,region match on any key component) and returns the candidate when
exactly ONE matches. Zero or multiple matches return None so the normal
clarification flow runs.
The one entity text names, or None.
Among several matches, the entity all of whose keys are named and that has
the most keys wins ("for Category and CustomerID" -> Category,CustomerID;
"for CustomerID" -> CustomerID, not Category,CustomerID). A tie is None.
The ways a key column is written in a question: CustomerID is also
"customer id", "customer_id" and "customerid" (camel case split, _ as space).
Every word an entity is written with (each key column, camel case split), plural forms too.
The entities of entities the text names, in their order, without repeats.
An entity matches when one of its key columns appears as whole words, in any
of its spellings (:func:_entity_spellings); a key spelt compact
("customerid") matches a word of the text. A multi-key entity matches on any
of its keys, so "CustomerID" names both CustomerID and
Category,CustomerID: callers prefer the exact entity (:func:_best_entity_match).
Fetch the DDL of a view via SHOW VIEW.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
view_name
|
str
|
Bare view name (no schema prefix needed — uses |
required |
Returns:
| Type | Description |
|---|---|
Optional[str]
|
DDL text, or |
Return the composite cache key used in accumulated_context_graph.features.
A feature's identity is the TRIPLET (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME).
Because the active domain can change mid-conversation, the domain is part of
the key so a cached feature from domain A is never served for domain B.
Format: DOMAIN::ENTITY::FEATURE_NAME (all uppercased). When entity/domain
are unknown/blank those segments are empty so the key never collides with a
concrete entry and stays findable by suffix scan. data_domain defaults to
the active tdfs4ds.DATA_DOMAIN.
Cross-store federation limit (v0.3.1.148): the triplet has NO store segment, so two read-only stores sharing a (domain, entity, feature) triplet would map to the same key (last-write-wins in accumulated_context_graph.features). This is a no-op for stores with disjoint domains/names (the common case); adding a store segment here is the documented follow-up for collision-proof multi-store memory.
_feat_cache_lookup(features_dict: dict, feature_name: str, entity_name: str = '', data_domain: Optional[str] = None) -> Optional[dict]
Look up a feature entry in accumulated_context_graph.features.
Triplet-aware (DOMAIN::ENTITY::NAME) with graceful fallback to older
snapshots that used NAME::ENTITY or a plain NAME key. Tries in order:
1. Exact triplet key (when domain+entity known) → exact.
2. Active-domain + entity, or active-domain + any entity (suffix scan).
3. Any key ending in ::NAME (legacy DOMAIN::ENTITY::NAME or new) —
first match (ambiguous; callers needing precision pass entity+domain).
4. Legacy NAME::ENTITY / NAME:: prefix and plain NAME keys.
_query_accumulated_context_for_lineage(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract lineage graph from accumulated_context without DDL scans.
Returns the edges and nodes found in the graph for this feature, or None if the feature is not cached (cache miss). On cache hit, includes depth estimate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Optional[dict]
|
Dict with keys: {"edges": [...], "nodes": {...}, "depth": int} or None (cache miss). |
|
Edges |
Optional[dict]
|
[{"source": "...", "target": "...", "relation": "depends_on"}] |
Nodes |
Optional[dict]
|
{"name": {"type": "feature|view|table|dataset", ...}} |
_query_accumulated_context_for_usage(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract dataset memberships from accumulated_context without catalog scans.
Returns the datasets that expose this feature, or None if the feature is not cached or has no dataset membership (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Type | Description |
|---|---|
Optional[dict]
|
Dict with key: {"datasets": [{"name": str, "database": str}]} or None (cache miss). |
_query_accumulated_context_for_source_tables(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract direct source tables from accumulated_context without DDL scans.
Returns the immediate predecessors (tables/views) that feed into the feature's process view, or None if the feature is not cached (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Type | Description |
|---|---|
Optional[dict]
|
Dict with keys: {"source_tables": [str], "view_name": str} or None (cache miss). |
_query_accumulated_context_for_freshness(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract freshness timeline from accumulated_context without catalog queries.
Returns the freshness metrics (last_successful_run, days_since_update, status) for the feature's process, or None if the feature is not cached (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Type | Description |
|---|---|
Optional[dict]
|
Dict with keys: {"last_successful_run": str, "days_since_update": int, |
Optional[dict]
|
"has_recent_failure": bool, "last_status": str, "_from_cache": True} |
Optional[dict]
|
or None (cache miss). |
_query_accumulated_context_for_calculation(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract calculation formula and source columns from accumulated_context.
Returns the calculation SQL and source columns for a feature, or None if the feature is not cached or has no calculation data (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Type | Description |
|---|---|
Optional[dict]
|
Dict with keys: {"calculation_sql": str, "source_columns": list, |
Optional[dict]
|
"explanation": str, "_from_cache": True} or None (cache miss). |
_query_accumulated_context_for_explanation(feature_name: str, entity_name: str, accumulated_context: Optional[dict]) -> Optional[dict]
Extract explanation and documentation sources from accumulated_context.
Returns the explanation text and documentation sources for a feature, or None if the feature is not cached (cache miss).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_name
|
str
|
Feature name (will be uppercased for lookup). |
required |
entity_name
|
str
|
Entity name for disambiguation. |
required |
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state. |
required |
Returns:
| Type | Description |
|---|---|
Optional[dict]
|
Dict with keys: {"explanation": str, "docs_sources": list, |
Optional[dict]
|
"_from_cache": True} or None (cache miss). |
_resolve_owning_object_for_column(column_name: str, hint_table_name: Optional[str] = None) -> Optional[dict]
Resolve a non-feature column to its owning object via FS_BUSINESS_DICTIONARY_COLUMNS.
Returns {"column": COL, "object": TABLE_NAME, "database": DB, "description": ...}
or None when the name is not a documented column. When hint_table_name is
supplied and the column appears under several objects, that object is preferred.
Infer the asked lineage direction: 'downstream' | 'upstream' | 'both'.
The persisted lineage graph at the turn's valid time, read once per turn.
At a past date (_AS_OF) the graph is read VALIDTIME AS OF from the
primary store, with no on-the-fly fallback: today's DDL would describe today.
'time manager' / 'filter manager' / 'manager table' (tagged at load), else view / table / dataset from the stored OBJECT_TYPE.
_position_object_from_graph(question: str, object_name: str, *, column_note: Optional[str] = None, correction_note: Optional[str] = None, direction: Optional[str] = None) -> Optional[dict]
Position ANY documented object in the persisted lineage graph (FS_LINEAGE_GRAPH).
Used as the fallback when object_name is not a feature/dataset/process view but
is a documented table/plain-view/dataset node — so the user can ask "where does the
RAW_CLAIMS table sit / what depends on it?" for non-process objects too.
Returns a lineage result dict (same shape as skill_lineage's slow path), or
None when the object is not a node in the graph (caller then falls through to the
normal not-found path). When the node exists but has no edges, returns a
lineage_empty result (no confabulation), with the object's BD description if any.
skill_lineage(question: str, object_name: str, hint_view_name: Optional[str] = None, accumulated_context: Optional[dict] = None, entity_name: Optional[str] = None, hint_table_name: Optional[str] = None) -> dict
Return the upstream data dependencies of a feature, object, or column.
Walks the dependency graph starting from the registered process view, and returns a structured list of upstream nodes (tables and views) with their types. Does not require LLM — the graph is deterministic.
When object_name is not a feature/dataset/process view (e.g. a plain
documented table, view, or a non-feature column), it falls back to the
persisted lineage graph (FS_LINEAGE_GRAPH) to position ANY documented
object. A column is first resolved to its owning object (TABLE_NAME)
and that object is positioned instead — see _position_object_from_graph.
Implements fast-path caching: queries accumulated_context_graph first (cache hit → return immediately), falls back to expensive DDL scans (cache miss → rebuild and merge results for next turn).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question (used to determine response language). |
required |
object_name
|
str
|
Feature, object (table/view/dataset), or column name. |
required |
hint_view_name
|
Optional[str]
|
Pre-resolved view name from a prior turn (skips catalog lookup). |
None
|
accumulated_context
|
Optional[dict]
|
The accumulated_context_graph from state (used for cache). |
None
|
entity_name
|
Optional[str]
|
Entity name for disambiguation (used in cache lookups). |
None
|
hint_table_name
|
Optional[str]
|
Owning object for a column (from a prior EXPLAIN/DEFINITION
turn via |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
dict
|
|
Explain how multiple features are calculated in a single LLM call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
features_list
|
list
|
List of |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
Authoritative one-line clarification(s) for the active interpretation hints.
Deterministic flag→sentence map, used as the EXPLAIN backstop when the LLM
explanation contradicts a hint (see :func:_find_hint_violations). Generalises
DEFINITION's inverted-only _direction_correction_note to every flag.
Resolve business descriptions for a formula's input columns (bounded).
Grounds the EXPLAIN LLM in what the inputs mean (so it can derive the output's direction/meaning instead of guessing from the top-level arithmetic). Deterministic, capped, and fail-open: any lookup failure for a column is skipped silently.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
identifiers
|
list
|
candidate real column names (already alias/temporary-filtered). |
required |
max_cols
|
int
|
cap on the number of columns resolved (prompt-size / round-trip bound). |
6
|
max_chars
|
int
|
per-description truncation. |
300
|
Returns:
| Type | Description |
|---|---|
str
|
A |
True when the view DDL references a Script Table Operator / tdstone2 STO view.
Extract the tdstone2 scoring-mapper UUID from a process view's DDL.
The process view of an ML feature selects FROM TDS_STO_SCORING_LIVE_<uuid>
(or TDS_STO_SCORING_<uuid>); the <uuid> is the tdstone2
ID_MAPPER_SCORING. Teradata table names use _ in place of the UUID's
-, so we normalise back to hyphenated canonical form.
Returns None when no STO scoring reference is present (i.e. not an ML feature).
Return the distinct top-level Python packages imported by code_text.
Scans import X / from X import … lines, keeps the top-level package,
drops standard-library modules, dedupes preserving first-seen order, and
surfaces sklearn first when present (it is the headline ML library).
True when value is the open-end sentinel (year >= 9999) or unparseable-as-real.
tdstone2's trained-models table can carry an open-period sentinel
(9999-01-01) in TD_TIMECODE instead of a real training time. Treat such
a value as "training date not recorded". A None/blank value is also treated as
a sentinel (no usable date).
Build the capped DISTINCT trained-model→code resolution query.
From the trained-models table, take DISTINCT ID_TRAINED_MODEL and join
TDS_MODEL_REPOSITORY to resolve ID_CODE + ARGUMENTS in one round
trip, capped at cap rows. In practice all partitions share one
ID_MODEL/ID_CODE so the result collapses to a single row.
Read the Python source of a tdstone2 code repository entry.
The CODE column is a BLOB; CAST(CODE AS VARCHAR) errors ("Conversion
between BYTE data and other types is illegal"), so it must be read via
FROM_BYTES(CODE,'ascii'), chunked with SUBSTR for long sources. Returns
the decoded text truncated to max_chars (a marker is appended on truncation).
Summarise a tdstone2 trained-models table: partitions, status, freshness.
Returns {n_partitions, n_successful, model_type, training_time_available}.
training_time_available is False when the max TD_TIMECODE is the 9999
open-end sentinel (or unreadable). Fail-open: returns zeros on query failure.
True when the question is about the code / packages / algorithm internals.
Explain the machine-learning model behind a feature.
Walks the tdfs4ds → tdstone2 chain: resolve the feature's process view, detect the Script-Table-Operator scoring view in its DDL, parse the scoring-mapper UUID, resolve the trained model → model arguments → Python code, and report the algorithm, packages, hyperparameters, model inputs and training-partition status in business language.
Distinct from :func:skill_explain (which explains the SQL formula): this
skill explains the ML model (algorithm / code / packages / hyperparameters).
Returns a dict carrying answer (so the synthesiser returns it directly)
plus structured fields (algorithm, packages, hyperparameters,
model_inputs, n_trained_partitions, training_time_available …) and
error on the failure paths.
Parse a tdstone2 Script-Table-Operator view DDL into its run-time invocation.
Extracts the reproduction-relevant pieces of the FROM Script(...) clause:
partition_by / order_by columns, the script_command string, the
returns output signature (list), and charset. Regex-based; fail-open —
returns {} (or partial) when a piece is absent.
_algorithm_system_prompt(facts: dict, model_code: Optional[str], wants_code: bool, input_meanings: str = '') -> str
Assemble the ALGORITHM skill's LLM system prompt (DB-free testable seam).
Injects the deterministic facts (algorithm/packages/hyperparameters/STO config/ structure/invocation), optional input-column business meanings, and the model source (only when the user asked for code). Frames the answer as a tdstone2 reproduction recipe without inventing API the package doesn't have.
The raw STO training view DDL (_raw_ddl inside sto_invocation) is
extracted and injected as a separate authoritative block so the LLM can quote
it verbatim instead of reconstructing SQL syntax.
Best-effort algorithm name from hyperparameter key prefixes or code text.
Parse a JSON value that may arrive as a dict, str, or fenced str.
tdstone2's ARGUMENTS/METADATA columns are JSON; depending on the driver they come back as a dict already or as a (possibly markdown-fenced) string.
skill_explain(question: str, object_name: str, hint_view_name: Optional[str] = None, hint_dataset_features: Optional[list] = None) -> dict
Explain how a feature is calculated by reading its process view DDL.
Fetches the SQL definition of the registered process view and asks the LLM to translate the logic into plain business language — no SQL jargon.
When hint_dataset_features is provided and object_name is empty, explains all features from the current dataset context (up to 10) in a single LLM call.
When object_name contains comma-separated feature names, explains each one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
object_name
|
str
|
Feature or view name. May be comma-separated for multiple features. |
required |
hint_view_name
|
Optional[str]
|
Pre-resolved view name from a prior turn (skips catalog lookup). |
None
|
hint_dataset_features
|
Optional[list]
|
Feature list from a prior SUMMARY turn on a dataset. |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
[(domain, entity, has_process)] of every catalog feature called name
at the turn's valid time; has_process: a current process lists it.
Fixed answer: the feature exists, here is where (and whether a process produced it at that date). No LLM.
sql_context.computation_context with the store's manager objects. Fail-open.
_extract_feature_list_from_question(question: str, available_features: list, available_entities: Optional[list] = None) -> tuple
Ask the LLM to extract feature names AND entity-key concepts from a multi-feature request.
Used when the user asks something like "show top 50 highest-risk claims with risk band, top driver, provider and pharmacy" — the question references several feature concepts plus entity keys (provider, pharmacy) needed for the SELECT projection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question text. |
required |
available_features
|
list
|
List of registered feature names in the active domain. |
required |
available_entities
|
Optional[list]
|
List of entity-key tokens (e.g. provider_id, pharmacy_id) parsed from the feature catalog. Optional. |
None
|
Returns:
| Type | Description |
|---|---|
tuple
|
|
tuple
|
Returns |
Best-effort: True/False from a SQL type string, None when unknown.
_choose_ranking_metric(feature_cols: list, feature_meta: dict, inverted_features: 'set | None' = None) -> dict
Pick a sortable ranking column for a dashboard top-N query.
Never orders by a categorical band with a bare DESC. Prefers a numeric risk/score column; falls back to a deterministic CASE over a band label.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_cols
|
list
|
requested feature/column names. |
required |
feature_meta
|
dict
|
|
required |
inverted_features
|
'set | None'
|
names (any case) whose formula inverts the score (lower = higher risk) → ranked ASC. |
None
|
Returns {"metric", "direction", "order_expr", "note"} (order_expr may be a
CASE expression; metric/order_expr are None when no sortable column exists).
_cover_features_with_datasets(feature_names: list, per_feature: list, dataset_entity_keys: dict, entity_targets: 'list | None' = None) -> dict
Choose the minimal set of datasets covering the requested features (and the requested entity columns) and the join keys to combine them — deterministic, no LLM, no fabrication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
feature_names
|
list
|
requested feature names. |
required |
per_feature
|
list
|
|
required |
dataset_entity_keys
|
dict
|
|
required |
entity_targets
|
'list | None'
|
entity columns the user asked to project (e.g.
|
None
|
Returns {"datasets": [fq, ...], "join_keys": [col, ...], "uncovered": [...]}.
uncovered lists requested features AND entity targets that no registered
dataset exposes.
Pick the dataset(s) for a dashboard query from the authoritative dataset model — deterministic, no LLM, no fabrication.
A dataset projects exactly its ENTITY key columns + its registered FEATURE
columns. The primary dataset projects the most requested columns; a second
dataset is JOINed only when it shares ≥1 entity key AND adds projectable
columns. Columns no chosen dataset can project are returned as unavailable.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_model
|
dict
|
|
required |
feature_cols
|
list
|
requested feature names. |
required |
entity_keys
|
list
|
requested entity-id columns (provider_id, …). |
required |
Returns {"datasets", "join_keys", "proj_entities", "proj_features",
"unavailable"}.
Build the authoritative dataset model from the three dataset-catalog views (CATALOG / ENTITY / FEATURES), per the tdfs4ds data model.
Returns {fq_dataset_name: {"dataset_id", "entity_keys": [...],
"features": {FEATURE_NAME_UPPER: {"name", "feature_type", "is_numeric"}}}}.
This is the single source of truth for the dashboard composition: what each
dataset actually exposes (its ENTITY key columns + its registered FEATURE
columns, with types) — not reconstructed from the global feature catalog.
Best-effort: returns {} on failure.
Return {fq_dataset_name: [entity_col, ...]} for all registered datasets.
Entity columns come from get_dataset_entity (parsing the ENTITY_ID /
column form). Used to derive join keys for multi-dataset dashboard queries.
Best-effort: returns {} on failure.
Return the subset of feature_names whose process-view formula inverts the
score (1 - …, so lower = higher risk). Best-effort; bounded to the small
requested set. Failures are swallowed (feature simply treated as non-inverted).
Multi-feature dataset composition for dashboard-style requests.
Decomposes the question into a feature list, then re-uses
:func:skill_dataset multi-feature logic to find datasets exposing all
of them (or report a JOIN gap when none does).
Returns the same shape as multi-feature skill_dataset plus the
extracted feature_names so the synthesiser can confirm what was matched.
Find datasets that expose one or more features.
When object_name contains comma-separated feature names, returns per-feature dataset lists plus an intersection of datasets that contain all features.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
object_name
|
str
|
Feature name(s) to look up, comma-separated for multi-feature. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Single-feature: |
dict
|
Multi-feature: |
dict
|
Status values: |
dict
|
|
The datasets a phrase that names no feature refers to (status='concept_match').
recognized comes from :func:_recognize_concept: a dataset (by its name or
its documentation) is answered with its features; features matched by their
description are answered with the datasets exposing them. The answer is
written by code and says how the phrase was matched.
Single-feature dataset lookup (original logic).
Feature identity is the TRIPLET, so when the dataset-features frame carries an
ENTITY_NAME and the feature name exists under multiple entities, constrain by
entity_name so "which datasets expose ambiguous_feature when the entity is needed but unknown).
A FEATURE_ID as an int, whatever form it is read in (' 6', 6.0, Decimal); None if not one.
Every registered feature called feature_name: [{domain, entity, feature_id, database}].
A feature is the triplet (data domain, entity, name): the same name can exist under several entities and domains. Restricted to the active data domain when it holds the name, and to entity_name when given and matching.
Order-insensitive entity key (catalog and process lists differ in order).
{index of identity: [DATASET_ID]} of the datasets exposing each feature.
Matched on (FEATURE_DATABASE, FEATURE_ID), never on the name: an id is unique within one store only, and a dataset may read another store's tables (seen live: DB_LAB.DS_CUSTOMER_FROM_PRODUCTION reads DB_PROD ids 9-12, and DB_LAB's id 9 is another feature). Keys of the result are the identity dicts' positions.
Add each dataset's data time (time, time_tag) and list current
snapshots first: a user asking which dataset exposes a feature usually wants
today's values, and a dated or historical dataset must not be mistaken for it.
The BUILD_DATASET answers written by code (None = let the generic path answer).
skill_list_datasets(entity: Optional[str] = None, data_domain: Optional[str] = None, topic=None, purpose: Optional[str] = None) -> dict
Every registered dataset: database-qualified name, kind, features and their entity.
status: dataset_list, datasets: [{name, kind_tag, feature_count, entities}].
Features are identified by (FEATURE_DATABASE, FEATURE_ID); a dataset reading
another store's features shows no entity for them.
entity (and optionally data_domain) keeps the datasets built on exactly
that entity: the conversation's "these datasets" / "datasets for CustomerID".
scope echoes them and total counts every dataset.
topic (words of the question) keeps the datasets whose name, entities, data
domain or description share a word with it, and purpose 'train' the
datasets that carry a target (a LEAD offset column) or "train" in their name
(0.3.5.14: "which datasets are used to train the propensity models?" listed
every dataset of the store). Each narrows only when something matches: the
list is otherwise unchanged and scope says the filter matched nothing.
The datasets whose name, entities, data domains or description share a word with words.
The datasets that carry a target: a LEAD offset column (its catalog), or "train" in the name.
{DATASET_ID: tag} from one read of the build table ({} when it cannot be read).
The 'no dataset exposes this feature' answer, by code (EN/FR).
Build the 'no_dataset' result: the feature's identity and its sibling features.
Siblings are the features of the SAME data domain and entity (a triplet), never every feature of an entity name across domains (an answer once counted "8 features of Category" including a CustomerID feature).
skill_propose_features(use_case: str, entity_name: str, all_features: list, top_n: int = 20) -> dict
Propose the most relevant features for a given use case.
When all_features has more than top_n entries the LLM selects the
most relevant ones. When there are top_n or fewer, all are returned
directly without an LLM call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
use_case
|
str
|
Free-text description of the business need (user's message). |
required |
entity_name
|
str
|
Entity the features belong to (context for the LLM). |
required |
all_features
|
list
|
Full list of |
required |
top_n
|
int
|
Maximum features to propose. |
20
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
_rank_documented_objects(use_case: str, documented_objects: list, documented_columns: list, diagnostic: list) -> tuple
Pick the use-case-relevant documented datasets / objects / columns.
One bounded LLM pass over the documented (EFS-governed) objects + non-feature
columns. Returns (relevant_datasets, relevant_objects, relevant_columns)
where datasets are the kind=='dataset' subset. Fail-open: on any error,
returns small head slices so the feasibility answer still lists something.
Assess a use case WITHOUT creating a dataset (read-only feasibility check).
Scoped strictly to the data governed and documented by the feature store — the registered feature catalog (+ column business descriptions) and the business-dictionary documentation tables (objects + columns + sections). It does NOT inspect physical source tables / external schemas.
Two halves:
- Relevant existing assets — ranks the registered features
(:func:
skill_propose_features) AND the other documented objects: datasets (BD_OBJECTS type'D'), tables/views ('T'/'V'), and non-feature documented columns (BD_COLUMNS), via :func:_rank_documented_objects. Features are the finest grain but any documented object that helps the use case is surfaced. - Missing capabilities — given the use case, the documented feature coverage (feature names + descriptions, grouped by entity) and the domain business-dictionary themes, the LLM names up to 6 capabilities a complete solution would need that the feature store does NOT yet document. Each is framed against the existing coverage (e.g. "no temporal trend feature at the provider grain"). No dataset is built; nothing outside the EFS is read.
Returns a direct-answer dict (its own answer) — no second LLM pass.
Check whether a proposed dataset name is already taken in the catalog.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
proposed_name
|
str
|
Dataset name proposed by the user. |
required |
feature_name
|
str
|
Feature for which the dataset is to be created (context only). |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
dict
|
|
dict
|
|
Resolve a list of {name, description} feature dicts to the selected_features format.
Calls get_feature_versions() and splits results into clean (single-version) and conflict (multi-version) groups.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entity_name
|
str
|
Entity the features belong to. |
required |
feature_list
|
list
|
List of {name, description} dicts (as produced by skill_propose_features). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
tuple
|
Tuple of (selected_features_dict, conflicts_list). |
|
selected_features_dict |
tuple
|
{feature_name: uuid_str} ready for build_dataset(). |
conflicts_list |
tuple
|
[{feature_name, versions: [{process_id, process_view_name}]}]. |
Every date / timestamp written in text, normalised as TIMESTAMP literals.
ISO dates with an optional time and offset (2024-03-31, 2024-03-31 14:30,
2024-03-31T14:30:00Z, ... +02:00), and dates in words (31 March 2024,
31 mars 2024, end of June 2024). A value without an offset is UTC.
Raises ValueError on a date that cannot be read.
'90 days' / '1 years 2 months' from "last 90 days", "les 6 derniers mois"; None if none.
Read the kind of dataset the user asks for, and its parameters. Pure.
Returns {"kind": "current" | "as_of" | "history" | None, "as_of", "start",
"end", "duration", "full", "problem"}. problem names what is missing or
unreadable (unrecognised, as_of_needs_date, history_needs_window,
ambiguous_date, unreadable_date, unreadable_duration); the kind is
complete when it is None. Menu numbers 1 / 2 / 3 are accepted.
A TIMESTAMP literal for people: 2024-03-31 14:30:00 UTC / ... +02:00.
One line saying what will be built.
The dataset name in a reply ("my_ds", "call it my_ds", "my_ds"); None when unclear.
May a new dataset be created under the name in reply?
status: name_available, invalid_name (not a plain identifier),
name_taken (a registered dataset), object_exists (a table or view of
that name already exists in the view database: creating would replace it) or
name_check_failed (the database could not be read - never treated as free).
skill_build_dataset(entity_name: str, selected_features: dict, dataset_name: str, schema_name: Optional[str] = None, return_query: bool = False, kind: Optional[dict] = None, join_type: Optional[str] = None, document: Optional[bool] = None, feature_offsets: Optional[dict] = None) -> dict
Create, register and (when a model is configured) document a dataset.
kind comes from :func:parse_dataset_kind: current (default) builds a
snapshot valid now, as_of a snapshot at that instant, history one row
per entity and period of constant values (build_historical_dataset,
window start / end / duration / delay, or the full history).
time_manager (kind["time_manager"] = 'db.view') is a snapshot
that follows that TimeManager's business date. The view is
created in the view database; the database refuses it when the user may not
create it, and that refusal is returned whole (error).
join_type ('INNER' / 'FULL OUTER', None = the builders' default) and
document (None = when an instruct model is configured) are for callers
that let the user choose (the admin app's dataset builder), as is
feature_offsets (lags and leads, see build_dataset).
Returns status (dataset_created, creation_failed,
no_features_resolved or query_returned), dataset_name,
schema, feature_count, kind, dataset_id (when registered),
registered and doc_status (success, failed, skipped).
Render a small provenance footer for FRESHNESS and LINEAGE answers.
Returns an empty string when no sources dict is present or the intent
is not provenance-eligible. The footer cites the underlying catalogs /
methods so the user can verify (or escalate) the claim.
synthesize_answer(question: str, skill_result: dict, intent: str, resolved_feature_list: Optional[list] = None, accumulated_context: Optional[dict] = None) -> str
Convert structured skill output into a plain-language response.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
Original user question. |
required |
skill_result
|
dict
|
Dict returned by the skill function. |
required |
intent
|
str
|
The classified intent (SEARCH, DEFINITION, etc.). |
required |
resolved_feature_list
|
Optional[list]
|
Optional list of feature names from current/prior turns (for group context). |
None
|
accumulated_context
|
Optional[dict]
|
Optional accumulated context graph with cached metadata. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
A natural language answer suitable for a non-technical business user. |
Return (forbidden_set, dataset_list, feat_names) from the live feature store catalogs.
forbidden_set contains uppercase bare names and DATABASE.TABLE forms for: - all registered process views - all feature storage tables
dataset_list contains dataset names the MCP sub-agent is allowed to query. feat_names is a sorted list of all known feature names (reused by the async skill to avoid a second feature_catalog round-trip). Returns (set(), [], []) on any catalog error.
["DB.DS: f1, f2 (entity E, data domain D)", ...] - which dataset exposes which feature.
Matched on (FEATURE_DATABASE, FEATURE_ID), the identity of a feature inside a store, never on the name alone. A dataset reading another store's features lists them by the name it stores, without entity.
A tool failure as text for the model: the database message, without driver frames.
Rewrite SELECT/SEL to SELECT TOP 20 / SEL TOP 20 when no TOP N is present.
The arguments to retry with when the server did not find a qualified name, else None.
The agent names datasets database-qualified (the MCP session's default database
is not the store's). Some servers take the database apart and look the whole
table_name='DB.T' up as one object name (Enterprise base_tablePreview:
Error 3807 "Object 'DB.T' does not exist", seen live on AI Studio); others
build DB.T themselves and answer (Community). So the call is first sent as
the model wrote it, and only a 3807 naming that qualified value gives a retry
with the database in the tool's own database argument (db_name or
database_name). A database the model gave is kept unless it is empty or
the same one; anything else returns None.
Wrap MCP tools with feature-store query guards and a TOP 20 row cap.
Returns new StructuredTool objects — StructuredTool is a Pydantic model so monkey-patching invoke/ainvoke is not possible; we create proper wrappers.
Guards applied on every tool invocation: 1. Reject inputs that reference forbidden feature store objects. 2. Rewrite SELECT queries to SELECT TOP 20 when not already limited. 3. Append a notice when the result appears to hit the 20-row limit.
Recover a renderable chart from an MCP tool's output, if present.
teradata-mcp-server's plot_* tools return Chart.js JSON, which
langchain-mcp-adapters surfaces as the ToolMessage .artifact
({"structured_content": {...}}). Returns {"title", "html"} for the
chatbot to embed, or None when the output carries no chart.
async
¶
_skill_mcp_tools_async(question: str, server_url: str, forbidden_set: set, dataset_list: list, known_features: list = None, transport: str = 'streamable_http') -> dict
Delegate a question to external MCP server tools.
Enforces feature-store query guards: - process views and feature storage tables are forbidden - SELECT queries are automatically capped to TOP 20 - dataset catalog is offered as the approved data source
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
question
|
str
|
User question. |
required |
server_url
|
str
|
MCP server endpoint URL. |
required |
transport
|
str
|
MCP transport protocol — |
'streamable_http'
|
Returns:
| Type | Description |
|---|---|
dict
|
Dict with |
The question handed to the MCP tools, carrying the object routing resolved (0.3.5.18): "give me a sample" about the conversation's dataset reached the MCP agent without the dataset, which then sampled others (seen live). Unchanged when no object was resolved or the question already names it.
LangGraph node: delegate to MCP server tools.
Dashboard-shaped questions are short-circuited to the grounded
:func:skill_compose_dashboard path FIRST — even with MCP enabled — so the
answer is composed from verified datasets/columns instead of free-form MCP
SQL that can fabricate column names. Falls through to MCP when the question
does not decompose into a feature composition.