tdfs4ds.genai.documentation¶
module-attribute
¶
HTML_STYLES = {'container': 'font-family: Arial, sans-serif; margin: 10px 0;', 'title': 'color: #1f618d; margin-bottom: 6px;', 'heading': 'color: #2c3e50; border-bottom: 2px solid #3498db; padding-bottom: 5px;', 'heading_margin': 'color: #2c3e50; border-bottom: 2px solid #3498db; padding-bottom: 5px; margin-top: 15px;', 'content': 'background-color: #ecf0f1; padding: 10px; border-radius: 5px; line-height: 1.6;', 'list': 'background-color: #ecf0f1; padding: 15px 30px; border-radius: 5px; line-height: 1.8;'}
module-attribute
¶
module-attribute
¶
module-attribute
¶
_YOU_ITEM_FALSE_POSITIVE_CATEGORIES = frozenset({'UNCORRELATED_SCALAR_SUBQUERY', 'CROSS_JOIN_SINGLE_ROW', 'INFORMATIONAL'})
module-attribute
¶
DATASET_SECTION_NAMES = ['OVERVIEW', 'ENTITY', 'FEATURE_THEMES', 'BUSINESS_QUESTIONS', 'INTENDED_AUDIENCE']
Standard section names produced by _generate_dataset_description.
module-attribute
¶
_SECTION_DISPLAY_LABELS = {'OVERVIEW': 'Overview', 'ENTITY': 'Entity', 'FEATURE_THEMES': 'Feature Themes', 'BUSINESS_QUESTIONS': 'Business Questions', 'INTENDED_AUDIENCE': 'Intended Audience'}
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Return the prompt template body (after frontmatter) for skill name.
The returned string may contain {variable} placeholders intended for
use with 
meth:str.format (pure-Python prompts) or passed directly to
ChatPromptTemplate.from_template (LangChain prompts).
Parameters¶
name : str
Skill identifier, e.g. 'qo-candidates'.
Returns¶
str Prompt body with frontmatter stripped.
Return name as a Teradata identifier, quoting it only if required.
Idempotent: an already-quoted name is returned unchanged rather than being
wrapped a second time. Embedded quotes are escaped as "".
Parameters¶
name : str
The raw object/database name.
force : bool, default False
Quote even when the name does not need it. Use only at the few sites
that have always emitted unconditional quotes and whose exact output is
already baked into persisted DDL (see
feature_store/feature_query_retrieval.py) — elsewhere forcing would
change the SQL of existing stores for no reason.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
True when a failed CREATE means the object was already there.
The single authority for that question. It is deliberately NOT "this is an error we tolerate" — for an idempotent CREATE, an object that already exists is the nominal outcome, and reporting it as an error is what makes logs unusable for supervision.
'FEATURE_GROUPS', 'DEEP_LAYOUT' for a SECTION_NAME NOT IN (...) predicate.
Return the Teradata database user for volatile-table scoping.
The logon name is used here as a DATABASE QUALIFIER (<user>.<volatile>),
i.e. an identifier slot, so it must be double-quoted whenever it is not a
bare-legal Teradata identifier. LDAP/AD logins routinely are not: a dotted
name ('DEINF.CARLOSN') unquoted is an invalid three-part name (Error 3707),
stripping it to the last segment points at the wrong database (Error 3807),
and a name with an '@' ('SVC@CORP') is a syntax error outright. Quoting makes
it one valid identifier: "SVC@CORP".table.
An ordinary name is returned unchanged, so nothing about the generated SQL changes for a normal login. This is the identifier form only -- the raw name is what belongs in a credential or a DBC string predicate.
Retrieve process information including SQL and columns from the process catalog.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process_id
|
str
|
The unique identifier of the process. |
required |
Returns: dict: A dictionary containing process SQL and columns.
Robustly extract and parse JSON from LLM responses. Handles markdown code fences, escaped characters, and formatting variations.
Parameters¶
response : str The raw response string from the LLM.
Returns¶
dict The parsed JSON as a dictionary.
Raises¶
ValueError If JSON cannot be extracted or parsed from the response.
True when exc looks like the backend refusing our structured-output parameter
(guided_json / response_format / format) rather than a genuine
model/content error. Covers the 4xx shapes seen from OpenAI-compatible gateways
(LiteLLM, Mistral, ...): extra_forbidden / "Extra inputs are not permitted" /
"unknown|unrecognized|unsupported field|parameter" naming one of those keys.
A genuine model error (context length, content policy) never names these parameters, so keying on them keeps the one-shot fallback from masking real failures.
_build_provider_llm_caller(llm: ChatOpenAI, provider: str, base_schema: Optional[Dict] = None, schema_name: str = 'Output')
Return a callable(messages) -> AIMessage that asks provider for JSON
matching base_schema (a plain JSON Schema dict) using that provider's native
structured-output mechanism, with a one-shot fallback to an unconstrained call if the
backend rejects the structured-output parameter.
This is the single source of truth for how each provider is asked for structured JSON — do not re-branch on provider elsewhere. Mechanisms:
vllm->extra_body={"guided_json": <schema>}openai/azure/mistralai/openai-compatible->response_format={"type":"json_schema", ...}(strictonly for openai/azure — see_STRICT_JSON_SCHEMA_PROVIDERS)ollama->format=<schema>- anything else -> no constraint (prompt-guided JSON only)
Parameters¶
llm : ChatOpenAI
The language model interface.
provider : str
The LLM provider (vllm, openai, azure, mistralai, openai-compatible, ollama, ...).
base_schema : dict, optional
Plain JSON Schema for the desired output. None -> unconstrained call.
schema_name : str
Name used inside the OpenAI json_schema wrapper (ignored by other mechanisms).
Returns¶
callable
A function that invokes the LLM with the appropriate structured-output
constraint and the one-shot unconstrained fallback. Two attributes are
set on it for diagnostics — mechanism (the wire parameter chosen for
this provider, or 'none') and fell_back (True once the
backend has refused that parameter and the call was retried
unconstrained). They exist because that fallback is otherwise
invisible: documentation keeps working on prompt-guided JSON while
every configuration report stays green, so
check_configuration() reads them to tell the difference between
"constrained" and "quietly degraded". Nothing about the call itself
changes.
One argument object from several parallel calls of the same tool.
Seen live (Bedrock, Ministral): asked for five dataset sections, the model sent them over several calls, and asked for FEATURE_THEMES alone it sent one call per theme; only the first call used to be read. Keys are merged in call order; a key repeated with another text is joined line by line, lists are concatenated. Arguments that are not objects: the first call's, as before.
Move tool-call arguments into the message content, if there are any.
unwrap: the property a non-object schema was wrapped in (see the
function_calling branch of :func:_build_provider_llm_caller); its value
becomes the content. Arguments without that property are kept whole, so a
model that answered with the bare list still parses.
Tool calling puts the structure in tool_calls and typically leaves
content empty — which every caller of the provider dispatcher would read
as an empty answer. Normalising here keeps the dispatcher's contract ("an
AIMessage whose content is the JSON") identical across all mechanisms, so no
call site needs to know which one was used.
Returns response unchanged when there is no tool call: an endpoint that
ignored tool_choice and answered in prose is exactly the degradation the
caller's parser and check_configuration already handle, and inventing
content here would hide it.
Stamp a provider caller with the mechanism it uses. See the note in
:func:_build_provider_llm_caller's Returns section.
_print_documentation(documented_sql: str, entity_description: str, documented_entity_columns: Dict[str, str], documented_feature_columns: Dict[str, str], process_id: Optional[str] = None, view_name: Optional[str] = None, explain_analysis: Optional[str] = None, user_score: Optional[int] = None, optimization_score: Optional[int] = None, explain_warnings: Optional[List[str]] = None, explain_recommendations: Optional[List[str]] = None, sql_query: Optional[str] = None, explain_plan: Optional[str] = None, n_steps: Optional[int] = None, n_spool_objects: Optional[int] = None) -> None
Pretty print documentation with context-aware formatting. Uses HTML in notebooks, text format in regular scripts.
Parameters¶
documented_sql : str The query business logic description. entity_description : str The entity description. documented_entity_columns : dict Mapping of entity column names to descriptions. documented_feature_columns : dict Mapping of feature column names to descriptions. process_id : str, optional The process identifier for the title. view_name : str, optional The view name for the title. explain_analysis : str, optional The EXPLAIN plan analysis description. optimization_score : int, optional Optimization score from 1 to 5. explain_warnings : list, optional List of warnings from EXPLAIN analysis. explain_recommendations : list, optional List of recommendations from EXPLAIN analysis. sql_query : str, optional The original SQL query to display. explain_plan : str, optional The raw EXPLAIN plan output to display.
build_llm(llm_service: str = None, api_key: str = None, model_id: str = None, temperature: float = 0.0, timeout: int = 120, provider: str = None)
Build and return a chat LLM client based on the configured provider.
When called with no arguments, reads from tdfs4ds global configuration: - INSTRUCT_MODEL_PROVIDER (provider) - INSTRUCT_MODEL_URL (llm_service) - INSTRUCT_MODEL_API_KEY (api_key) - INSTRUCT_MODEL_MODEL (model_id) - INSTRUCT_MODEL_REGION (AWS region for Bedrock) - INSTRUCT_MODEL_AZURE_DEPLOYMENT, INSTRUCT_MODEL_AZURE_API_VERSION (Azure) - INSTRUCT_MODEL_GCP_PROJECT, INSTRUCT_MODEL_GCP_LOCATION (GCP)
Injects tdfs4ds config variables into environment so that boto3, Azure, and Google Cloud clients pick up credentials and endpoints automatically.
Parameters¶
llm_service : str, optional Base URL of the LLM service (overrides INSTRUCT_MODEL_URL). api_key : str, optional API key for authentication (overrides INSTRUCT_MODEL_API_KEY). model_id : str, optional Model identifier (overrides INSTRUCT_MODEL_MODEL). temperature : float Sampling temperature for response diversity. timeout : int Request timeout in seconds. provider : str, optional LLM provider name (overrides INSTRUCT_MODEL_PROVIDER).
Returns¶
BaseChatModel Configured LLM client (ChatOpenAI, AzureChatOpenAI, ChatBedrockConverse, ChatVertexAI, etc.).
Raises¶
Exception If LLM client creation fails or provider is not supported.
Build a provider-appropriate JSON Schema used to enforce strict JSON output for SQL column documentation across multiple LLM backends.
This function returns different schema shapes depending on the LLM provider, because each ecosystem uses a different structured-output mechanism:
Provider Modes¶
-
provider="openai", "azure" Returns the JSON Schema wrapped in OpenAI's
response_format={"type": "json_schema", ...}structure. Supported by GPT-4.1, GPT-4o, GPT-3.5-Turbo, and Azure OpenAI. -
provider="anthropic", "claude" Returns an Anthropic tool schema definition. Claude 3.x models use tool schemas to enforce strict JSON output.
-
provider="ollama" Returns the raw JSON schema that Ollama expects under the
format=parameter of the generate API. (Ollama 0.2+ supports response schemas.) -
provider="vllm" Returns plain JSON Schema for use with vLLM's
guided_jsonconstrained decoding. -
provider="bedrock" Bedrock Claude follows the Anthropic tool schema format. Bedrock Llama / Titan accept plain JSON schema. This function returns the base schema and leaves the final wrapping to the caller.
-
provider="generic" Returns plain JSON schema. Useful for LLM backends that do not support constrained decoding, prompt-only JSON generation, or post-processing repair.
Parameters¶
columns : list[str] Column names to include as required JSON object keys. Each column will map to a string description generated by the model.
str, optional
The model provider or backend type. Determines the structural format required for constrained generation. One of: "openai", "anthropic", "ollama", "vllm", "bedrock", "generic".
Returns¶
dict A dictionary representing the JSON Schema or provider-specific wrapper used to enforce strict JSON output during LLM generation.
Notes¶
-
All schemas require that:
- the output be a JSON object
- keys match exactly the column names
- all values be strings
- additional properties be disallowed
-
Not all providers enforce schemas equally:
- OpenAI, Claude, and vLLM offer hard guarantees.
- Ollama enforces schema reasonably well.
- Generic models may require post-processing.
build_sql_documentation_chain(llm: ChatOpenAI, entity_columns: Sequence[str], feature_columns: Sequence[str], provider: str = 'vllm', json_constraint: bool = True, prompt_only: bool = False) -> Runnable
Build a LangChain Runnable that generates business-focused documentation
for lists of entity and feature columns from a SQL query output, with optional provider-specific JSON
constraints (vLLM, OpenAI, Ollama, etc.).
The resulting chain expects two input variables:
- sql_query: str → the SQL query whose output is being documented
- columns_str: str → formatted list of entity and feature columns (e.g. "Entity columns:
- col1
Feature columns: - col2")
Parameters
----------
llm : ChatOpenAI
The language model interface (may point to vLLM, OpenAI, Ollama, etc.).
entity_columns : Sequence[str]
List of entity/identifier columns that must appear as keys in the output JSON.
feature_columns : Sequence[str]
List of feature columns that must appear as keys in the output JSON.
provider : str, optional (default="vllm")
Indicates which structured-output mechanism to use.
Supported values:
- "vllm" → uses `guided_json` for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via `response_format`
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via `response_format` (NOT
vLLM `guided_json`; that is a vLLM-only extension)
- "ollama" → uses Ollama's `format=` schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True:
- a JSON Schema is generated from the column lists
- provider-specific constrained decoding is applied
If False:
- the chain does not enforce JSON structure at the LLM level
- the model is only guided by the prompt (weaker guarantees)
prompt_only : bool, optional (default=False)
If True:
- returns only the prompt template, without attaching the LLM or parser
- useful for debugging, testing, or customizing the prompt before execution
If False:
- returns the full chain: prompt → LLM (optionally schema-guided) → JSON parser
Returns
-------
Runnable
If prompt_only=False:
A LangChain Runnable that executes:
prompt → LLM (optionally schema-guided) → JSON parser
When invoked with:
{
"sql_query": "...",
"columns_str": "Entity columns:
- column1
Feature columns: - column2 ..." } It returns: dict[str, str] A mapping of each requested column name to a short, business-oriented description (≤ 5 sentences), plus a 'query_business_logic' key containing a high-level description of the query's business logic (5-10 sentences), and an 'entity_description' key with a holistic description of the entity (3-5 sentences). If prompt_only=True: The prompt template itself, for inspection or further customization.
Notes
-----
- The chain enforces valid JSON when possible:
* vLLM → `guided_json`
* OpenAI → `response_format={"type": "json_schema", ...}`
* Ollama → `format=<schema>`
- For unsupported providers, the model may emit imperfect JSON.
- Descriptions focus on business meaning, business logic,
and optionally technical details only when relevant.
run_sql_documentation(chain: Runnable, sql_query: str, entity_columns: Sequence[str], feature_columns: Sequence[str], language: str = 'English', upstream_context: str = '') -> Dict[str, str]
Execute a previously constructed SQL-documentation chain and return business-friendly documentation for the specified SQL output columns.
This function prepares the chain inputs (SQL query, formatted column list,
target language) and invokes the chain. The chain itself must have been
created using build_sql_documentation_chain(), which ensures the model
produces structured JSON suitable for parsing.
Parameters¶
chain : Runnable
A LangChain Runnable returned by build_sql_documentation_chain().
This Runnable encapsulates:
- the prompt template
- a provider-specific LLM invocation (with or without JSON constraints)
- a JSON output parser
str
The SQL query whose resulting columns should be documented. This query is shown to the model so it can infer business logic, derivation rules, and column meaning.
Sequence[str]
The list of entity/identifier column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
Sequence[str]
The list of feature column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
str, optional (default="English")
The target output language for the generated documentation.
This value is passed into the prompt’s {language} variable.
Examples: "English", "French", "German", "Spanish", "Japanese".
Returns¶
dict[str, str] A dictionary mapping each column name to a human-readable, business-oriented description generated by the model, plus a 'query_business_logic' key with the query's business logic description, and an 'entity_description' key with the holistic entity description. Example: { "customer_id": "Unique customer identifier used for ...", "order_date": "Business date when the order was created ...", "entity_description": "The customer entity represents...", "query_business_logic": "This query provides a view of ..." }
Notes¶
- The output format is determined by the chain's JSON parser. If the model
fails to produce valid JSON (e.g., due to unsupported constraints),
a
OutputParserExceptionmay be raised. - The resulting descriptions are typically ≤ 5 sentences per column, unless modified in the chain's prompt.
Return True when the SQL inverts a score, i.e. contains a 1 - (...) /
1.0 - X transform (so a lower output value means higher risk).
Single source of truth for the inversion signal, shared by document-time
hint generation (:func:_extract_interpretation_hints) and the agent's
runtime DEFINITION/EXPLAIN reconciliation guard.
Scan the full SQL text for signals that constrain how column descriptions must read.
Operates on the entire SQL string (including WHERE, GROUP BY, subqueries, CTEs) so it catches patterns that a single-level SELECT AST walk would miss.
Returns a dict of boolean flags. All-False means plain arithmetic — no extra constraints are added to the LLM prompt.
Convert interpretation hints into a context snippet appended to upstream_context.
Returns an empty string when all hints are False (no constraints needed).
Check feature descriptions against interpretation hints.
Returns a list of (column_name, violation_reason) tuples for any description that contradicts a hint. Empty list when all descriptions are consistent.
document_sql_query_columns(sql_query: str, entity_columns: Sequence[str], feature_columns: Sequence[str], language: str = 'English', provider: Optional[str] = None, json_constraint: bool = True, prompt_only: bool = False, upstream_context: str = '') -> Dict[str, Any]
Convenience function to generate business-focused documentation for SQL query output columns using the configured instruction model from tdfs4ds settings.
This function automatically builds the LLM client using the tdfs4ds configuration variables (INSTRUCT_MODEL_URL, INSTRUCT_MODEL_API_KEY, INSTRUCT_MODEL_MODEL), constructs the documentation chain, and executes it to produce column descriptions.
Parameters¶
sql_query : str The SQL query whose resulting columns should be documented. This query is shown to the model so it can infer business logic, derivation rules, and column meaning.
Sequence[str]
The list of entity/identifier column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
Sequence[str]
The list of feature column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
str, optional (default="English")
The target output language for the generated documentation. This value is passed into the prompt's {language} variable. Examples: "English", "French", "German", "Spanish", "Japanese".
str, optional (default=None)
Indicates which structured-output mechanism to use for the LLM.
If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
bool, optional (default=True)
If True: - a JSON Schema is generated from the column lists - provider-specific constrained decoding is applied If False: - the chain does not enforce JSON structure at the LLM level - the model is only guided by the prompt (weaker guarantees)
bool, optional (default=False)
If True: - returns only the prompt template, without executing the chain - useful for debugging, testing, or customizing the prompt before execution If False: - executes the full chain and returns structured documentation
Returns¶
dict If prompt_only=False: A dictionary with four keys: - "query_business_logic": str containing the high-level business logic description of the query - "entity_description": str containing the holistic description of the entity - "entity_columns": dict[str, str] mapping each entity column name to its description - "feature_columns": dict[str, str] mapping each feature column name to its description If prompt_only=True: The prompt template itself, for inspection or further customization.
Raises¶
ValueError If any of the required tdfs4ds configuration variables (INSTRUCT_MODEL_URL, INSTRUCT_MODEL_API_KEY, INSTRUCT_MODEL_MODEL, INSTRUCT_MODEL_PROVIDER) are not set.
Notes¶
- This function requires that the tdfs4ds instruction model configuration is properly set.
- The resulting descriptions are typically ≤ 5 sentences per column, focusing on business meaning and logic.
- If the model fails to produce valid JSON, an exception will be raised.
{feature: [entity columns named]} for the feature descriptions that refer
to the entity by its key column names (whole word, case-insensitive) rather
than by its business meaning. Entity-column descriptions are not checked.
Correction note for the retry: name the entity by its meaning.
Classify each [You] item by semantic category using a single LLM call.
Returns a list of dicts {"item": ..., "category": ...} in the same order
as the input. Falls back to "OTHER" for every item when the LLM is not
available or the call fails.
Categories¶
UNCORRELATED_SCALAR_SUBQUERY
Flags an uncorrelated scalar subquery (e.g. SELECT AVG(col) FROM t)
as causing redundant scans or per-row evaluation. Teradata evaluates
these once — this is a false positive.
CROSS_JOIN_SINGLE_ROW
Flags a CROSS JOIN to a single-row aggregate result as a Cartesian
product. 1 x N = N rows, not a real performance problem.
INFORMATIONAL
The item explicitly states no action is needed, the query is already
correct, or contradicts itself (e.g. "Teradata evaluates it once").
PRODUCT_JOIN
Real Cartesian product between multi-row tables.
REDUNDANT_SCAN
A table is scanned multiple times for a non-scalar reason (e.g. self-join).
PI_MISMATCH
Join on non-primary-index columns forcing data redistribution.
TYPE_CAST
Implicit type conversion in join or WHERE predicates.
OTHER
Any issue that does not match the above categories.
_compute_user_score_from_you_items(warnings: list, recommendations: list, explain_text: str = None, classify: bool = True) -> int
Derive user_score deterministically from [You]-prefixed items.
The LLM identifies issues reliably (factual observations) but assigns
numeric scores inconsistently — the same SQL can score 3/5 or 4/5
depending on prompt context. This function replaces the LLM-assigned
score with a deterministic mapping based on the number and severity
of [You] items the LLM identified:
=== ============================================ 5 No [You] items — no SQL-level issues found 4 1 distinct issue 3 2–3 distinct issues 2 4+ issues or significant structural flaw 1 Critical bug (Cartesian product, wrong results) === ============================================
Parameters¶
classify : bool, default True When True, runs the LLM semantic classification to filter false-positive [You] items. Set to False during intermediate scoring (e.g. the simplification baseline) where every [You] item should count so the optimizer has the raw signal to drive structural improvements.
Scoring pipeline:
- Extract
[You]-prefixed warnings and recommendations. - LLM semantic classification (when
classify=True) — each item is classified into a category via a single LLM call. Items in false-positive categories are filtered out. Falls back to regex-based filtering when the LLM is unavailable. Skipped entirely whenclassify=False. - EXPLAIN validation (when
explain_textis provided) — items that claim an issue not present in the raw EXPLAIN are filtered. - Severity classification + count mapping — deterministic 1–5 score.
Extract deterministic execution metrics from raw Teradata EXPLAIN text.
Returns a dict with:
- n_steps: number of numbered execution steps (e.g. 1) First, ...)
- n_spool_objects: count of distinct Spool N references
Extract concrete execution signals from raw Teradata EXPLAIN text.
Returns a dict of ground-truth signals that can be used to validate
LLM-generated [You] items against what the EXPLAIN actually shows.
Keys:
- has_product_join: True if the EXPLAIN contains a real product join
- table_scan_counts: {TABLE_NAME: n_scans} — how many data-access
steps read from each table (Spool reads excluded)
- has_duplicate_scans: True if any table is scanned more than once
- has_redistribution: True if any step redistributes data by hash
build_explain_documentation_chain(llm: ChatOpenAI, provider: str = 'vllm', json_constraint: bool = True, prompt_only: bool = False) -> Runnable
Build a LangChain Runnable that analyzes SQL EXPLAIN plans and generates optimization scores, warnings, and recommendations.
The resulting chain expects two input variables
- sql_query: str → the original SQL query
- explain_plan: str → the EXPLAIN output from the database
Parameters¶
llm : ChatOpenAI
The language model interface (may point to vLLM, OpenAI, Ollama, etc.).
provider : str, optional (default="vllm")
Indicates which structured-output mechanism to use.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True: a JSON Schema is generated and provider-specific constrained decoding is applied.
If False: the chain does not enforce JSON structure at the LLM level.
Returns¶
Runnable A LangChain Runnable that executes: prompt → LLM (optionally schema-guided) → JSON parser
When invoked with:
{
"sql_query": "SELECT ...",
"explain_plan": "..."
}
It returns:
dict with keys:
- "explanation": str describing the EXPLAIN plan in business terms
- "optimization_score": int from 1 (poorly optimized) to 5 (well optimized)
- "warnings": list[str] of potential issues or concerns
- "recommendations": list[str] of actionable optimization suggestions
run_explain_documentation(chain: Runnable, sql_query: str, explain_plan: str, classify: bool = True) -> Dict[str, Any]
Execute an EXPLAIN-documentation chain and return optimization analysis.
Parameters¶
chain : Runnable
A LangChain Runnable returned by build_explain_documentation_chain().
sql_query : str
The original SQL query.
explain_plan : str
The EXPLAIN output from the database.
classify : bool, default True
Pass through to _compute_user_score_from_you_items. Set to False
during intermediate scoring (simplification, candidate evaluation)
to keep the raw [You] count and avoid filtering useful structural signals.
Returns¶
dict A dictionary with keys: "explanation", "optimization_score", "warnings", "recommendations"
document_sql_query_explain(sql_query: str, provider: Optional[str] = None, json_constraint: bool = True, prompt_only: bool = False, classify: bool = True) -> Dict[str, Any]
Analyze a SQL query's EXPLAIN plan and return optimization recommendations.
This function automatically builds the LLM client using tdfs4ds configuration, constructs the EXPLAIN analysis chain, and executes it.
Parameters¶
sql_query : str The original SQL query. explain_plan : str The EXPLAIN output from the database. provider : str, optional (default=None) Indicates which structured-output mechanism to use for the LLM. If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config. Supported values: "vllm", "openai", "azure", "ollama", etc. json_constraint : bool, optional (default=True) If True: use provider-specific constrained decoding. If False: rely on prompt guidance only.
Returns¶
dict A dictionary with keys: - "explanation": str describing the EXPLAIN plan - "optimization_score": int from 1 to 5 - "warnings": list[str] of potential issues - "recommendations": list[str] of actionable suggestions
Raises¶
ValueError If tdfs4ds instruction model configuration is incomplete.
Create the necessary documentation tables in the database if they do not already exist. tdml: The tdfs4ds TDML connection object.
(Re)create the CURRENT VALIDTIME + LOCKING ROW FOR ACCESS views over the three
business dictionary tables — used by the Teradata Enterprise Vector Store backend
(vector_store_teradata.py) so a permanent teradataml.DataFrame can be passed
directly to from_datasets()/add_datasets(), instead of routing through
from_documents()/add_documents()'s Document-conversion path, whose internal
staging call (DataFrame.from_dict(persist=True)) has no database argument
and may materialize as a volatile table invisible to the Vector Store's
separate REST session. REPLACE VIEW keeps this idempotent across re-runs,
so it is safe (and intended) to call unconditionally on every connect()/setup(),
mirroring feature_store_catalog_view_creation()/process_store_catalog_view_creation()
— this avoids depending on a table-existence probe to decide whether the views
need (re)creating.
ID mirrors the doc-id convention in agent/vector_index.py's _objects_to_docs /
_sections_to_docs / _columns_to_docs (f"{db}__{obj}" etc.) so switching between
the Chroma and Teradata backends keeps stable, comparable identifiers. The
blank/NULL-description filter mirrors those functions' if not text.strip():
continue guard, pushed into SQL so the view itself excludes rows not worth
embedding.
(Re)create the public views over the three process-documentation tables.
Their names were declared (DOCUMENTATION_PROCESS_*_VIEW) but the views
were never created, so a store attached read-only could share its business
dictionary but not its process documentation (0.3.3.15: the admin app reads
both from every attached store, through views, since read rights on the
views are all a reader is expected to have). CURRENT VALIDTIME +
LOCKING ROW FOR ACCESS, like the business-dictionary views. Never
raises: a view that cannot be replaced is kept and reported.
Ensure the FS_PROCESS_DOCUMENTATION_EXPLAIN table has all expected columns.
Check-first: inspects the live DDL and only runs ALTER TABLE for columns that are actually missing. No-op (and no log at INFO) when already up-to-date.
Columns added over time: - USER_SCORE (v0.2.x) — deterministic score from [You] items - N_STEPS (v0.2.9) — EXPLAIN step count - N_SPOOL_OBJECTS (v0.2.9) — distinct Spool object count
New columns start as NULL for existing rows and are filled on the next
document_process() or document_process_incremental() run.
Ensure FS_BUSINESS_DICTIONARY_COLUMNS has the interpretation-contract columns.
Check-first: inspects live DDL and only runs ALTER TABLE for columns that are actually missing. No-op (and no log at INFO) when already up-to-date.
Columns added over time: - MONOTONIC_DIRECTION (v0.3.1.35) — direction hint derived from formula - VALUE_DOMAIN (v0.3.1.35) — value range hint - IS_ABSOLUTE_VALUE (v0.3.1.35) — True when ABS() detected in formula - FORMULA_CONSISTENCY_WARNING (v0.3.1.35) — set when LLM violated hint after retry
New columns start as NULL for existing rows.
document_process(process_id: str, language: str = 'English', json_constraint: bool = True, show_sql_query: bool = False, show_explain_plan: bool = False, display: bool = True, upload: bool = True, prompt_only=False) -> Optional[Dict[str, Any]]
Generate and store documentation for a data process identified by process_id. This function retrieves the SQL query and output columns for the process, generates business-focused documentation using an LLM, and stores the results in the appropriate documentation tables.
Parameters¶
process_id : str The unique identifier of the data process to document.
str, optional (default="English")
The target output language for the generated documentation. This value is
passed into the prompt’s {language} variable. Examples: "English", "French", "German", "Spanish", "Japanese".
provider : str, optional (default=None)
Indicates which structured-output mechanism to use for the LLM.
If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True:
- a JSON Schema is generated from the column list
- provider-specific constrained decoding is applied
If False:
- the chain does not enforce JSON structure at the LLM level
- the model is only guided by the prompt (weaker guarantees)
show_sql_query : bool, optional (default=False)
If True, display the original SQL query at the end of the documentation report.
show_explain_plan : bool, optional (default=False)
If True, display the raw EXPLAIN plan output at the end of the documentation report.
display : bool, optional (default=True)
If True, print the generated documentation to the console.
upload_documentation : bool, optional (default=True)
If True, upload the generated documentation to the documentation tables.
Returns¶
dict or None A dictionary containing the generated documentation and analysis, or None if an error occurred. The dictionary includes keys: - PROCESS_ID - DOCUMENTED_SQL - ENTITY_DESCRIPTION - DOCUMENTED_ENTITY_COLUMNS - DOCUMENTED_FEATURE_COLUMNS - EXPLAIN_ANALYSIS (if show_explain_plan is True) - OPTIMIZATION_SCORE (if show_explain_plan is True) - EXPLAIN_WARNINGS (if show_explain_plan is True) - EXPLAIN_RECOMMENDATIONS (if show_explain_plan is True) - RAW_EXPLAIN_PLAN (if show_explain_plan is True) Notes
- This function requires that the tdfs4ds instruction model configuration is properly set.
- If the model fails to produce valid JSON, an exception will be raised.
- The resulting descriptions are typically ≤ 5 sentences per column, focusing on business meaning and logic.
Replace characters outside Teradata LATIN (ISO-8859-1) character set.
LLM-generated text often contains Unicode characters (smart quotes, em dashes, ellipsis, etc.) that trigger Teradata Error 6706 when inserted into LATIN-charset columns. This function replaces the most common offenders with ASCII equivalents, then encodes the remainder as ISO-8859-1, substituting any still-unmappable code points with '?'.
(database, view) of a process, as the business dictionary keys it.
Keep the human-written parts of a process's documentation (0.3.2.1).
Reads the IMPORTED / MANUAL parts of the process view from the provenance
table and overlays their stored text onto process_info (business
logic, entity description, entity columns, features), so what is uploaded,
displayed and compared (_doc_changed) is the final text. Returns what is
kept: {'features': {NAME}, 'description': bool, 'business_logic': bool,
'entity_columns': {NAME}} - all empty when nothing is protected, when
overwrite_manual is in force, or when the provenance table is absent.
FEATURE_IDs of the features THIS process produces (name, domain AND entity).
The feature MERGE of :func:upload_documentation used to join the catalog on
name and data domain only: a domain holding one name for two entities got the
description written under both FEATURE_IDs (seen live), and the extra row could
then hide a hand-written correction when the rows are read back by name.
Entities compare order-insensitively, like catalog_counts. None when it
cannot be resolved (no entity, catalog unreadable, features not registered yet):
the caller then keeps its SQL unchanged.
Find (and with dry_run=False close) feature-documentation rows filed under
another entity's FEATURE_ID.
Before 0.3.2.1 :func:upload_documentation wrote a process's feature
description under every FEATURE_ID of that name in the data domain, whatever
the entity. Returns the rows concerned (PROCESS_ID, FEATURE_ID, FEATURE_NAME,
the process entity and the feature's entity). History is kept: the rows are
closed with CURRENT VALIDTIME DELETE.
_overlay_human_parts(process_info: Dict[str, Any], human: Dict[str, Any], keep: Dict[str, Any]) -> None
Lay the human-written parts (human, filled by
:func:_apply_protected_process_parts) over freshly generated documentation.
Prompt block: the parts of this process a person wrote, as authoritative.
Upload the generated documentation for a data process into the documentation tables.
Parameters¶
process_info : dict A dictionary containing the process documentation information. Expected keys: - PROCESS_ID: str - DOCUMENTED_SQL: str - ENTITY_DESCRIPTION: str - DOCUMENTED_ENTITY_COLUMNS: dict[str, str] - DOCUMENTED_FEATURE_COLUMNS: dict[str, str]
Retrieve the documentation for a data process from the documentation tables.
Parameters¶
process_id : str The unique identifier of the data process.
Returns¶
dict A dictionary containing the documentation information with keys: - documented_sql: str - entity_description: str - documented_entity_columns: dict[str, str] - documented_feature_columns: dict[str, str]
Retrieve the EXPLAIN documentation for a data process from the documentation tables.
Parameters¶
process_id : str The unique identifier of the data process.
Returns¶
dict A dictionary containing the EXPLAIN documentation information with keys: - explanation: str - optimization_score: int - warnings: list[str] - recommendations: list[str]
Upload the EXPLAIN documentation for a data process into the documentation tables.
Parameters¶
process_id : str The unique identifier of the data process. explain_documentation : dict A dictionary containing the EXPLAIN documentation information with keys: - explanation: str - optimization_score: int - warnings: list[str] - recommendations: list[str]
Pretty print the documentation and EXPLAIN analysis for a data process from process_info dict or by retrieving it using process_id.
Parameters¶
process_info : dict, optional (default=None) A dictionary containing the process documentation information. If None, process_id must be provided to retrieve the information. process_id : str, optional (default=None) The unique identifier of the data process. If process_info is None, this parameter is used to retrieve the documentation.
Returns None
feed_process_info_with_prompt_result(process_info, sql_documentation_response=None, sql_explain_response=None, display_info=True, upload_info=True)
Enriches a process_info dictionary with SQL documentation and EXPLAIN plan analysis results, with options to display the results and upload the enriched information.
This function integrates the results of SQL documentation and EXPLAIN plan analysis into the provided
process_info dictionary. It extracts and organizes documentation for entity and feature columns,
as well as optimization insights, to provide a comprehensive view of the SQL query's business logic,
performance, and potential improvements. It also supports optional display of the enriched information
and automatic upload of the documentation and EXPLAIN analysis to a backend system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process_info
|
dict
|
A dictionary containing metadata about the SQL process, including: - 'ENTITY_COLUMNS': List of columns representing the entity in the SQL query. - 'FEATURE_COLUMNS': List of columns representing features in the SQL query. |
required |
sql_documentation_response
|
dict
|
A dictionary containing SQL documentation results, including descriptions for entity/feature columns and query business logic. Expected keys: - 'query_business_logic': Description of the query's purpose and logic. - 'entity_description': Description of the entity represented by the query. - Column names as keys, with their descriptions as values. |
None
|
sql_explain_response
|
dict
|
A dictionary containing SQL EXPLAIN plan analysis results, including: - 'explanation': Detailed analysis of the EXPLAIN plan. - 'optimization_score': Integer score (1-5) indicating query optimization level. - 'warnings': List of potential issues identified in the EXPLAIN plan. - 'recommendations': List of actionable recommendations for query optimization. |
None
|
display_info
|
bool
|
If True, displays the enriched process_info using |
True
|
upload_info
|
bool
|
If True, uploads the enriched documentation and EXPLAIN analysis to a backend system. Defaults to True. |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The enriched |
Example
process_info = { ... 'ENTITY_COLUMNS': ['customer_id', 'order_id'], ... 'FEATURE_COLUMNS': ['order_amount', 'order_date'] ... } sql_documentation_response = { ... 'query_business_logic': 'This query joins customer and order data...', ... 'entity_description': 'The customer entity represents...', ... 'customer_id': 'Unique identifier for customers.', ... 'order_amount': 'Total amount of the order.' ... } sql_explain_response = { ... 'explanation': 'The EXPLAIN plan shows a nested loop join...', ... 'optimization_score': 3, ... 'warnings': ['Full table scan on orders table'], ... 'recommendations': ['Add index on orders.customer_id'] ... } enriched_info = feed_process_info_with_prompt_result( ... process_info, ... sql_documentation_response, ... sql_explain_response, ... display_info=True, ... upload_info=True ... ) print(enriched_info.keys()) ['ENTITY_COLUMNS', 'FEATURE_COLUMNS', 'DOCUMENTED_SQL', 'ENTITY_DESCRIPTION', 'DOCUMENTED_ENTITY_COLUMNS', 'DOCUMENTED_FEATURE_COLUMNS', 'EXPLAIN_ANALYSIS', 'OPTIMIZATION_SCORE', 'EXPLAIN_WARNINGS', 'EXPLAIN_RECOMMENDATIONS']
Provenance + log of a business-dictionary load (source set).
keys are (db, obj, part, part_name). One log row per object, all
under one parent row for the call, unless the caller already logs
(event_id given: the manual-documentation API). Never raises.
Unique name of a business-dictionary staging table (tdfs_tmp_<12 hex>).
The prefix is the one cleanup_temp_tables already recognises, so a
table left behind by a crash can be listed and dropped with it.
Write df to a regular staging table in the store schema; return its name.
This used to be a VOLATILE table (temporary=True) read back by a second
statement. A volatile table lives only in the session that created it, and
the admin app's threads share teradataml's pooled connection, so the MERGE
could run in another session (or after a reconnect) and fail with Error 3807
"Object ... does not exist" -- which is how an edit made in the app failed
while the same upload from a notebook worked. A uniquely named regular table
does not depend on which connection runs which statement.
upload_business_dictionary_objects(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update object-level descriptions in the business dictionary.
source (0.3.2.1) records who wrote these rows in the provenance table:
'IMPORTED' (default: a load of an existing data dictionary, e.g. source
data documentation) or 'MANUAL'. Both are kept by later LLM
documentation runs. author is the declared editor name (default: the OS
user). The LLM pipelines call this with source=None (nothing recorded).
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the object
- OBJECT_NAME : str — table or view name
- OBJECT_TYPE : str — 'T' for table, 'V' for view
- BUSINESS_DESCRIPTION: str — business-oriented description of the object
Rows are merged into FS_BUSINESS_DICTIONARY_OBJECTS using
CURRENT VALIDTIME semantics (Teradata temporal). Existing rows
matched on (DATABASE_NAME, OBJECT_NAME) are updated; new rows are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, OBJECT_NAME, OBJECT_TYPE, BUSINESS_DESCRIPTION.
upload_business_dictionary_columns(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update column-level descriptions in the business dictionary.
source / author: see :func:upload_business_dictionary_objects.
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the table
- TABLE_NAME : str — table or view name
- COLUMN_NAME : str — column name
- BUSINESS_DESCRIPTION: str — business-oriented description of the column
Rows are merged into FS_BUSINESS_DICTIONARY_COLUMNS using
CURRENT VALIDTIME semantics (Teradata temporal). Existing rows
matched on (DATABASE_NAME, TABLE_NAME, COLUMN_NAME) are updated;
new rows are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, TABLE_NAME, COLUMN_NAME, BUSINESS_DESCRIPTION.
upload_business_dictionary_sections(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update section-level descriptions in the business dictionary.
source / author: see :func:upload_business_dictionary_objects.
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the object
- OBJECT_NAME : str — object name (table, view, dataset)
- SECTION_NAME : str — section identifier (e.g. 'OVERVIEW', 'FEATURE_THEMES')
- SECTION_CONTENT: str — section text content
Rows are merged into FS_BUSINESS_DICTIONARY_SECTIONS using
CURRENT VALIDTIME semantics. Existing rows matched on
(DATABASE_NAME, OBJECT_NAME, SECTION_NAME) are updated; new rows
are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, OBJECT_NAME, SECTION_NAME, SECTION_CONTENT.
Remove all documentation rows for a process from every documentation table.
Deletes from:
- FS_PROCESS_DOCUMENTATION_BUSINESS_LOGIC (keyed by PROCESS_ID)
- FS_PROCESS_DOCUMENTATION_FEATURES (keyed by PROCESS_ID)
- FS_PROCESS_DOCUMENTATION_EXPLAIN (keyed by PROCESS_ID)
- FS_BUSINESS_DICTIONARY_OBJECTS (keyed by the process VIEW_NAME)
- FS_BUSINESS_DICTIONARY_COLUMNS (keyed by the process VIEW_NAME)
- FS_BUSINESS_DICTIONARY_SECTIONS (keyed by the process VIEW_NAME)
Uses NONSEQUENCED VALIDTIME DELETE to fully remove all temporal versions,
mirroring the approach used by remove_process for the process catalog.
Each step is independent and non-fatal: a failure on one table is logged as a warning and the remaining tables are still cleaned up.
Parameters¶
process_id : str UUID of the process whose documentation should be removed.
delete_business_dictionary_objects(object_names: list, *, database_name: Optional[str] = None) -> None
Expire currently-valid object-level entries from the business dictionary.
Parameters¶
object_names : list of str
Object names to delete (e.g. ['TRANSACTIONS', 'CUSTOMERS']).
database_name : str, optional
When provided, only rows with this DATABASE_NAME are deleted.
When omitted, all databases are affected (useful to clean up stale
entries that were uploaded under a wrong database name).
delete_business_dictionary_columns(table_name: str, *, column_names: Optional[list] = None, database_name: Optional[str] = None) -> None
Expire currently-valid column-level entries from the business dictionary.
Parameters¶
table_name : str Table whose column descriptions should be deleted. column_names : list of str, optional Specific columns to delete. When omitted all columns of the table are deleted. database_name : str, optional When provided, only rows with this DATABASE_NAME are deleted. When omitted, all databases are affected.
delete_business_dictionary_sections(object_name: str, *, section_names: Optional[list] = None, database_name: Optional[str] = None) -> None
Expire currently-valid section entries from the business dictionary.
Parameters¶
object_name : str
Object whose section entries should be deleted (OBJECT_NAME column).
section_names : list of str, optional
Specific sections to delete (e.g. ['OVERVIEW', 'ENTITY']).
When omitted all sections for the object are deleted.
database_name : str, optional
When provided, only rows with this DATABASE_NAME are deleted.
When omitted, all databases are affected.
Remove all business dictionary documentation for a given object in one call.
Deletes all currently-valid rows keyed on object_name from:
FS_BUSINESS_DICTIONARY_OBJECTS— object-level descriptionFS_BUSINESS_DICTIONARY_SECTIONS— structured sections (OVERVIEW, ENTITY, …)FS_BUSINESS_DICTIONARY_COLUMNS— column / feature descriptions
Designed to be called alongside DROP VIEW / DROP TABLE for any
documented object: dataset views, process views, source tables, etc.
Failures on individual tables are non-fatal warnings so a partial cleanup
does not abort the caller.
Parameters¶
object_name : str The OBJECT_NAME / TABLE_NAME to remove (leading/trailing quotes are stripped automatically). database_name : str, optional Restrict deletion to this database. When omitted, all databases are affected (useful when the database is unknown or may vary).
Strip outer quotes from a graph node name, return (database_upper, object_upper).
Handles: '"DB"."OBJ"', '
Return a set of (DATABASE_UPPER, TABLE_UPPER) for all physical feature storage tables.
Return True for internal feature store objects that should be silently skipped.
Tag view nodes in-place as feature store views ('fs_type': 'F') if they appear in FS_V_PROCESS_CATALOG. Adds 'fs_process_id' for matched nodes. Uses a single batch query for all view nodes.
Return node names bottom-up: leaves (base tables, no dependencies) first, root last. Edges are (parent, child) — parent depends on child. Kahn's on the reversed direction gives leaves first.
Return direct dependencies (children) of node_name in the graph.
Look up documentation for (db, obj) in FS_BUSINESS_DICTIONARY_OBJECTS and FS_BUSINESS_DICTIONARY_COLUMNS. Returns None if no object-level entry found.
Format upstream context as a structured text block for the LLM prompt. Returns empty string when context_map is empty so the prompt stays clean.
Look up stored sections for (db, obj) in FS_BUSINESS_DICTIONARY_SECTIONS. Returns a dict mapping section_name -> section_content, or empty dict.
Return ordered column names for a view or table via DBC.ColumnsV.
_store_intermediate_view_doc(db: str, obj: str, object_description: str, column_docs: Dict[str, str]) -> None
Persist generated documentation for an intermediate view to the business dictionary.
Human-written parts (IMPORTED / MANUAL provenance) are left as they are.
_document_intermediate_view(node_name: str, node_meta: Dict[str, Any], graph: Dict[str, Any], language: str = 'English', entity_columns: List[str] = []) -> None
Generate and store documentation for a non-feature-store intermediate view, using its direct children's already-resolved context as upstream anchors.
document_process_incremental(process_id: str, language: str = 'English', json_constraint: bool = True, show_sql_query: bool = False, show_explain_plan: bool = False, display: bool = True, upload: bool = True, force_update: bool = False, _visited_processes: Optional[set] = None, _visited_views: Optional[set] = None, explain: bool = True, overwrite_manual: bool = False) -> Optional[Dict[str, Any]]
Generate documentation for a process incrementally and bottom-up.
Human-written parts (imported or edited by a person, see
:func:tdfs4ds.set_feature_documentation) are kept and given to the LLM as
authoritative context; overwrite_manual=True lets the LLM replace them.
Traverses the full dependency graph of the process SQL. At each level: - Internal feature store tables (in FS_FEATURE_CATALOG or _HIDDEN suffix) are silently skipped. - Base tables with no business dictionary entry trigger a warning; the process continues best-effort without their context. - Regular intermediate views with no documentation are automatically documented (their own sources resolved first before proceeding upward). - Feature store views (type 'F' in FS_BUSINESS_DICTIONARY_OBJECTS, or detected via FS_V_PROCESS_CATALOG) are treated as leaves; if undocumented, a warning is emitted and they are omitted from context.
The current process is documented last, using the fully resolved upstream context from its direct source objects as ground truth anchors for the LLM.
Parameters¶
process_id : str UUID of the process to document. language : str Output language for LLM-generated descriptions. json_constraint : bool Whether to enforce JSON schema in the LLM output. show_sql_query : bool Include original SQL in the display output. show_explain_plan : bool Include raw EXPLAIN plan in the display output. display : bool Whether to print/render the documentation. upload : bool Whether to persist the documentation to the catalog tables. force_update : bool If True, regenerate documentation even for already-documented nodes. _visited_processes : set, optional Internal recursion guard — do not pass manually. _visited_views : set, optional Internal recursion guard — do not pass manually. explain : bool Run the LLM analysis of the view's EXPLAIN plan (scores, warnings, recommendations) and store it. False skips it and leaves any stored EXPLAIN analysis unchanged. Default True.
Returns¶
dict or None Same structure as document_process().
cascade_redocumentation(process_id: str, language: str = 'English', overwrite_manual: bool = False) -> List[str]
Re-document every downstream process/dataset that depends on process_id.
Walks the persisted lineage graph (FS_LINEAGE_GRAPH) in-memory via
networkx to find the descendants of the process's view, resolves each
descendant object to a registered process or dataset, and re-runs
document_process_incremental / document_dataset_incremental with
force_update=True. A single shared _visited_processes /
_visited_views set is threaded across all descendant calls so shared
upstream views are documented at most once (avoiding an LLM-cost explosion).
Intended to be called only when the parent's documentation actually changed
(the _doc_changed gate in upload_features). Objects in the
descendant path that are not registered processes/datasets (plain
intermediate views) are skipped as documentation targets — they are
re-documented as a side effect of their owning process's walk.
Parameters¶
process_id : str The process whose documentation just changed. language : str Output language passed through to the re-documentation calls.
Returns¶
list[str] Identifiers (process_id / dataset_id) that were re-documented.
Human-written parts of the descendants are kept unless overwrite_manual.
Parse a dataset view DDL and return a mapping of short feature name (upper) to PROCESS_ID.
Dataset column aliases use a qualified format to disambiguate the same feature appearing from multiple processes: B1.FEATURE_VALUE AS {short_name}__{database}___{view_name}_
This function strips the qualifier to return the short business name so it matches what retrieve_documentation() returns from the feature catalog. When the same short name appears in multiple process subqueries, the first encountered process wins (any documented process provides the same description).
The dataset's columns not named like their feature (offsets, aliases). Never raises.
The base name comes from the feature catalog by (storage database,
FEATURE_ID) when it can be read, else (offsets only) from the alias
without its suffix. view is the process view computing each version.
_apply_offset_descriptions(propagated: Dict[str, Any], offset_cols, feature_process_map, language) -> None
Give each offset column its base description with the instant it reads; in place.
A base name that is not itself a column of the dataset (only its lags were asked for) is removed again: no description for a column that does not exist.
Resolve a dataset_id to its metadata: name, database, DDL, entity columns, and feature-to-process_id mapping.
Returns None if the dataset is not found in the catalog.
{process_id: {COLUMN_UPPER: description}}: the business-dictionary
descriptions of each process view's columns.
The second home of a feature's text (see :func:upload_documentation, which
mirrors it there). It is the fallback when FS_PROCESS_DOCUMENTATION_FEATURES
has no row: a first upload_features documents the process before its first
run registers the features, so the feature MERGE (joined on the feature
catalog) used to write nothing. Two reads; fail-open ({}).
Fill FS_PROCESS_DOCUMENTATION_FEATURES for the features of process_id
that have no row yet, from the descriptions of its view's columns in the
business dictionary. No LLM. Returns the number of rows written (0 when
nothing was missing, -1 when it could not run).
upload_features documents a new process before its first run registers
the features, and the feature MERGE of :func:upload_documentation joins
the feature catalog: it wrote nothing then. The text did reach the business
dictionary (the view's columns). upload_features calls this after the
run; it can be called on any process to repair one documented that way.
Rows that exist are never touched (an existing description, LLM or human, wins).
_propagate_docs_from_processes(feature_process_map: Dict[str, str], entity_columns: List[str]) -> Dict[str, Any]
Propagate feature and entity documentation from process docs.
For each feature, retrieves the process documentation and copies the matching feature description. Also collects entity descriptions and process-level business logic descriptions.
Returns a dict with keys
- documented_feature_columns: {feature_name: description}
- documented_entity_columns: {entity_col: description}
- entity_description: str
- process_descriptions: {process_id: business_logic_description}
- missing_features: [feature_names with no doc found]
_generate_dataset_description(dataset_name: str, entity_description: str, entity_column_docs: Dict[str, str], feature_docs: Dict[str, str], process_descriptions: Dict[str, str], language: str = 'English') -> Dict[str, str]
Generate a structured business description for a dataset using a single LLM call with JSON-constrained output.
Returns a dict with one key per section, each independently storable as
a RAG document in FS_BUSINESS_DICTIONARY_SECTIONS:
- OVERVIEW — what the dataset represents, at what grain, primary purpose
- ENTITY — business meaning of each entity column
- FEATURE_THEMES — features grouped by logical themes
- BUSINESS_QUESTIONS — 3-5 concrete questions the dataset answers
- INTENDED_AUDIENCE — who benefits and in what workflows
Returns¶
dict[str, str] Mapping of section name to section content. Empty dict on LLM failure.
A section's content as text. A string is kept as it is.
A model asked to "group the features into themes" may answer with a list or an object; it is written out line by line instead of being lost.
_missing_dataset_sections(missing, prompt, json_schema, *, language, dataset_name, entity_description, entity_cols_text, features_text, processes_text, parse) -> Dict[str, str]
Ask once more for the dataset sections a reply left out. {} on any failure.
Same prompt and context as the first call, plus one line naming the missing keys, and a schema restricted to them.
_dataset_section_as_text(key, prompt, *, language, dataset_name, entity_description, entity_cols_text, features_text, processes_text) -> str
One dataset section asked as plain text, no structured output. '' on failure.
A reply that is still JSON (the system prompt asks for it) is read for key.
_store_dataset_doc(db: str, obj: str, sections: Dict[str, str], column_docs: Dict[str, str]) -> None
Persist generated documentation for a dataset to the business dictionary.
Stores: - Object-level entry in FS_BUSINESS_DICTIONARY_OBJECTS (type 'D') with the OVERVIEW section as BUSINESS_DESCRIPTION. - One row per section in FS_BUSINESS_DICTIONARY_SECTIONS. - One row per column in FS_BUSINESS_DICTIONARY_COLUMNS.
Human-written parts (IMPORTED / MANUAL provenance: the description, a section, a column) are left as they are (0.3.2.1).
_print_dataset_documentation(dataset_name: str, dataset_database: str, dataset_id: str, sections: Dict[str, str], documented_entity_columns: Dict[str, str], documented_feature_columns: Dict[str, str]) -> None
Pretty-print dataset documentation with context-aware formatting.
document_dataset_incremental(dataset_id: str, language: str = 'English', display: bool = True, upload: bool = True, force_update: bool = False, _visited_processes: Optional[Set[str]] = None, _visited_views: Optional[Set[str]] = None, overwrite_manual: bool = False) -> Optional[Dict[str, Any]]
Generate documentation for a dataset incrementally and bottom-up.
Human-written parts (imported, or edited with
:func:tdfs4ds.set_dataset_documentation) are kept; overwrite_manual=True
lets the LLM replace them.
Walks the full lineage graph from the dataset view through its contributing process views down to source tables. At each level:
- Internal feature store tables (FS_T_*, _HIDDEN) are silently skipped.
- Base tables with no business dictionary entry trigger a warning.
- Intermediate views with no documentation are auto-documented via LLM.
- Process views (feature store type 'F') with no documentation are actively
documented via
document_process_incremental. - The dataset itself is documented last: feature and entity column descriptions are propagated from process documentation (no LLM needed for 1:1 columns), and a single LLM call generates the dataset-level business description.
Documentation is stored in the business dictionary tables
(FS_BUSINESS_DICTIONARY_OBJECTS with type 'D',
FS_BUSINESS_DICTIONARY_COLUMNS).
Parameters¶
dataset_id : str
UUID of the dataset (from dataset_catalog()).
language : str
Output language for LLM-generated descriptions.
display : bool
Whether to render the documentation.
upload : bool
Whether to persist the documentation to the business dictionary.
force_update : bool
If True, regenerate documentation even for already-documented nodes.
_visited_processes : set, optional
Shared recursion guard from an outer orchestration call — do not pass manually.
When provided, processes already in this set are skipped without re-generating docs.
_visited_views : set, optional
Shared recursion guard from an outer orchestration call — do not pass manually.
Returns¶
dict or None A dictionary with the dataset documentation, or None on error.
_document_from_graph(graph: Dict[str, Any], proc_df, ds_df, *, all_process_ids: Set[str], all_dataset_ids: Set[str], language: str, json_constraint: bool, display: bool, upload: bool, force_update: bool, log_prefix: str = '_document_from_graph') -> Dict[str, List[str]]
Walk a dependency graph leaves-first and document each registered process / dataset exactly once, sharing visited-sets so common upstream views are documented only once.
Shared back-end of :func:document_feature_store_incremental (on-the-fly
combined graph) and :func:rebuild_documentation (persisted graph). Mirrors
the original steps 2 (lookup maps), 4–5 (topo + walk) and 6 (fallback).
graph is the {'nodes','edges','roots'} dict; proc_df / ds_df
are the process and dataset catalogs; all_process_ids / all_dataset_ids
bound the fallback pass.
rebuild_documentation(objects=None, force_update: bool = False, all_data_domains: bool = True, language: str = 'English', json_constraint: bool = True, display: bool = False, upload: bool = True, rebuild_lineage: bool = True, rebuild_vector_indexing: bool = True, extra_views=None, overwrite_manual: bool = False) -> Dict[str, List[str]]
Document the feature store from the persisted lineage graph
(FS_LINEAGE_GRAPH) — no on-the-fly SHOW VIEW re-derivation.
The persisted graph is read once (get_persisted_graph), topologically
sorted (leaves first), and walked with shared visited-sets so each process /
dataset is documented exactly once and shared upstream views are never
re-documented. Seeds are effectively the top consumers of the graph (datasets
and process views); base tables / intermediate views / tdstone2 TDS_*
objects are connectors, documented only as a side effect of their owning
process's walk.
Run :func:tdfs4ds.rebuild_lineage_graph first to populate the graph, or set
rebuild_lineage=True (the default) to have this function call it
automatically.
Scoped to changed objects (objects=[...]): re-document only the impact
set — each named object plus everything downstream of it
(get_descendants) — because a changed (possibly non-process) view
invalidates the upstream context of everything that consumes it. Names are
validated by :func:resolve_objects (case / quoting / db.name safe; raises on
any unmatched name). Scoped mode forces force_update=True so the stale
descendants actually regenerate.
Parameters¶
objects : str | Iterable[str] | None
When None (default), document the whole store. Otherwise re-document only
the impact set of these changed objects.
force_update : bool
Whole-store mode: when False (default) already-documented objects are
reused as context and only gaps get an LLM call; True regenerates every
object. (Scoped mode always forces True.)
all_data_domains : bool
Include processes from all DATA_DOMAINs (default) or only the active one.
rebuild_lineage : bool
When True (default), calls :func:tdfs4ds.rebuild_lineage_graph before
the documentation walk so the persisted graph is up to date. Set to False
to skip (e.g. you already called rebuild_lineage_graph() just before).
A warning is emitted when False.
rebuild_vector_indexing : bool
When True (default), calls :func:tdfs4ds.rebuild_vector_indexing after
the documentation walk so the vector index reflects the freshly written
business dictionary rows. Set to False to skip (e.g. you will call it
separately or want to defer indexing). A warning is emitted when False.
extra_views : str | Iterable[str] | None
Standalone view name(s) (not a process view / dataset) to also capture +
document — e.g. a hand-built reporting view nothing registered references.
Passed to rebuild_lineage_graph(extra_views=…) so the view enters
FS_LINEAGE_GRAPH and then appears as a root for the documentation walk
(documented via the intermediate-view path). Only effective when
rebuild_lineage=True. Validated (raises on unmatched name); no schema
scan.
language, json_constraint, display, upload :
Passed through to document_process_incremental / document_dataset_incremental.
overwrite_manual : bool
False (default): parts written or imported by a person are kept; the
LLM only writes the rest. True lets it replace them too.
Returns¶
dict
Summary with processes_documented / datasets_documented /
processes_skipped / datasets_skipped (lists of IDs).
Rebuild the vector index from the current business dictionary.
On the Chroma backend, a full rebuild (force_rebuild=True, the default)
destroys and re-creates all three collections so removed or renamed
documentation does not linger.
On the Teradata backend, force_rebuild=True destroys and recreates each
collection (since 0.3.1.167). destroy() is admin-only: when it is
refused, the call falls back to refreshing the existing collection with a
warning. force_rebuild=False refreshes an existing collection
(update(data_refresh=True), else destroy + recreate) and creates a
missing one — see
tdfs4ds.agent.vector_store_teradata.build_teradata_index's docstring.
Use force_rebuild=False for an incremental update on either backend
(on Chroma it embeds only missing or changed rows).
Works with both Chroma and Teradata backends
(tdfs4ds.VECTOR_STORE_BACKEND).
document_feature_store_incremental(language: str = 'English', json_constraint: bool = True, display: bool = True, upload: bool = True, force_update: bool = False, all_data_domains: bool = False) -> Dict[str, Any]
Document all registered processes and datasets in a single optimised pass.
Objects are processed in dependency order — leaves (deepest upstream dependencies) first, roots last — so that upstream context is always available when a higher-level process or dataset is reached. A single pair of visited-sets is shared across every call so each process view is documented at most once, regardless of how many datasets reference it.
Strategy¶
- Fetch all registered processes (
process_catalog) and datasets (dataset_catalog). - Build a combined lineage graph seeded from every dataset DDL
(
expand_datasets_via_process_catalog=True) and from any orphan process not already covered by a dataset. - Topologically sort the combined graph (leaves first, roots last).
- Walk the sorted order, calling
document_process_incrementalfor registered process views anddocument_dataset_incrementalfor dataset views, sharing the same_visited_processes/_visited_viewssets throughout. - Any registered object absent from the graph (e.g. if its lineage build failed) is documented individually as a fallback.
Parameters¶
language : str
Output language for LLM-generated descriptions.
json_constraint : bool
Whether to enforce JSON schema in the LLM output.
display : bool
Whether to render each object's documentation as it is produced.
upload : bool
Whether to persist the documentation to the catalog tables.
force_update : bool
If True, regenerate documentation even for already-documented objects.
all_data_domains : bool
If True, include processes from all DATA_DOMAINs; otherwise only the
current tdfs4ds.DATA_DOMAIN is included.
Returns¶
dict
Summary with keys "processes_documented", "datasets_documented",
"processes_skipped", "datasets_skipped" (each a list of IDs).
document_view_incremental(database: str, view_name: str, language: str = 'English', force_update: bool = False, display: bool = True, upload: bool = True, _visited_views: Optional[set] = None) -> None
Document an arbitrary view and all its upstream dependencies.
Mirrors the inner loop of document_process_incremental but seeded from
any SELECT * FROM {database}.{view_name} — the view does not need to be
registered in the process or dataset catalog.
Useful for custom analytical views, reporting views, or any intermediate objects that are not tracked by the feature store catalogs.
A shared _visited_views set can be passed when calling this function
in a loop so that views referenced by multiple seeds are only documented once.
Parameters¶
database : str
Database (schema) that owns the view.
view_name : str
Name of the view to document.
language : str
Output language for LLM-generated descriptions.
force_update : bool
If True, regenerate docs even if the view is already documented.
display : bool
Whether to emit INFO log lines when done.
upload : bool
Whether to persist docs to the catalog tables (passed through to
the underlying calls — currently informational, upload happens inside
_document_intermediate_view).
_visited_views : set, optional
Cross-call deduplication guard. Pass the same set when calling this
function for multiple views in a single loop so shared dependencies
are only documented once.