tdfs4ds.genai¶
module-attribute
¶
__all__ = ['set_feature_documentation', 'set_process_documentation', 'set_dataset_documentation', 'set_object_documentation', 'set_deep_table_documentation', 'deep_table_tags', 'propagate_documentation', 'get_documentation', 'mark_documentation_manual', 'release_manual_documentation', 'documentation_log', 'documentation_tracking_creation', 'document_sql_query_columns', 'document_process', 'document_process_incremental', 'document_dataset_incremental', 'document_feature_store_incremental', 'rebuild_documentation', 'document_view_incremental', 'documentation_tables_creation', 'documentation_business_dictionary_view_creation', 'migrate_documentation_explain_schema', 'migrate_documentation_columns_schema', 'document_sql_query_explain', 'build_explain_documentation_chain', 'run_explain_documentation', 'build_sql_documentation_chain', 'run_sql_documentation', 'build_llm', 'get_the_explain', 'display_process_info', 'feed_process_info_with_prompt_result', 'upload_business_dictionary_objects', 'upload_business_dictionary_columns', 'upload_business_dictionary_sections', 'delete_business_dictionary_objects', 'delete_business_dictionary_columns', 'delete_business_dictionary_sections', 'delete_process_documentation', 'remove_object_documentation', 'repair_feature_documentation_entities']
document_sql_query_columns(sql_query: str, entity_columns: Sequence[str], feature_columns: Sequence[str], language: str = 'English', provider: Optional[str] = None, json_constraint: bool = True, prompt_only: bool = False, upstream_context: str = '') -> Dict[str, Any]
Convenience function to generate business-focused documentation for SQL query output columns using the configured instruction model from tdfs4ds settings.
This function automatically builds the LLM client using the tdfs4ds configuration variables (INSTRUCT_MODEL_URL, INSTRUCT_MODEL_API_KEY, INSTRUCT_MODEL_MODEL), constructs the documentation chain, and executes it to produce column descriptions.
Parameters¶
sql_query : str The SQL query whose resulting columns should be documented. This query is shown to the model so it can infer business logic, derivation rules, and column meaning.
Sequence[str]
The list of entity/identifier column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
Sequence[str]
The list of feature column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
str, optional (default="English")
The target output language for the generated documentation. This value is passed into the prompt's {language} variable. Examples: "English", "French", "German", "Spanish", "Japanese".
str, optional (default=None)
Indicates which structured-output mechanism to use for the LLM.
If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
bool, optional (default=True)
If True: - a JSON Schema is generated from the column lists - provider-specific constrained decoding is applied If False: - the chain does not enforce JSON structure at the LLM level - the model is only guided by the prompt (weaker guarantees)
bool, optional (default=False)
If True: - returns only the prompt template, without executing the chain - useful for debugging, testing, or customizing the prompt before execution If False: - executes the full chain and returns structured documentation
Returns¶
dict If prompt_only=False: A dictionary with four keys: - "query_business_logic": str containing the high-level business logic description of the query - "entity_description": str containing the holistic description of the entity - "entity_columns": dict[str, str] mapping each entity column name to its description - "feature_columns": dict[str, str] mapping each feature column name to its description If prompt_only=True: The prompt template itself, for inspection or further customization.
Raises¶
ValueError If any of the required tdfs4ds configuration variables (INSTRUCT_MODEL_URL, INSTRUCT_MODEL_API_KEY, INSTRUCT_MODEL_MODEL, INSTRUCT_MODEL_PROVIDER) are not set.
Notes¶
- This function requires that the tdfs4ds instruction model configuration is properly set.
- The resulting descriptions are typically ≤ 5 sentences per column, focusing on business meaning and logic.
- If the model fails to produce valid JSON, an exception will be raised.
document_process(process_id: str, language: str = 'English', json_constraint: bool = True, show_sql_query: bool = False, show_explain_plan: bool = False, display: bool = True, upload: bool = True, prompt_only=False) -> Optional[Dict[str, Any]]
Generate and store documentation for a data process identified by process_id. This function retrieves the SQL query and output columns for the process, generates business-focused documentation using an LLM, and stores the results in the appropriate documentation tables.
Parameters¶
process_id : str The unique identifier of the data process to document.
str, optional (default="English")
The target output language for the generated documentation. This value is
passed into the prompt’s {language} variable. Examples: "English", "French", "German", "Spanish", "Japanese".
provider : str, optional (default=None)
Indicates which structured-output mechanism to use for the LLM.
If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True:
- a JSON Schema is generated from the column list
- provider-specific constrained decoding is applied
If False:
- the chain does not enforce JSON structure at the LLM level
- the model is only guided by the prompt (weaker guarantees)
show_sql_query : bool, optional (default=False)
If True, display the original SQL query at the end of the documentation report.
show_explain_plan : bool, optional (default=False)
If True, display the raw EXPLAIN plan output at the end of the documentation report.
display : bool, optional (default=True)
If True, print the generated documentation to the console.
upload_documentation : bool, optional (default=True)
If True, upload the generated documentation to the documentation tables.
Returns¶
dict or None A dictionary containing the generated documentation and analysis, or None if an error occurred. The dictionary includes keys: - PROCESS_ID - DOCUMENTED_SQL - ENTITY_DESCRIPTION - DOCUMENTED_ENTITY_COLUMNS - DOCUMENTED_FEATURE_COLUMNS - EXPLAIN_ANALYSIS (if show_explain_plan is True) - OPTIMIZATION_SCORE (if show_explain_plan is True) - EXPLAIN_WARNINGS (if show_explain_plan is True) - EXPLAIN_RECOMMENDATIONS (if show_explain_plan is True) - RAW_EXPLAIN_PLAN (if show_explain_plan is True) Notes
- This function requires that the tdfs4ds instruction model configuration is properly set.
- If the model fails to produce valid JSON, an exception will be raised.
- The resulting descriptions are typically ≤ 5 sentences per column, focusing on business meaning and logic.
document_process_incremental(process_id: str, language: str = 'English', json_constraint: bool = True, show_sql_query: bool = False, show_explain_plan: bool = False, display: bool = True, upload: bool = True, force_update: bool = False, _visited_processes: Optional[set] = None, _visited_views: Optional[set] = None, explain: bool = True, overwrite_manual: bool = False) -> Optional[Dict[str, Any]]
Generate documentation for a process incrementally and bottom-up.
Human-written parts (imported or edited by a person, see
:func:tdfs4ds.set_feature_documentation) are kept and given to the LLM as
authoritative context; overwrite_manual=True lets the LLM replace them.
Traverses the full dependency graph of the process SQL. At each level: - Internal feature store tables (in FS_FEATURE_CATALOG or _HIDDEN suffix) are silently skipped. - Base tables with no business dictionary entry trigger a warning; the process continues best-effort without their context. - Regular intermediate views with no documentation are automatically documented (their own sources resolved first before proceeding upward). - Feature store views (type 'F' in FS_BUSINESS_DICTIONARY_OBJECTS, or detected via FS_V_PROCESS_CATALOG) are treated as leaves; if undocumented, a warning is emitted and they are omitted from context.
The current process is documented last, using the fully resolved upstream context from its direct source objects as ground truth anchors for the LLM.
Parameters¶
process_id : str UUID of the process to document. language : str Output language for LLM-generated descriptions. json_constraint : bool Whether to enforce JSON schema in the LLM output. show_sql_query : bool Include original SQL in the display output. show_explain_plan : bool Include raw EXPLAIN plan in the display output. display : bool Whether to print/render the documentation. upload : bool Whether to persist the documentation to the catalog tables. force_update : bool If True, regenerate documentation even for already-documented nodes. _visited_processes : set, optional Internal recursion guard — do not pass manually. _visited_views : set, optional Internal recursion guard — do not pass manually. explain : bool Run the LLM analysis of the view's EXPLAIN plan (scores, warnings, recommendations) and store it. False skips it and leaves any stored EXPLAIN analysis unchanged. Default True.
Returns¶
dict or None Same structure as document_process().
document_dataset_incremental(dataset_id: str, language: str = 'English', display: bool = True, upload: bool = True, force_update: bool = False, _visited_processes: Optional[Set[str]] = None, _visited_views: Optional[Set[str]] = None, overwrite_manual: bool = False) -> Optional[Dict[str, Any]]
Generate documentation for a dataset incrementally and bottom-up.
Human-written parts (imported, or edited with
:func:tdfs4ds.set_dataset_documentation) are kept; overwrite_manual=True
lets the LLM replace them.
Walks the full lineage graph from the dataset view through its contributing process views down to source tables. At each level:
- Internal feature store tables (FS_T_*, _HIDDEN) are silently skipped.
- Base tables with no business dictionary entry trigger a warning.
- Intermediate views with no documentation are auto-documented via LLM.
- Process views (feature store type 'F') with no documentation are actively
documented via
document_process_incremental. - The dataset itself is documented last: feature and entity column descriptions are propagated from process documentation (no LLM needed for 1:1 columns), and a single LLM call generates the dataset-level business description.
Documentation is stored in the business dictionary tables
(FS_BUSINESS_DICTIONARY_OBJECTS with type 'D',
FS_BUSINESS_DICTIONARY_COLUMNS).
Parameters¶
dataset_id : str
UUID of the dataset (from dataset_catalog()).
language : str
Output language for LLM-generated descriptions.
display : bool
Whether to render the documentation.
upload : bool
Whether to persist the documentation to the business dictionary.
force_update : bool
If True, regenerate documentation even for already-documented nodes.
_visited_processes : set, optional
Shared recursion guard from an outer orchestration call — do not pass manually.
When provided, processes already in this set are skipped without re-generating docs.
_visited_views : set, optional
Shared recursion guard from an outer orchestration call — do not pass manually.
Returns¶
dict or None A dictionary with the dataset documentation, or None on error.
document_feature_store_incremental(language: str = 'English', json_constraint: bool = True, display: bool = True, upload: bool = True, force_update: bool = False, all_data_domains: bool = False) -> Dict[str, Any]
Document all registered processes and datasets in a single optimised pass.
Objects are processed in dependency order — leaves (deepest upstream dependencies) first, roots last — so that upstream context is always available when a higher-level process or dataset is reached. A single pair of visited-sets is shared across every call so each process view is documented at most once, regardless of how many datasets reference it.
Strategy¶
- Fetch all registered processes (
process_catalog) and datasets (dataset_catalog). - Build a combined lineage graph seeded from every dataset DDL
(
expand_datasets_via_process_catalog=True) and from any orphan process not already covered by a dataset. - Topologically sort the combined graph (leaves first, roots last).
- Walk the sorted order, calling
document_process_incrementalfor registered process views anddocument_dataset_incrementalfor dataset views, sharing the same_visited_processes/_visited_viewssets throughout. - Any registered object absent from the graph (e.g. if its lineage build failed) is documented individually as a fallback.
Parameters¶
language : str
Output language for LLM-generated descriptions.
json_constraint : bool
Whether to enforce JSON schema in the LLM output.
display : bool
Whether to render each object's documentation as it is produced.
upload : bool
Whether to persist the documentation to the catalog tables.
force_update : bool
If True, regenerate documentation even for already-documented objects.
all_data_domains : bool
If True, include processes from all DATA_DOMAINs; otherwise only the
current tdfs4ds.DATA_DOMAIN is included.
Returns¶
dict
Summary with keys "processes_documented", "datasets_documented",
"processes_skipped", "datasets_skipped" (each a list of IDs).
rebuild_documentation(objects=None, force_update: bool = False, all_data_domains: bool = True, language: str = 'English', json_constraint: bool = True, display: bool = False, upload: bool = True, rebuild_lineage: bool = True, rebuild_vector_indexing: bool = True, extra_views=None, overwrite_manual: bool = False) -> Dict[str, List[str]]
Document the feature store from the persisted lineage graph
(FS_LINEAGE_GRAPH) — no on-the-fly SHOW VIEW re-derivation.
The persisted graph is read once (get_persisted_graph), topologically
sorted (leaves first), and walked with shared visited-sets so each process /
dataset is documented exactly once and shared upstream views are never
re-documented. Seeds are effectively the top consumers of the graph (datasets
and process views); base tables / intermediate views / tdstone2 TDS_*
objects are connectors, documented only as a side effect of their owning
process's walk.
Run :func:tdfs4ds.rebuild_lineage_graph first to populate the graph, or set
rebuild_lineage=True (the default) to have this function call it
automatically.
Scoped to changed objects (objects=[...]): re-document only the impact
set — each named object plus everything downstream of it
(get_descendants) — because a changed (possibly non-process) view
invalidates the upstream context of everything that consumes it. Names are
validated by :func:resolve_objects (case / quoting / db.name safe; raises on
any unmatched name). Scoped mode forces force_update=True so the stale
descendants actually regenerate.
Parameters¶
objects : str | Iterable[str] | None
When None (default), document the whole store. Otherwise re-document only
the impact set of these changed objects.
force_update : bool
Whole-store mode: when False (default) already-documented objects are
reused as context and only gaps get an LLM call; True regenerates every
object. (Scoped mode always forces True.)
all_data_domains : bool
Include processes from all DATA_DOMAINs (default) or only the active one.
rebuild_lineage : bool
When True (default), calls :func:tdfs4ds.rebuild_lineage_graph before
the documentation walk so the persisted graph is up to date. Set to False
to skip (e.g. you already called rebuild_lineage_graph() just before).
A warning is emitted when False.
rebuild_vector_indexing : bool
When True (default), calls :func:tdfs4ds.rebuild_vector_indexing after
the documentation walk so the vector index reflects the freshly written
business dictionary rows. Set to False to skip (e.g. you will call it
separately or want to defer indexing). A warning is emitted when False.
extra_views : str | Iterable[str] | None
Standalone view name(s) (not a process view / dataset) to also capture +
document — e.g. a hand-built reporting view nothing registered references.
Passed to rebuild_lineage_graph(extra_views=…) so the view enters
FS_LINEAGE_GRAPH and then appears as a root for the documentation walk
(documented via the intermediate-view path). Only effective when
rebuild_lineage=True. Validated (raises on unmatched name); no schema
scan.
language, json_constraint, display, upload :
Passed through to document_process_incremental / document_dataset_incremental.
overwrite_manual : bool
False (default): parts written or imported by a person are kept; the
LLM only writes the rest. True lets it replace them too.
Returns¶
dict
Summary with processes_documented / datasets_documented /
processes_skipped / datasets_skipped (lists of IDs).
document_view_incremental(database: str, view_name: str, language: str = 'English', force_update: bool = False, display: bool = True, upload: bool = True, _visited_views: Optional[set] = None) -> None
Document an arbitrary view and all its upstream dependencies.
Mirrors the inner loop of document_process_incremental but seeded from
any SELECT * FROM {database}.{view_name} — the view does not need to be
registered in the process or dataset catalog.
Useful for custom analytical views, reporting views, or any intermediate objects that are not tracked by the feature store catalogs.
A shared _visited_views set can be passed when calling this function
in a loop so that views referenced by multiple seeds are only documented once.
Parameters¶
database : str
Database (schema) that owns the view.
view_name : str
Name of the view to document.
language : str
Output language for LLM-generated descriptions.
force_update : bool
If True, regenerate docs even if the view is already documented.
display : bool
Whether to emit INFO log lines when done.
upload : bool
Whether to persist docs to the catalog tables (passed through to
the underlying calls — currently informational, upload happens inside
_document_intermediate_view).
_visited_views : set, optional
Cross-call deduplication guard. Pass the same set when calling this
function for multiple views in a single loop so shared dependencies
are only documented once.
Create the necessary documentation tables in the database if they do not already exist. tdml: The tdfs4ds TDML connection object.
(Re)create the CURRENT VALIDTIME + LOCKING ROW FOR ACCESS views over the three
business dictionary tables — used by the Teradata Enterprise Vector Store backend
(vector_store_teradata.py) so a permanent teradataml.DataFrame can be passed
directly to from_datasets()/add_datasets(), instead of routing through
from_documents()/add_documents()'s Document-conversion path, whose internal
staging call (DataFrame.from_dict(persist=True)) has no database argument
and may materialize as a volatile table invisible to the Vector Store's
separate REST session. REPLACE VIEW keeps this idempotent across re-runs,
so it is safe (and intended) to call unconditionally on every connect()/setup(),
mirroring feature_store_catalog_view_creation()/process_store_catalog_view_creation()
— this avoids depending on a table-existence probe to decide whether the views
need (re)creating.
ID mirrors the doc-id convention in agent/vector_index.py's _objects_to_docs /
_sections_to_docs / _columns_to_docs (f"{db}__{obj}" etc.) so switching between
the Chroma and Teradata backends keeps stable, comparable identifiers. The
blank/NULL-description filter mirrors those functions' if not text.strip():
continue guard, pushed into SQL so the view itself excludes rows not worth
embedding.
Ensure the FS_PROCESS_DOCUMENTATION_EXPLAIN table has all expected columns.
Check-first: inspects the live DDL and only runs ALTER TABLE for columns that are actually missing. No-op (and no log at INFO) when already up-to-date.
Columns added over time: - USER_SCORE (v0.2.x) — deterministic score from [You] items - N_STEPS (v0.2.9) — EXPLAIN step count - N_SPOOL_OBJECTS (v0.2.9) — distinct Spool object count
New columns start as NULL for existing rows and are filled on the next
document_process() or document_process_incremental() run.
Ensure FS_BUSINESS_DICTIONARY_COLUMNS has the interpretation-contract columns.
Check-first: inspects live DDL and only runs ALTER TABLE for columns that are actually missing. No-op (and no log at INFO) when already up-to-date.
Columns added over time: - MONOTONIC_DIRECTION (v0.3.1.35) — direction hint derived from formula - VALUE_DOMAIN (v0.3.1.35) — value range hint - IS_ABSOLUTE_VALUE (v0.3.1.35) — True when ABS() detected in formula - FORMULA_CONSISTENCY_WARNING (v0.3.1.35) — set when LLM violated hint after retry
New columns start as NULL for existing rows.
document_sql_query_explain(sql_query: str, provider: Optional[str] = None, json_constraint: bool = True, prompt_only: bool = False, classify: bool = True) -> Dict[str, Any]
Analyze a SQL query's EXPLAIN plan and return optimization recommendations.
This function automatically builds the LLM client using tdfs4ds configuration, constructs the EXPLAIN analysis chain, and executes it.
Parameters¶
sql_query : str The original SQL query. explain_plan : str The EXPLAIN output from the database. provider : str, optional (default=None) Indicates which structured-output mechanism to use for the LLM. If None, uses INSTRUCT_MODEL_PROVIDER from tdfs4ds config. Supported values: "vllm", "openai", "azure", "ollama", etc. json_constraint : bool, optional (default=True) If True: use provider-specific constrained decoding. If False: rely on prompt guidance only.
Returns¶
dict A dictionary with keys: - "explanation": str describing the EXPLAIN plan - "optimization_score": int from 1 to 5 - "warnings": list[str] of potential issues - "recommendations": list[str] of actionable suggestions
Raises¶
ValueError If tdfs4ds instruction model configuration is incomplete.
build_explain_documentation_chain(llm: ChatOpenAI, provider: str = 'vllm', json_constraint: bool = True, prompt_only: bool = False) -> Runnable
Build a LangChain Runnable that analyzes SQL EXPLAIN plans and generates optimization scores, warnings, and recommendations.
The resulting chain expects two input variables
- sql_query: str → the original SQL query
- explain_plan: str → the EXPLAIN output from the database
Parameters¶
llm : ChatOpenAI
The language model interface (may point to vLLM, OpenAI, Ollama, etc.).
provider : str, optional (default="vllm")
Indicates which structured-output mechanism to use.
Supported values:
- "vllm" → uses guided_json for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via response_format
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via response_format (NOT
vLLM guided_json; that is a vLLM-only extension)
- "ollama" → uses Ollama's format= schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True: a JSON Schema is generated and provider-specific constrained decoding is applied.
If False: the chain does not enforce JSON structure at the LLM level.
Returns¶
Runnable A LangChain Runnable that executes: prompt → LLM (optionally schema-guided) → JSON parser
When invoked with:
{
"sql_query": "SELECT ...",
"explain_plan": "..."
}
It returns:
dict with keys:
- "explanation": str describing the EXPLAIN plan in business terms
- "optimization_score": int from 1 (poorly optimized) to 5 (well optimized)
- "warnings": list[str] of potential issues or concerns
- "recommendations": list[str] of actionable optimization suggestions
run_explain_documentation(chain: Runnable, sql_query: str, explain_plan: str, classify: bool = True) -> Dict[str, Any]
Execute an EXPLAIN-documentation chain and return optimization analysis.
Parameters¶
chain : Runnable
A LangChain Runnable returned by build_explain_documentation_chain().
sql_query : str
The original SQL query.
explain_plan : str
The EXPLAIN output from the database.
classify : bool, default True
Pass through to _compute_user_score_from_you_items. Set to False
during intermediate scoring (simplification, candidate evaluation)
to keep the raw [You] count and avoid filtering useful structural signals.
Returns¶
dict A dictionary with keys: "explanation", "optimization_score", "warnings", "recommendations"
build_sql_documentation_chain(llm: ChatOpenAI, entity_columns: Sequence[str], feature_columns: Sequence[str], provider: str = 'vllm', json_constraint: bool = True, prompt_only: bool = False) -> Runnable
Build a LangChain Runnable that generates business-focused documentation
for lists of entity and feature columns from a SQL query output, with optional provider-specific JSON
constraints (vLLM, OpenAI, Ollama, etc.).
The resulting chain expects two input variables:
- sql_query: str → the SQL query whose output is being documented
- columns_str: str → formatted list of entity and feature columns (e.g. "Entity columns:
- col1
Feature columns: - col2")
Parameters
----------
llm : ChatOpenAI
The language model interface (may point to vLLM, OpenAI, Ollama, etc.).
entity_columns : Sequence[str]
List of entity/identifier columns that must appear as keys in the output JSON.
feature_columns : Sequence[str]
List of feature columns that must appear as keys in the output JSON.
provider : str, optional (default="vllm")
Indicates which structured-output mechanism to use.
Supported values:
- "vllm" → uses `guided_json` for strict JSON output
- "openai" / "azure" → uses OpenAI JSON Schema via `response_format`
- "mistralai" / "openai-compatible"
→ OpenAI JSON Schema via `response_format` (NOT
vLLM `guided_json`; that is a vLLM-only extension)
- "ollama" → uses Ollama's `format=` schema
- any other value → fall back to unconstrained text output
json_constraint : bool, optional (default=True)
If True:
- a JSON Schema is generated from the column lists
- provider-specific constrained decoding is applied
If False:
- the chain does not enforce JSON structure at the LLM level
- the model is only guided by the prompt (weaker guarantees)
prompt_only : bool, optional (default=False)
If True:
- returns only the prompt template, without attaching the LLM or parser
- useful for debugging, testing, or customizing the prompt before execution
If False:
- returns the full chain: prompt → LLM (optionally schema-guided) → JSON parser
Returns
-------
Runnable
If prompt_only=False:
A LangChain Runnable that executes:
prompt → LLM (optionally schema-guided) → JSON parser
When invoked with:
{
"sql_query": "...",
"columns_str": "Entity columns:
- column1
Feature columns: - column2 ..." } It returns: dict[str, str] A mapping of each requested column name to a short, business-oriented description (≤ 5 sentences), plus a 'query_business_logic' key containing a high-level description of the query's business logic (5-10 sentences), and an 'entity_description' key with a holistic description of the entity (3-5 sentences). If prompt_only=True: The prompt template itself, for inspection or further customization.
Notes
-----
- The chain enforces valid JSON when possible:
* vLLM → `guided_json`
* OpenAI → `response_format={"type": "json_schema", ...}`
* Ollama → `format=<schema>`
- For unsupported providers, the model may emit imperfect JSON.
- Descriptions focus on business meaning, business logic,
and optionally technical details only when relevant.
run_sql_documentation(chain: Runnable, sql_query: str, entity_columns: Sequence[str], feature_columns: Sequence[str], language: str = 'English', upstream_context: str = '') -> Dict[str, str]
Execute a previously constructed SQL-documentation chain and return business-friendly documentation for the specified SQL output columns.
This function prepares the chain inputs (SQL query, formatted column list,
target language) and invokes the chain. The chain itself must have been
created using build_sql_documentation_chain(), which ensures the model
produces structured JSON suitable for parsing.
Parameters¶
chain : Runnable
A LangChain Runnable returned by build_sql_documentation_chain().
This Runnable encapsulates:
- the prompt template
- a provider-specific LLM invocation (with or without JSON constraints)
- a JSON output parser
str
The SQL query whose resulting columns should be documented. This query is shown to the model so it can infer business logic, derivation rules, and column meaning.
Sequence[str]
The list of entity/identifier column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
Sequence[str]
The list of feature column names that must appear as keys in the output JSON. Only these columns will be documented. The order does not matter.
str, optional (default="English")
The target output language for the generated documentation.
This value is passed into the prompt’s {language} variable.
Examples: "English", "French", "German", "Spanish", "Japanese".
Returns¶
dict[str, str] A dictionary mapping each column name to a human-readable, business-oriented description generated by the model, plus a 'query_business_logic' key with the query's business logic description, and an 'entity_description' key with the holistic entity description. Example: { "customer_id": "Unique customer identifier used for ...", "order_date": "Business date when the order was created ...", "entity_description": "The customer entity represents...", "query_business_logic": "This query provides a view of ..." }
Notes¶
- The output format is determined by the chain's JSON parser. If the model
fails to produce valid JSON (e.g., due to unsupported constraints),
a
OutputParserExceptionmay be raised. - The resulting descriptions are typically ≤ 5 sentences per column, unless modified in the chain's prompt.
build_llm(llm_service: str = None, api_key: str = None, model_id: str = None, temperature: float = 0.0, timeout: int = 120, provider: str = None)
Build and return a chat LLM client based on the configured provider.
When called with no arguments, reads from tdfs4ds global configuration: - INSTRUCT_MODEL_PROVIDER (provider) - INSTRUCT_MODEL_URL (llm_service) - INSTRUCT_MODEL_API_KEY (api_key) - INSTRUCT_MODEL_MODEL (model_id) - INSTRUCT_MODEL_REGION (AWS region for Bedrock) - INSTRUCT_MODEL_AZURE_DEPLOYMENT, INSTRUCT_MODEL_AZURE_API_VERSION (Azure) - INSTRUCT_MODEL_GCP_PROJECT, INSTRUCT_MODEL_GCP_LOCATION (GCP)
Injects tdfs4ds config variables into environment so that boto3, Azure, and Google Cloud clients pick up credentials and endpoints automatically.
Parameters¶
llm_service : str, optional Base URL of the LLM service (overrides INSTRUCT_MODEL_URL). api_key : str, optional API key for authentication (overrides INSTRUCT_MODEL_API_KEY). model_id : str, optional Model identifier (overrides INSTRUCT_MODEL_MODEL). temperature : float Sampling temperature for response diversity. timeout : int Request timeout in seconds. provider : str, optional LLM provider name (overrides INSTRUCT_MODEL_PROVIDER).
Returns¶
BaseChatModel Configured LLM client (ChatOpenAI, AzureChatOpenAI, ChatBedrockConverse, ChatVertexAI, etc.).
Raises¶
Exception If LLM client creation fails or provider is not supported.
Pretty print the documentation and EXPLAIN analysis for a data process from process_info dict or by retrieving it using process_id.
Parameters¶
process_info : dict, optional (default=None) A dictionary containing the process documentation information. If None, process_id must be provided to retrieve the information. process_id : str, optional (default=None) The unique identifier of the data process. If process_info is None, this parameter is used to retrieve the documentation.
Returns None
feed_process_info_with_prompt_result(process_info, sql_documentation_response=None, sql_explain_response=None, display_info=True, upload_info=True)
Enriches a process_info dictionary with SQL documentation and EXPLAIN plan analysis results, with options to display the results and upload the enriched information.
This function integrates the results of SQL documentation and EXPLAIN plan analysis into the provided
process_info dictionary. It extracts and organizes documentation for entity and feature columns,
as well as optimization insights, to provide a comprehensive view of the SQL query's business logic,
performance, and potential improvements. It also supports optional display of the enriched information
and automatic upload of the documentation and EXPLAIN analysis to a backend system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process_info
|
dict
|
A dictionary containing metadata about the SQL process, including: - 'ENTITY_COLUMNS': List of columns representing the entity in the SQL query. - 'FEATURE_COLUMNS': List of columns representing features in the SQL query. |
required |
sql_documentation_response
|
dict
|
A dictionary containing SQL documentation results, including descriptions for entity/feature columns and query business logic. Expected keys: - 'query_business_logic': Description of the query's purpose and logic. - 'entity_description': Description of the entity represented by the query. - Column names as keys, with their descriptions as values. |
None
|
sql_explain_response
|
dict
|
A dictionary containing SQL EXPLAIN plan analysis results, including: - 'explanation': Detailed analysis of the EXPLAIN plan. - 'optimization_score': Integer score (1-5) indicating query optimization level. - 'warnings': List of potential issues identified in the EXPLAIN plan. - 'recommendations': List of actionable recommendations for query optimization. |
None
|
display_info
|
bool
|
If True, displays the enriched process_info using |
True
|
upload_info
|
bool
|
If True, uploads the enriched documentation and EXPLAIN analysis to a backend system. Defaults to True. |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
dict |
The enriched |
Example
process_info = { ... 'ENTITY_COLUMNS': ['customer_id', 'order_id'], ... 'FEATURE_COLUMNS': ['order_amount', 'order_date'] ... } sql_documentation_response = { ... 'query_business_logic': 'This query joins customer and order data...', ... 'entity_description': 'The customer entity represents...', ... 'customer_id': 'Unique identifier for customers.', ... 'order_amount': 'Total amount of the order.' ... } sql_explain_response = { ... 'explanation': 'The EXPLAIN plan shows a nested loop join...', ... 'optimization_score': 3, ... 'warnings': ['Full table scan on orders table'], ... 'recommendations': ['Add index on orders.customer_id'] ... } enriched_info = feed_process_info_with_prompt_result( ... process_info, ... sql_documentation_response, ... sql_explain_response, ... display_info=True, ... upload_info=True ... ) print(enriched_info.keys()) ['ENTITY_COLUMNS', 'FEATURE_COLUMNS', 'DOCUMENTED_SQL', 'ENTITY_DESCRIPTION', 'DOCUMENTED_ENTITY_COLUMNS', 'DOCUMENTED_FEATURE_COLUMNS', 'EXPLAIN_ANALYSIS', 'OPTIMIZATION_SCORE', 'EXPLAIN_WARNINGS', 'EXPLAIN_RECOMMENDATIONS']
upload_business_dictionary_objects(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update object-level descriptions in the business dictionary.
source (0.3.2.1) records who wrote these rows in the provenance table:
'IMPORTED' (default: a load of an existing data dictionary, e.g. source
data documentation) or 'MANUAL'. Both are kept by later LLM
documentation runs. author is the declared editor name (default: the OS
user). The LLM pipelines call this with source=None (nothing recorded).
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the object
- OBJECT_NAME : str — table or view name
- OBJECT_TYPE : str — 'T' for table, 'V' for view
- BUSINESS_DESCRIPTION: str — business-oriented description of the object
Rows are merged into FS_BUSINESS_DICTIONARY_OBJECTS using
CURRENT VALIDTIME semantics (Teradata temporal). Existing rows
matched on (DATABASE_NAME, OBJECT_NAME) are updated; new rows are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, OBJECT_NAME, OBJECT_TYPE, BUSINESS_DESCRIPTION.
upload_business_dictionary_columns(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update column-level descriptions in the business dictionary.
source / author: see :func:upload_business_dictionary_objects.
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the table
- TABLE_NAME : str — table or view name
- COLUMN_NAME : str — column name
- BUSINESS_DESCRIPTION: str — business-oriented description of the column
Rows are merged into FS_BUSINESS_DICTIONARY_COLUMNS using
CURRENT VALIDTIME semantics (Teradata temporal). Existing rows
matched on (DATABASE_NAME, TABLE_NAME, COLUMN_NAME) are updated;
new rows are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, TABLE_NAME, COLUMN_NAME, BUSINESS_DESCRIPTION.
upload_business_dictionary_sections(df: DataFrame, *, source: Optional[str] = 'IMPORTED', author: Optional[str] = None, _event_id: Optional[str] = None) -> None
Insert or update section-level descriptions in the business dictionary.
source / author: see :func:upload_business_dictionary_objects.
The DataFrame must contain the following columns
- DATABASE_NAME : str — database (or schema) that owns the object
- OBJECT_NAME : str — object name (table, view, dataset)
- SECTION_NAME : str — section identifier (e.g. 'OVERVIEW', 'FEATURE_THEMES')
- SECTION_CONTENT: str — section text content
Rows are merged into FS_BUSINESS_DICTIONARY_SECTIONS using
CURRENT VALIDTIME semantics. Existing rows matched on
(DATABASE_NAME, OBJECT_NAME, SECTION_NAME) are updated; new rows
are inserted.
Parameters¶
df : pd.DataFrame DataFrame with columns DATABASE_NAME, OBJECT_NAME, SECTION_NAME, SECTION_CONTENT.
delete_business_dictionary_objects(object_names: list, *, database_name: Optional[str] = None) -> None
Expire currently-valid object-level entries from the business dictionary.
Parameters¶
object_names : list of str
Object names to delete (e.g. ['TRANSACTIONS', 'CUSTOMERS']).
database_name : str, optional
When provided, only rows with this DATABASE_NAME are deleted.
When omitted, all databases are affected (useful to clean up stale
entries that were uploaded under a wrong database name).
delete_business_dictionary_columns(table_name: str, *, column_names: Optional[list] = None, database_name: Optional[str] = None) -> None
Expire currently-valid column-level entries from the business dictionary.
Parameters¶
table_name : str Table whose column descriptions should be deleted. column_names : list of str, optional Specific columns to delete. When omitted all columns of the table are deleted. database_name : str, optional When provided, only rows with this DATABASE_NAME are deleted. When omitted, all databases are affected.
delete_business_dictionary_sections(object_name: str, *, section_names: Optional[list] = None, database_name: Optional[str] = None) -> None
Expire currently-valid section entries from the business dictionary.
Parameters¶
object_name : str
Object whose section entries should be deleted (OBJECT_NAME column).
section_names : list of str, optional
Specific sections to delete (e.g. ['OVERVIEW', 'ENTITY']).
When omitted all sections for the object are deleted.
database_name : str, optional
When provided, only rows with this DATABASE_NAME are deleted.
When omitted, all databases are affected.
Remove all documentation rows for a process from every documentation table.
Deletes from:
- FS_PROCESS_DOCUMENTATION_BUSINESS_LOGIC (keyed by PROCESS_ID)
- FS_PROCESS_DOCUMENTATION_FEATURES (keyed by PROCESS_ID)
- FS_PROCESS_DOCUMENTATION_EXPLAIN (keyed by PROCESS_ID)
- FS_BUSINESS_DICTIONARY_OBJECTS (keyed by the process VIEW_NAME)
- FS_BUSINESS_DICTIONARY_COLUMNS (keyed by the process VIEW_NAME)
- FS_BUSINESS_DICTIONARY_SECTIONS (keyed by the process VIEW_NAME)
Uses NONSEQUENCED VALIDTIME DELETE to fully remove all temporal versions,
mirroring the approach used by remove_process for the process catalog.
Each step is independent and non-fatal: a failure on one table is logged as a warning and the remaining tables are still cleaned up.
Parameters¶
process_id : str UUID of the process whose documentation should be removed.
Remove all business dictionary documentation for a given object in one call.
Deletes all currently-valid rows keyed on object_name from:
FS_BUSINESS_DICTIONARY_OBJECTS— object-level descriptionFS_BUSINESS_DICTIONARY_SECTIONS— structured sections (OVERVIEW, ENTITY, …)FS_BUSINESS_DICTIONARY_COLUMNS— column / feature descriptions
Designed to be called alongside DROP VIEW / DROP TABLE for any
documented object: dataset views, process views, source tables, etc.
Failures on individual tables are non-fatal warnings so a partial cleanup
does not abort the caller.
Parameters¶
object_name : str The OBJECT_NAME / TABLE_NAME to remove (leading/trailing quotes are stripped automatically). database_name : str, optional Restrict deletion to this database. When omitted, all databases are affected (useful when the database is unknown or may vary).
Find (and with dry_run=False close) feature-documentation rows filed under
another entity's FEATURE_ID.
Before 0.3.2.1 :func:upload_documentation wrote a process's feature
description under every FEATURE_ID of that name in the data domain, whatever
the entity. Returns the rows concerned (PROCESS_ID, FEATURE_ID, FEATURE_NAME,
the process entity and the feature's entity). History is kept: the rows are
closed with CURRENT VALIDTIME DELETE.
set_feature_documentation(feature: str, description: str, *, entity=None, data_domain=None, propagate: bool = True, refresh_index: bool = True, regenerate_dependents: bool = False, overwrite_manual: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write the business description of one feature by hand.
The feature is resolved as the triplet (data domain, entity, name); a name
registered under several entities or domains raises with the candidates -
pass entity= / data_domain=. The text is written for every process
producing the feature (process documentation + business dictionary on the
process view), marked MANUAL, and copied to each dataset exposing it
(propagate). No LLM is used.
Example::
tdfs4ds.set_feature_documentation(
'nb_days_since_last_transactions',
'Days between the business date and the customer\'s latest purchase.',
entity='CustomerID', author='Jane Doe')
set_process_documentation(process: str, *, business_logic: Optional[str] = None, entity_description: Optional[str] = None, entity_columns: Optional[Dict[str, str]] = None, features: Optional[Dict[str, str]] = None, propagate: bool = True, refresh_index: bool = True, regenerate_dependents: bool = False, overwrite_manual: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write a process's documentation by hand (None keeps a part as it is).
process: PROCESS_ID or view name. business_logic: what the process
computes. entity_description: what one row (the entity) is.
entity_columns: {key column: description}. features:
{feature name: description} for the features this process produces.
Example::
tdfs4ds.set_process_documentation(
'DB_LAB.FEAT_ENG_CUST',
business_logic='Monthly spending profile of each customer.',
features={'mean_Transaction_Amount': 'Average amount of the customer\'s purchases.'})
set_dataset_documentation(dataset: str, *, overview: Optional[str] = None, sections: Optional[Dict[str, str]] = None, columns: Optional[Dict[str, str]] = None, refresh_index: bool = True, regenerate_dependents: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write a dataset's documentation by hand (None keeps a part).
overview is the dataset's description (also its OVERVIEW section);
sections: {name: text} - OVERVIEW, ENTITY, FEATURE_THEMES,
BUSINESS_QUESTIONS, INTENDED_AUDIENCE or a name of your own; columns:
{column: description}.
set_object_documentation(database: str, name: str, *, description: Optional[str] = None, columns: Optional[Dict[str, str]] = None, sections: Optional[Dict[str, str]] = None, object_type: Optional[str] = None, source: str = 'MANUAL', refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Document a source table or a plain view by hand.
object_type: 'T' (table, default when unknown) or 'V'.
source='IMPORTED' when the text comes from an existing data dictionary.
set_deep_table_documentation(database: str, table: str, *, id_columns=None, tag_column=None, payload_column=None, timestamp_column=None, timestamp_timezone=None, description: Optional[str] = None, columns: Optional[Dict[str, str]] = None, tags=None, tag_name_column=None, tag_text_column=None, unit_column=None, source: str = 'MANUAL', regenerate_dependents: bool = False, refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Document a deep (historian) table: its layout, its columns and each of its tags.
A deep table stores one row per (id columns, timestamp, tag) with the value in a payload column. Each tag - a value of the tag column, such as a sensor name - is documented as if it were a column of the pivoted table.
The layout (id_columns, tag_column, payload_column,
timestamp_column, timestamp_timezone - UTC by default) is required the
first time and read back afterwards, so a later call can pass tags= alone.
tags: {tag: description} or a DataFrame (tag_name_column,
tag_text_column, optional unit_column; by default its first two
columns). source='IMPORTED' for a historian tag list, 'MANUAL' for a
correction. columns documents the physical columns.
The report lists the processes whose SQL names a written tag (stale): their
documentation was written from the old text. regenerate_dependents=True
re-documents them when an instruct model is configured.
propagate_documentation(datasets=None, *, overwrite_manual: bool = False, refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Fill dataset column descriptions from the feature documentation - no LLM.
datasets: a dataset (name / id) or a list; None = every dataset.
Columns a person wrote are kept unless overwrite_manual. This is how a
store without an LLM gets documented datasets.
get_documentation(target: str, kind: Optional[str] = None, *, entity=None, data_domain=None, history: int = 10) -> dict
Current documentation of a feature, process, dataset or object, with who wrote each part.
kind: 'feature' | 'process' | 'dataset' | 'object'; guessed when omitted
(process id or view, dataset, feature name, then 'DB.OBJECT'). Each part
carries source (MANUAL / IMPORTED / PROPAGATED, or 'LLM or legacy'),
label, editor (declared), db_user and since; history holds
the latest log events of the object.
mark_documentation_manual(targets, parts: Optional[Iterable[str]] = None, *, kind: Optional[str] = None, source: str = 'MANUAL', author: Optional[str] = None) -> dict
Protect existing documentation from LLM runs, without changing its text.
For documentation written by hand before 0.3.2.1 (it carries no mark).
parts narrows it: ['DESCRIPTION', 'BUSINESS_LOGIC', 'COLUMN:amount', 'SECTION:OVERVIEW'].
release_manual_documentation(target: str, parts: Optional[Iterable[str]] = None, *, kind: Optional[str] = None, author: Optional[str] = None) -> dict
Let the LLM rewrite parts again (closes their IMPORTED / MANUAL mark; history kept).
deep_table_tags(database: str, table: str, *, since=None, until=None, documented: Optional[bool] = None)
The tags present in a deep table, with their row counts and first/last timestamps.
One GROUP BY over the table, bounded by since / until on the timestamp
column when given (interpreted in the layout's time zone, UTC by default).
This reads the table: on a large historian, give a window. It is never run
by tdfs4ds on its own.
documented=False keeps the tags without documentation, True the
documented ones; None keeps all, with a DOCUMENTED column.
The documentation activity log as a teradataml DataFrame (sort it with .sort).
target: an object name ('DB.OBJ' or 'OBJ'); since: a timestamp
string; actions: a list of ACTION values.
Create the provenance and log tables (and their views) when missing.
Never raises: a failure is remembered (no retry on reads) and reported once.
Returns {'provenance': bool, 'log': bool}.