fs-document

LLM-powered process documentation + EXPLAIN plan quality score (1–5).

---

name: "Feature Store: Document" description: "Document a tdfs4ds feature store — LLM-generated (one process, a dataset, or the whole store from the lineage graph with rebuild_documentation), loaded from a data dictionary, or written by hand with provenance (including deep/historian tables), and keep the vector index in step. Also covers the EXPLAIN plan quality score (1-5). Invoke when the user wants to document, review or fill documentation gaps." argument-hint: "[process-id][llm-provider: openai|bedrock|vllm|azure]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

---

Generate Python code to produce and persist LLM-powered documentation for a registered tdfs4ds process, including an EXPLAIN plan quality score.

User context: $ARGUMENTS

Related skills

Step	Skill	Why
Before	fs-upload	The process must be registered before documentation can run.
Sibling	fs-lineage	Pure dependency graph without the LLM cost.
If low score	fs-analyze	Full scalability analysis + FilterManager rewrite proposal.
If raw EXPLAIN	td-explain	Visual flow diagram of any Teradata EXPLAIN output.
After	fs-vector-store	The vector index is built from what this skill writes; it must be refreshed to search new text.
Downstream	fs-agent	The consumer agent reads the business dictionary populated here.

Demos: demos/notebooks/G1 - Documentation and query optimisation.ipynb, T3 - What the documentation knows.ipynb (deep / historian tables).

Which way to document

Situation	Call	Index refreshed?
At registration	upload_features(..., document=True, vector_indexing=True) (cascade_redocumentation=True also re-documents what depends on it)	with vector_indexing=True
A dataset when it is built	build_dataset(..., document=True, vector_indexing=True)	with vector_indexing=True
One process, now	document_process(process_id) (always rewrites) or document_process_incremental(process_id, force_update=False) (skips a documented one, documents upstream views first)	no: call build_vector_index()
One dataset	document_dataset_incremental(dataset_id)	no
A plain view that is not a process	document_view_incremental('<DATABASE>', '<VIEW_NAME>')	no
The whole store (first time, or to fill gaps)	tdfs4ds.rebuild_documentation(force_update=False): walks the persisted lineage graph once, upstream first, each object documented once	yes (rebuild_vector_indexing=True, default)
After changing some views	tdfs4ds.rebuild_documentation(objects=['DB.VIEW']): the objects and everything downstream, forced	yes
By hand, no LLM	set_feature_documentation(...) and the other set_* below	yes (refresh_index=True, default)

• rebuild_documentation refreshes the lineage graph first (rebuild_lineage=True); a store documented by hand only needs rebuild_lineage=False. extra_views=[...] adds standalone views that no process or dataset reaches.
• force_update=False skips what is already documented: re-running after a stop (an expired token, a time-out) resumes where it stopped.
• What is missing: the admin app's Documentation › Coverage panel lists every process, feature, dataset, intermediate view and source table as documented / partial / missing, with its index status, and downloads the gaps as CSV. Documentation › Objects shows one object, its provenance and history, and edits it.

Ask for any missing values before generating code: - Process ID or view name — Either the UUID, or the qualified view name (e.g. "demo_user"."FEAT_ENG_CUST"). Prefer the view name when the user provides it — use get_process_id to resolve it (see pattern below). - LLM provider — 'openai', 'azure', 'bedrock', or 'vllm'. Credentials loaded from env vars.

document_process produces and uploads: - Business logic description of the SQL query - Entity description - Per-feature column descriptions - EXPLAIN plan analysis with a quality score 1–5 (5 = optimal use of PI and partitioning)

Emit the following pattern, substituting real values:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)    # teradataml context from TDFS4DS_TD_HOST / _USERNAME / _PASSWORD
tdfs4ds.connect(database='<DATABASE>', create_if_missing=True)
import teradataml as tdml
tdfs4ds.select_data_domain('<DATA_DOMAIN>')   # or set TDFS4DS_DATA_DOMAIN in .env

# ---------------------------------------------------------------------------
# LLM configuration — choose one provider block.
# Preferred: set TDFS4DS_INSTRUCT_MODEL_* in .env — auto-loaded at import.
# The explicit assignments below override any file/env settings if needed.
# ---------------------------------------------------------------------------

# Option A: OpenAI-compatible (openai, vllm, azure)
tdfs4ds.INSTRUCT_MODEL_PROVIDER = '<PROVIDER>'                  # 'openai', 'vllm', or 'azure'
tdfs4ds.INSTRUCT_MODEL_URL      = '<MODEL_URL>'       # None for default OpenAI endpoint
# API key: keep it in .env as TDFS4DS_INSTRUCT_MODEL_API_KEY (never in code)
tdfs4ds.INSTRUCT_MODEL_MODEL    = '<MODEL_NAME>'

# Option B: AWS Bedrock
# tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'bedrock'
# tdfs4ds.INSTRUCT_MODEL_MODEL    = 'anthropic.claude-3-sonnet-20240229-v1:0'
# (uses boto3 default credentials — no API key needed)

# ---------------------------------------------------------------------------
# 1. Inspect available processes
# ---------------------------------------------------------------------------
tdfs4ds.process_catalog()

# ---------------------------------------------------------------------------
# 2. Resolve process ID — two options
# ---------------------------------------------------------------------------

# Option A: you already have the UUID
process_id = '<PROCESS_ID>'

# Option B: resolve from a qualified view name (preferred when user supplies a view name)
from tdfs4ds.process_store.process_query_administration import get_process_id

process_id = get_process_id('"<DATABASE>"."<VIEW_NAME>"')
# get_process_id strips quotes, handles "db"."view" or plain view names,
# and falls back to tdfs4ds.SCHEMA when no database prefix is given.

# ---------------------------------------------------------------------------
# 3. Document the process
#    - Calls the LLM to generate business logic, entity, and feature docs
#    - Runs EXPLAIN on the process SQL and scores it 1–5
#    - Persists everything to the feature store documentation tables
# ---------------------------------------------------------------------------
from tdfs4ds.genai import document_process

process_info = document_process(
    process_id=process_id,
    show_sql_query=True,      # print the process SQL alongside the docs
    show_explain_plan=True,   # print the raw EXPLAIN plan
)

# The returned dict contains all generated documentation:
# process_info['DOCUMENTED_SQL']           — business logic + feature descriptions
# process_info['EXPLAIN_ANALYSIS']         — plain-text EXPLAIN plan analysis
# process_info['USER_SCORE']               — int 1–5: SQL quality (what you control)
# process_info['OPTIMIZATION_SCORE']       — int 1–5: overall execution quality (infra included)
# process_info['EXPLAIN_WARNINGS']         — list[str]: flagged issues
# process_info['EXPLAIN_RECOMMENDATIONS']  — list[str]: split [You] / [Data Engineer / DBA]

# ---------------------------------------------------------------------------
# 4. Document all processes in the current data domain (batch)
# ---------------------------------------------------------------------------
# process_ids = (
#     tdfs4ds.process_catalog()[['PROCESS_ID']].to_pandas().PROCESS_ID.values
# )
# for pid in process_ids:
#     document_process(process_id=pid)

# ---------------------------------------------------------------------------
# 4b. Document a non-catalog view (base/staging views not registered as processes)
#     Use document_view_incremental when SHOW VIEW is available but the view is
#     not in the process catalog.
# ---------------------------------------------------------------------------
# from tdfs4ds.genai import document_view_incremental
# document_view_incremental('<DATABASE>', '<VIEW_NAME>')   # e.g. 'DB_SOURCE', 'FEAT_BASE_CUSTOMER'

# ---------------------------------------------------------------------------
# 4c. Make the new text searchable by the consumer agent (document_process does
#     not touch the vector index; rebuild_documentation and the set_* functions do)
# ---------------------------------------------------------------------------
# from tdfs4ds.agent import build_vector_index
# build_vector_index(force_rebuild=False)    # only new or changed rows (see fs-vector-store)

# ---------------------------------------------------------------------------
# 5. Retrieve previously stored documentation (no LLM call)
# ---------------------------------------------------------------------------
from tdfs4ds.process_store.process_store_catalog_management import get_process_info

stored = get_process_info(process_id)
print(stored.get('DOCUMENTED_SQL'))
print(stored.get('EXPLAIN_ANALYSIS'))

Processes that call a model or a known operator (0.3.5.15+)

A process view that scores a BYOM model (ONNXPredict, PMMLPredict, H2OPredict, ...) or compresses series with td-ts-compressor (ts_compress_to) is documented with a fixed block written by code: for BYOM, the model table's metadata rows (non-LOB columns, the row the view's WHERE selects first) and the selection rule; for ts_compress_to, the parameters, the mode and the meaning of the output columns. Nothing to call: it happens in upload_features(document=True), document_process_incremental and the agent's EXPLAIN. To make it useful, keep the metadata in the model table (save_byom(..., additional_columns={'trained_as_of': ..., 'model_comment': ...})); preview the block with from tdfs4ds.genai.operator_context import operator_context; print(operator_context(view_sql)).

EXPLAIN score interpretation

Score	Meaning
5/5	Query fully leverages primary index and/or partitioning — optimal
3–4/5	Partial optimisation — some full-AMP or full-partition scans remain
1–2/5	Full-table scans, product joins, or spool-heavy operations — consider rewriting or adding a FilterManager

Formula consistency validation (v0.3.1.35+)

After LLM documentation runs, the engine cross-checks each generated column description against SQL signals extracted from the view DDL (ABS, STDDEV, PERCENTILE_CONT, SIGN, aggregations). Four additional columns are written to FS_BUSINESS_DICTIONARY_COLUMNS:

Column	Type	Meaning
MONOTONIC_DIRECTION	VARCHAR	'increasing', 'decreasing', or NULL — direction implied by the formula
VALUE_DOMAIN	VARCHAR	Expected value range or domain (e.g. '[0, 1]', 'non-negative')
IS_ABSOLUTE_VALUE	CHAR(1)	'Y' when the formula applies ABS — description must not mention sign
FORMULA_CONSISTENCY_WARNING	VARCHAR	Plain-language note when the description contradicts the SQL signals

Migrating an existing install: run once after upgrading to v0.3.1.35+ to add these columns to an existing table (idempotent):

from tdfs4ds.genai.documentation import migrate_documentation_columns_schema
migrate_documentation_columns_schema()

---

Temporary credentials (v0.3.1.215+)

When the instruct model uses a short-lived token (for example a Bedrock bearer token in INSTRUCT_MODEL_API_KEY), a long rebuild_documentation() can outlive it. On expiry tdfs4ds asks for a renewed token with hidden input, in a terminal or a notebook, and retries the failed call. It asks up to tdfs4ds.CREDENTIAL_RENEWAL_ATTEMPTS times (default 3).

• A blank answer stops the run cleanly. The summary carries stopped_on_credentials=True.
• To resume, set a valid token and re-run rebuild_documentation(force_update=False). Objects that are already documented are skipped.
• Without an interactive session (a server or papermill), nothing is prompted and the expiry error propagates.
• Set tdfs4ds.CREDENTIAL_RENEWAL_ATTEMPTS = 0 to turn prompting off.

Business dictionary

Two temporal tables store business-oriented descriptions for database objects and their columns, independently of the process documentation workflow. They are provisioned automatically by tdfs4ds.connect(create_if_missing=True).

Table	Key columns	Purpose
FS_BUSINESS_DICTIONARY_OBJECTS	DATABASE_NAME, OBJECT_NAME	One row per table or view
FS_BUSINESS_DICTIONARY_COLUMNS	DATABASE_NAME, TABLE_NAME, COLUMN_NAME	One row per column

Both tables are VALIDTIME temporal — every write uses CURRENT VALIDTIME MERGE, so the full history of descriptions is preserved.

Export the documentation (0.3.3.16)

tdfs4ds.export_documentation('documentation.xlsx')                   # About / Objects / Columns / Sections
tdfs4ds.export_documentation('documentation.html')                   # one self-contained page
tdfs4ds.export_documentation('sources.xlsx', scope='outside')        # source tables and other views only
tdfs4ds.export_documentation('fs.html', scope='feature_store', databases='DB_LAB')

Every documented table and view with its description, documented columns and sections, each row tagged in the feature store (process view, dataset, Time/FilterManager) or outside the feature store (source tables and other views, with the feature processes that read them, from lineage). The same rule drives the consumer agent's answers (tdfs4ds.documentation_export.object_scope). Admin app: Documentation › Coverage › Export.

What is not documented yet is listed too (STATUS='missing', empty text, yellow in the XLSX): process views, datasets and lineage tables/views without a description, their undocumented columns (from DBC, in table order) and a dataset's missing standard sections. include_missing=False for documented rows only.

Round trip — fill the yellow cells, then:

tdfs4ds.import_documentation('documentation.xlsx')                                   # dry run
tdfs4ds.import_documentation('documentation.xlsx', dry_run=False, author='Jane Doe')

Only new or changed texts are written, through set_process_documentation / set_dataset_documentation / set_object_documentation (MANUAL provenance, kept by LLM runs; a feature's text also reaches the process documentation and the datasets). An emptied cell deletes nothing. Read-only stores, managers, internal objects and process-view sections are skipped with a reason. One index refresh at the end.

Upload object descriptions

import pandas as pd
from tdfs4ds.genai import upload_business_dictionary_objects

objects_df = pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'OBJECT_NAME'         : 'CUSTOMER',
        'OBJECT_TYPE'         : 'T',           # 'T' = table, 'V' = view
        'BUSINESS_DESCRIPTION': 'Core customer table. Each row represents a unique customer enrolled in the loyalty programme.',
    },
    {
        'DATABASE_NAME'       : 'MY_DB',
        'OBJECT_NAME'         : 'V_CUSTOMER_ORDERS',
        'OBJECT_TYPE'         : 'V',
        'BUSINESS_DESCRIPTION': 'Business view joining customers with their order history for reporting.',
    },
])
upload_business_dictionary_objects(objects_df)

Upload column descriptions

from tdfs4ds.genai import upload_business_dictionary_columns

columns_df = pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'TABLE_NAME'          : 'CUSTOMER',
        'COLUMN_NAME'         : 'CUSTOMER_ID',
        'BUSINESS_DESCRIPTION': 'Unique customer identifier assigned at enrolment.',
    },
    {
        'DATABASE_NAME'       : 'MY_DB',
        'TABLE_NAME'          : 'CUSTOMER',
        'COLUMN_NAME'         : 'BIRTH_DATE',
        'BUSINESS_DESCRIPTION': 'Customer date of birth, used for age-band segmentation.',
    },
])
upload_business_dictionary_columns(columns_df)

Both functions validate that all required columns are present and raise ValueError with the list of missing columns if not. Since 0.3.2.1 the rows they load are tagged IMPORTED (pass author='...'). That is the right tag for source-data dictionaries, and LLM runs keep those rows. Pass source=None to record nothing.

Writing documentation by hand (0.3.2.1)

Use these when a user asks how to save a comment or description on a feature, a process or a dataset. No LLM is needed. The text is tagged MANUAL and kept by every later LLM run.

tdfs4ds.set_feature_documentation('nb_days_since_last_transactions', 'Days since the last purchase.',
                                   entity='CustomerID', author='Jane Doe')          # both places + dataset copies
tdfs4ds.set_process_documentation(pid, business_logic='...', features={'sum_x': '...'}, author='Jane Doe')
tdfs4ds.set_dataset_documentation('DS_CUSTOMER', overview='...', sections={'BUSINESS_QUESTIONS': '...'})
tdfs4ds.set_object_documentation('DB_SOURCE', 'transactions', columns={'amount': '...'})  # source table / view
tdfs4ds.propagate_documentation('DS_CUSTOMER')     # dataset columns from feature docs, no LLM
tdfs4ds.link_feature_documentation(pid)            # feature rows of a process from its view's column docs (repair, no LLM)
tdfs4ds.get_documentation('DS_CUSTOMER')           # each part: label (imported/manual/propagated/...), editor, since
tdfs4ds.mark_documentation_manual(target, parts)   # protect legacy hand-written docs
tdfs4ds.release_manual_documentation(target, parts)  # let the LLM rewrite again
tdfs4ds.documentation_log(target='DB.OBJ').to_pandas()  # who did what, when, from where
# a deep (historian) table: layout once, then each tag as if it were a column of the pivoted table
tdfs4ds.set_deep_table_documentation('DB', 'HISTORIAN', id_columns=['MACHINE_NR'], tag_column='SENSOR_NAME',
                                     payload_column='PAYLOAD', timestamp_column='TD_TIMECODE',
                                     tags=tag_df, tag_name_column='SENSOR_NAME', tag_text_column='DESCRIPTION',
                                     unit_column='UNIT', source='IMPORTED')
tdfs4ds.deep_table_tags('DB', 'HISTORIAN', since='2026-09-01', documented=False)  # reads the table: give a window

• Deep tables. The layout is the reserved section DEEP_LAYOUT; a tag is a column row whose name is not a physical column. The LLM gets only the tags the documented SQL names (its literals), whatever the SQL shape. A table without a layout behaves exactly as before.

• Tables read through a view (0.3.4.6). When users read a table through a view (LOCKING ROW FOR ACCESS, often a row restriction on the user or a join to an entitlement table), document the table: a view whose every column is a column of that table (renames allowed, row filters allowed) takes its documentation, layout and tags included, marked PROPAGATED, with no right on the table needed. It happens by itself in the documentation walk (EXPOSED_VIEW_AUTO_ATTACH, default True); tdfs4ds.suggest_exposed_views(databases=[...]) lists candidates, attach_exposed_view('DB.VIEW') does it now, refresh_exposed_views() after an import. Edits on the table through set_object_documentation / set_deep_table_documentation reach its views by themselves.

• Soft clones of a TimeManager (0.3.6.0). With a domain clock every process reads its own clone (<process view>_TM). Document the clock once (set_object_documentation('<DB>.<CLOCK>', description=..., columns={'BUSINESS_DATE': ...})): the documentation walk copies it onto each clone (PROPAGATED), with no LLM call per clone. A clock without documentation leaves its clones to the usual path.

• Attach a view or table (0.3.5.2). A view that joins a dataset with a source table, read by no process, is outside the feature store until it is attached. attach_objects records its lineage and documents it: pass-through columns take the description of their source column (PROPAGATED), computed columns and the object description are written by the LLM (skipped without an instruct model). A table is documented by its owner (set_object_documentation). IMPORTED and MANUAL parts are never overwritten.

tdfs4ds.suggest_attachments(databases=['<DB>'])                  # read only: views reading a dataset or process
tdfs4ds.attach_objects(['<DB>.<VIEW>'], data_domain='<DOMAIN>', dry_run=True)   # report first
tdfs4ds.attach_objects(['<DB>.<VIEW>'], data_domain='<DOMAIN>')
tdfs4ds.attached_objects()                                       # DATABASE_NAME, OBJECT_NAME, KIND, DATA_DOMAIN, DOCUMENTED_AT
tdfs4ds.refresh_attached_objects(dry_run=True)                   # unchanged / sources_changed / changed / broken / disappeared
tdfs4ds.refresh_attached_objects(dry_run=False)                  # also run by rebuild_documentation
tdfs4ds.detach_objects(['<DB>.<VIEW>'])

A process view, a dataset, an FS_* object or a name not in the database is refused. refresh_attached_objects re-documents only what changed (the LLM only for a computed column or changed sources); a broken object is reported and nothing is written; a disappeared one is detached, its documentation ended (readable at a past date).

• Check before writing. dry_run=True returns the report without writing: written, propagated, skipped_manual, stale (the LLM text derived from the old text) and index.
• Regenerating dependents. regenerate_dependents=True regenerates the stale LLM documentation, only when an instruct model is configured.
• Ambiguous names. A feature name shared by several entities or domains raises with the candidates. Pass entity= / data_domain= (the default domain is tdfs4ds.DATA_DOMAIN).
• Overwriting human text. The LLM functions take overwrite_manual=True to replace human text.
• Where it is stored. Provenance lives in FS_DOCUMENTATION_PROVENANCE and the log in FS_DOCUMENTATION_LOG. Both are optional tables created by connect(); the six documentation tables are never altered, and FS_FOLLOW_UP is never used for documentation.
• In the admin app: Documentation › Objects › Edit documentation (admin view). It requires a name and shows a preview before saving.

Rules

• When the user provides a view name instead of a UUID, always use get_process_id from tdfs4ds.process_store.process_query_administration to resolve it — never ask the user to look up the UUID manually.
• document_process both generates and persists documentation — it is safe to re-run; it overwrites previous LLM-written docs for the same process_id (imported / manual parts are kept unless overwrite_manual=True).
• LLM config can be set via TDFS4DS_INSTRUCT_MODEL_* in .env or tdfs4ds.json (auto-loaded at import tdfs4ds). Explicit programmatic assignment always wins and must appear before calling document_process.
• For 'bedrock' provider, no INSTRUCT_MODEL_URL or INSTRUCT_MODEL_API_KEY is needed — it uses boto3 credentials.
• A low EXPLAIN score (1–2) is a signal to run fs-analyze to check for partition/PI opportunities and a possible FilterManager rewrite.
• upload_business_dictionary_objects / upload_business_dictionary_columns do not require an LLM — descriptions are provided by the user as plain text in a pandas DataFrame.
• To document a whole store, use rebuild_documentation(), not a loop over document_process: the walk documents upstream views before the processes that read them (the LLM sees their meaning), each object once, and refreshes the index at the end.
• After any documentation that does not refresh the index itself (table above), the consumer agent's search keeps the old text until build_vector_index() runs.
• Order: sources first, then processes, then datasets. A process documented before its source tables were described is documented from the column names alone and is not regenerated by a plain re-run (force_update=False skips documented objects): describe the sources, then re-document with force_update=True.
• build_dataset(document=True) / document_dataset_incremental also documents the upstream processes and views that have no documentation yet: one LLM pass each. On a new store, document the processes first (or call rebuild_documentation() once) rather than paying for it inside each dataset build.