Skip to content

Dataset documentation

document_dataset_incremental walks a dataset's lineage bottom-up and produces structured documentation without re-running anything that is already documented.

What it does

  1. Builds the dependency graph with expand_datasets_via_process_catalog=True.
  2. For every undocumented intermediate view or process view found in the graph, calls document_process_incremental so upstream context is available before the dataset itself is described.
  3. Propagates feature and entity column descriptions from the process docs into the dataset's column docs — no extra LLM call.
  4. Issues a single JSON-constrained LLM call that produces five sections:
  5. OVERVIEW
  6. ENTITY
  7. FEATURE_THEMES
  8. BUSINESS_QUESTIONS
  9. INTENDED_AUDIENCE
  10. Stores each section as one row in FS_BUSINESS_DICTIONARY_SECTIONS so the consumer agent can retrieve them without chunking.

Usage

from tdfs4ds.genai import document_dataset_incremental

result = document_dataset_incremental(
    dataset_id   = '<UUID>',   # from dataset_catalog()
    force_update = False,
    upload       = True,
)

result['DATASET_SECTIONS']     # dict of {section_name: content}
result['COLUMN_DESCRIPTIONS']  # propagated column docs

Pass force_update=True to ignore the unchanged-hash short-circuit when the underlying SQL has not changed but you want a fresh LLM pass.

Finding the dataset ID

tdfs4ds.dataset_catalog()
# Pick a row; its DATASET_ID column is what document_dataset_incremental wants.

You can also retrieve it directly:

qry = f"""
CURRENT VALIDTIME
SELECT DATASET_ID
FROM {tdfs4ds.SCHEMA}.V_{tdfs4ds.DATASET_CATALOG_NAME}_CATALOG
WHERE DATASET_NAME = 'DS_CUSTOMER_SPEND'
"""
dataset_id = tdml.execute_sql(qry).fetchall()[0][0]

Why bottom-up

If the dataset depends on an undocumented process view, the LLM has no context for the columns it sees. The bottom-up walk guarantees that:

  • Source tables are documented (or skipped if a description already exists).
  • Every process view between source and dataset is documented.
  • The dataset description sits on top of a fully resolved column-level context — no hallucinations about what total_amount means.

A shared visited-set across the recursive calls ensures every process view is documented at most once, even when several datasets share it.

What the agent reads

The consumer agent's LINEAGE, DEFINITION, and EXPLAIN skills query the business dictionary populated by this call. Skipping documentation for a dataset means the agent's SEARCH and DEFINITION answers will fall back to the underlying process docs — usually fine, but you lose the dataset-level overview.