Business dictionary

Three temporal tables store business-oriented descriptions for any database object, its columns, and its sections. They form a 3-level hierarchy designed for chunking-free hierarchical RAG.

Level	Table	Key	Extra columns
0	FS_BUSINESS_DICTIONARY_OBJECTS	(DATABASE_NAME, OBJECT_NAME)	OBJECT_TYPE CHAR(1) ('T'/'V'/'D'), BUSINESS_DESCRIPTION
1	FS_BUSINESS_DICTIONARY_SECTIONS	(DATABASE_NAME, OBJECT_NAME, SECTION_NAME)	SECTION_CONTENT VARCHAR(32000)
2	FS_BUSINESS_DICTIONARY_COLUMNS	(DATABASE_NAME, TABLE_NAME, COLUMN_NAME)	BUSINESS_DESCRIPTION

All three are VALIDTIME temporal — every write uses CURRENT VALIDTIME MERGE, so the full history of descriptions is preserved.

Provisioning

The tables are created automatically by tdfs4ds.setup() and tdfs4ds.connect(create_if_missing=True). Override the default names via keyword arguments to connect:

tdfs4ds.connect(
    database = 'my_db',
    documentation_business_dictionary_objects = 'MY_BIZ_OBJECTS',
    documentation_business_dictionary_columns = 'MY_BIZ_COLUMNS',
)

Populating without an LLM

To document one feature, process or dataset by hand, use the set_*_documentation functions in Writing documentation by hand. They write every place a description is read from, copy feature descriptions to datasets, and protect the text from later LLM runs. The loaders below suit a bulk load of an existing data dictionary; since 0.3.2.1 their rows are tagged imported and are kept by LLM runs (source=None records nothing).

import pandas as pd
from tdfs4ds.genai import (
    upload_business_dictionary_objects,
    upload_business_dictionary_columns,
    upload_business_dictionary_sections,
)

# Level 0 — objects
upload_business_dictionary_objects(pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'OBJECT_NAME'         : 'CUSTOMER',
        'OBJECT_TYPE'         : 'T',
        'BUSINESS_DESCRIPTION': 'Core customer table — one row per enrolled customer.',
    },
    {
        'DATABASE_NAME'       : 'MY_DB',
        'OBJECT_NAME'         : 'V_CUSTOMER_ORDERS',
        'OBJECT_TYPE'         : 'V',
        'BUSINESS_DESCRIPTION': 'Business view joining customers with their order history for reporting.',
    },
]))

# Level 1 — sections
upload_business_dictionary_sections(pd.DataFrame([
    {
        'DATABASE_NAME'  : 'MY_DB',
        'OBJECT_NAME'    : 'DS_CUSTOMER_SPEND',
        'SECTION_NAME'   : 'OVERVIEW',
        'SECTION_CONTENT': 'Customer-level analytical dataset combining spend metrics …',
    },
]))

# Level 2 — columns
upload_business_dictionary_columns(pd.DataFrame([
    {
        'DATABASE_NAME'       : 'MY_DB',
        'TABLE_NAME'          : 'CUSTOMER',
        'COLUMN_NAME'         : 'CUSTOMER_ID',
        'BUSINESS_DESCRIPTION': 'Unique customer identifier assigned at enrolment.',
    },
    {
        'DATABASE_NAME'       : 'MY_DB',
        'TABLE_NAME'          : 'CUSTOMER',
        'COLUMN_NAME'         : 'BIRTH_DATE',
        'BUSINESS_DESCRIPTION': 'Customer date of birth, used for age-band segmentation.',
    },
]))

All three functions:

• Validate that the required columns are present and raise ValueError with the list of missing columns if not.
• Use CURRENT VALIDTIME MERGE — re-running them updates existing descriptions and preserves history.
• Require no LLM — provide whatever text you want.

How LLM documentation populates the dictionary

Call	What it writes to the dictionary
document_process	Mirrors the process's entity description into FS_BUSINESS_DICTIONARY_OBJECTS (OBJECT_TYPE='V', the process view) and the feature descriptions into FS_BUSINESS_DICTIONARY_COLUMNS. The business logic stays in FS_PROCESS_DOCUMENTATION_BUSINESS_LOGIC. Parts imported or written by a person are kept.
document_dataset_incremental	Writes OBJECT_TYPE='D' into FS_BUSINESS_DICTIONARY_OBJECTS, the five generated sections into FS_BUSINESS_DICTIONARY_SECTIONS, and propagated column docs into FS_BUSINESS_DICTIONARY_COLUMNS.
document_feature_store_incremental	Calls both of the above for every catalogued process and dataset.

Why a sections table

A common RAG mistake is to store one giant text blob per object and rely on embedding-time chunking. With a (OBJECT_NAME, SECTION_NAME) key, every section is independently embeddable and retrievable. The consumer agent uses this granularity: it returns the right section for the asked intent (OVERVIEW for "summary", BUSINESS_QUESTIONS for "what can I ask about …", etc.) without ever splitting a section in the middle.