Writing documentation by hand¶

Every feature, process, dataset and source table can be documented by hand: from Python, or from the admin app with a confirmation step. No LLM is needed.

Without an LLM, this is how the store is documented.
With an LLM, the LLM writes a first version and a person corrects or completes it.
An LLM added later documents only what nobody wrote, and keeps what people wrote.

Whatever a person writes or imports is kept by every later LLM documentation run, and every change is logged with who made it.

Document one feature¶

import tdfs4ds

report = tdfs4ds.set_feature_documentation(
    'nb_days_since_last_transactions',
    "Days between the business date and the customer's most recent purchase: 0 means the customer bought today.",
    entity='CustomerID',              # needed when the name exists for several entities
    data_domain='Customer Transaction Analytics',   # default: tdfs4ds.DATA_DOMAIN
    author='Jane Doe')                # recorded with the edit (default: your OS user)

Identifying the feature. A feature is identified by its data domain, entity and name. A name shared by several entities or domains raises an error listing the candidates; pass entity= and/or data_domain=. When data_domain is omitted, the session's tdfs4ds.DATA_DOMAIN is used to choose.

What is written:

the description in both places tdfs4ds reads a feature's description from:
the process documentation, which datasets inherit from;
the business dictionary on the process view, which the assistant, the vector index and the admin app read;
a copy on the column of every dataset exposing the feature (propagate=True, the default). A dataset column that a person wrote on the dataset itself is left alone, unless overwrite_manual=True.

The report (the function's return value) lists:

Key	What it holds
`written`	the parts written
`propagated`	the dataset columns updated
`skipped_manual`	the dataset columns left as a person wrote them
`stale`	documentation the LLM had written from the old text: the process's business logic, the dataset sections, downstream objects
`index`	the vector index refresh
`regenerated`	what was regenerated, with `regenerate_dependents=True`

dry_run=True returns the same report without writing anything.

Document a process¶

tdfs4ds.set_process_documentation(
    'DB_LAB.FEAT_ENG_CUST',                     # a PROCESS_ID or the process view
    business_logic='Spending profile of each customer: total, average and number of purchases, '
                   'and the days since the last one.',
    entity_description='One row per customer.',
    entity_columns={'CustomerID': 'Identifier of the customer.'},
    features={'sum_Transaction_Amount': 'Total amount the customer spent, in EUR.',
              'count_Transaction_Amount': 'Number of purchases the customer made.'},
    author='Jane Doe')

An argument left out (None) keeps that part as it is. A feature name that the process does not produce is refused.

Document a dataset¶

tdfs4ds.set_dataset_documentation(
    'DS_CUSTOMER',                               # name, DB.NAME or DATASET_ID
    overview='Customer activity and diversity: one row per customer.',
    sections={'BUSINESS_QUESTIONS': 'Which customers are active and how much they spend.',
              'INTENDED_AUDIENCE': 'Marketing analysts.'},
    columns={'CustomerID': 'Identifier of the customer.'},
    author='Jane Doe')

Sections. The standard sections are OVERVIEW, ENTITY, FEATURE_THEMES, BUSINESS_QUESTIONS and INTENDED_AUDIENCE, and you can add sections of your own. FEATURE_GROUPS is reserved: the consumer agent writes it.

Columns, without an LLM. A dataset's feature columns take their descriptions from the features:

tdfs4ds.propagate_documentation('DS_CUSTOMER')   # or a list, or None for every dataset

Document the source data¶

The LLM never documents a source table: its meaning belongs to the people who own it. You can load it from your existing data dictionary; those rows are tagged imported:

import pandas as pd

tdfs4ds.upload_business_dictionary_objects(pd.DataFrame([{
    'DATABASE_NAME': 'DB_SOURCE', 'OBJECT_NAME': 'transactions', 'OBJECT_TYPE': 'T',
    'BUSINESS_DESCRIPTION': 'One row per card transaction.'}]), author='Data owner')

dictionary = pd.read_csv('data_dictionary.csv')   # DATABASE_NAME, TABLE_NAME, COLUMN_NAME, BUSINESS_DESCRIPTION
tdfs4ds.upload_business_dictionary_columns(dictionary, author='Data owner')

You can also correct one item by hand. That part is then tagged manual:

tdfs4ds.set_object_documentation('DB_SOURCE', 'transactions',
                                 columns={'Transaction_Amount': 'Amount paid, in EUR, taxes included.'})

When the LLM later documents a view that reads this table, it is given these descriptions as context.

Deep (historian) tables¶

A historian stores one row per id, timestamp and tag (a sensor, a signal), with the value in a payload column. What a reader needs documented is each tag, as if it were a column of the pivoted table. Declare the layout once, then document the tags:

import pandas as pd

tags = pd.read_csv('tag_list.csv')            # e.g. SENSOR_NAME, DESCRIPTION, UNIT
tdfs4ds.set_deep_table_documentation(
    'HCF_TEST', 'HISTORIAN',
    id_columns=['MACHINE_NR'], tag_column='SENSOR_NAME', payload_column='PAYLOAD',
    timestamp_column='TD_TIMECODE', timestamp_timezone='UTC',
    description='Sensor readings of the drawing machines.',
    tags=tags, tag_name_column='SENSOR_NAME', tag_text_column='DESCRIPTION', unit_column='UNIT',
    source='IMPORTED', author='Process engineering')

The layout (id columns, tag, payload, timestamp, time zone) is checked against the table's columns and stored as a reserved section DEEP_LAYOUT. It is required the first time and read back afterwards.
Each tag is stored as a column row of the table: searchable, protected from LLM runs, tracked like any other. A tag that is also a physical column, longer than 128 characters or not Latin is refused with the reason.
Correcting one tag later needs no layout:

tdfs4ds.set_deep_table_documentation('HCF_TEST', 'HISTORIAN',
                                     tags={'Drawing_Length': 'Length of fibre drawn since the start, in km'})

The report lists, as stale, the processes whose SQL names that tag (regenerate_dependents=True re-documents them with an instruct model). - Undocumented tags. tdfs4ds.deep_table_tags('HCF_TEST', 'HISTORIAN', since='2026-09-01', documented=False) lists the tags present in the table with no documentation. It reads the table (one GROUP BY): on a large historian, give a window. - What the LLM receives. When the LLM documents a process or view that reads a deep table, it gets the layout, the physical columns, and only the tags its SQL names (its string and number literals), whatever the SQL's shape: CASE WHEN, IN (...), PIVOT, a join condition. When the SQL names none, at most tdfs4ds.DEEP_CONTEXT_MAX_TAGS (40) are listed. A tag named in the SQL but not documented is logged, so you know which to document next. - The agent answers "what is Drawing_Length?" by first saying it is a tag of the historian (a value of SENSOR_NAME, its readings in PAYLOAD), then explaining it. - In the admin app, Documentation > Objects > Edit shows the layout, the columns and a Tags group (filter, pages of 200, Add a tag); each tag is corrected, protected or released on its own.

A table without a layout is unchanged everywhere. The demo track T1-T3 (demos/README.md) shows all of this on stock prices.

Tables read through a view¶

Users are often given a view over a table instead of the table itself: a LOCKING ROW FOR ACCESS view, frequently with a row restriction (a WHERE on the user, a join to an entitlement table). The view's columns mean what the table's columns mean, so document the table once: the view takes its documentation.

tdfs4ds.suggest_exposed_views(databases=['HCF_ACCESS'])          # read-only: which views expose which table
tdfs4ds.attach_exposed_view('HCF_ACCESS.HISTORIAN_V')             # declare + copy now (optional, see below)
tdfs4ds.exposed_views()                                           # the declared pairs

Which views qualify. Every output column must be a column of one table, unchanged or renamed. Clauses that only remove rows (WHERE, QUALIFY, DISTINCT, TOP, SAMPLE, joins whose tables give no output column) make the view filtered, and its description says the rows are restricted. Aggregation, GROUP BY, set operations or columns from two tables do not qualify; computed columns are left undocumented.
What is copied. The table's description, sections and columns (through the renames), and for a historian table its layout and tags, all marked PROPAGATED. A part a person wrote on the view is kept.
No right on the table is needed. The mapping comes from the view's SHOW VIEW text and the documentation from the business dictionary. The table's column list (from DBC or SHOW TABLE, when allowed) only helps tell a tag from a column the view leaves out.
Automatic. When the documentation walk (upload_features(document=True), rebuild_documentation, document_process_incremental, ...) meets a view that exposes a documented table with no computed column, it attaches it and copies the documentation instead of calling the LLM. A view the LLM documented before is switched on the next run. tdfs4ds.EXPOSED_VIEW_AUTO_ATTACH = False turns this off.
Kept up to date. set_object_documentation and set_deep_table_documentation on the table copy the new text onto its views and list the processes to re-document as stale. After other changes (an import, a direct load of the business dictionary), run tdfs4ds.refresh_exposed_views().
The pair is stored as the reserved section EXPOSES of the view. detach_exposed_view(view) ends it; the copied text stays.
Lineage. When the lineage stops at a table the user may not read but reads through such a view, an info line says so instead of a warning asking for a grant. The entitlement tables are not reported as undocumented.

Attached views and tables¶

A view that joins a registered dataset with a source table, read by no process, or a table brought in by hand, is outside the feature store: the agent called it "outside data" and a full rebuild_lineage_graph() closed the edges an extra_views= call had recorded. Attaching an object makes it part of the feature store.

tdfs4ds.suggest_attachments(databases=['<DB>'])                       # read only: VIEW, READS, ATTACHED
tdfs4ds.attach_objects(['<DB>.<VIEW>'], data_domain='<DOMAIN>', dry_run=True)   # report, writes nothing
tdfs4ds.attach_objects(['<DB>.<VIEW>'], data_domain='<DOMAIN>')
tdfs4ds.attached_objects()                                            # DATABASE_NAME, OBJECT_NAME, KIND, DATA_DOMAIN, DOCUMENTED_AT
tdfs4ds.refresh_attached_objects(dry_run=False)
tdfs4ds.detach_objects(['<DB>.<VIEW>'])

Refused: process views, datasets, FS_* / *_HIDDEN objects and names not found in the database.
Marker. The reserved section ATTACHED of the object (kind, optional data domain, fingerprint, documentation time). No new table.
Lineage. The edges of an attached view are recorded at attach, and every full rebuild_lineage_graph() keeps them. An attached table that no view reads has no edge until an attached view reads it.
Documentation. A pass-through column (a plain column of one FROM / JOIN relation, renamed or not, t.* too) takes the description of the source column, marked PROPAGATED. Computed columns and the object description are written by the LLM (skipped without an instruct model). A table is documented by its owner with set_object_documentation or a dictionary import. IMPORTED and MANUAL parts are never overwritten.
Refresh. refresh_attached_objects (also run by rebuild_documentation) compares a fingerprint: the normalised view text plus one hash per column, or the column types of a table. It is a dry run by default.

State	Meaning	Action
`unchanged`	nothing moved	skipped: no write, no LLM
`sources_changed`	a source's documentation changed after the last pass	pass-through columns copied again, no LLM
`changed`	view text or columns changed	added and changed columns documented, rows of removed columns ended, LLM only for a computed column or changed sources, lineage recorded again
`broken`	the object exists, `EXPLAIN` fails	reported, nothing written
`disappeared`	the object is gone	marker ended, lineage closed, documentation ended (readable at a past date), a person's parts kept and reported, row in the removal log

Agent. Attached objects are inside the feature store. The answer says "Attached to the feature store: X is an attached view ... It reads DS (dataset), T", and a dataset or process view they read is "used by the attached views ...". Search answers show attached objects in their own block before the outside block.
Guards. Removing a dataset or process read by an attached view is refused and the error names the view. The removal plan detaches the view; it never drops it. The workflow assessment counts attached views as consumers.

In the admin app¶

Documentation › Attached objects (admin view) has the same actions, in four cards:

Attached: the attached objects, with Sample and Detach on each row.
Check for changes gives the refresh states.
Apply the changes runs the refresh.
Attach: the objects are typed one per line. Choose a data domain and enter your name, then Preview.
The preview is a dry run. It lists, for each object, what it reads, the columns copied from their source, and the computed columns.
An object that cannot be attached shows the reason; the other objects are not affected.
Attach then writes. The LLM option is disabled, with its reason, when no model is configured.
Suggestions: the views of the given databases that read a dataset or a process view. Each one has Add to the list and Sample buttons.
Sample: 20 rows, read only after the query's EXPLAIN has been checked.
Why: SAMPLE applies to the view's result, so the whole view is computed.
Above the limits: the plan's estimated time is above 60 s (tdfs4ds.SAMPLE_MAX_SECONDS), or an intermediate result is above 10 GB (tdfs4ds.SAMPLE_MAX_SPOOL_GB), or a spool is too large for the optimizer to print (***).
- Confident estimate: nothing is read and the plan is shown. Sample anyway runs it.
- Low- or no-confidence estimate (statistics missing): a warning, and the sample runs. Such estimates can be far off: on a demo store, a dataset view of 9 categories was estimated at millions of hours and read in 2 s.
Cancel: a sample above the limits runs on its own database connection, shows its elapsed time, and has a Cancel button. Cancel stops the query in the database (connection.cancel(), Error 3110). A sample still running after 10 minutes is cancelled (tdfs4ds.SAMPLE_JOB_MAX_SECONDS).
Needs the TDFS4DS_TD_* settings: a cancellable run opens its own connection from them. Without them, a sample above the limits is refused, and Sample anyway reads it on the app's connection, which cannot be cancelled.
Only an attached object or a suggested view can be sampled, never an arbitrary name.

Documentation › Objects: the detail of a view or source table has Attach to the feature store…, which opens the panel with the name filled in, or Detach from the feature store.

Explore › Dataset builder: next to an entity's datasets it lists the views attached to them, each with the same EXPLAIN-checked sample.

Read it back¶

doc = tdfs4ds.get_documentation('DS_CUSTOMER')           # or a feature, a process, 'DB.OBJECT'
pd.DataFrame(doc['parts'])[['part', 'name', 'label', 'editor', 'db_user', 'since', 'text']]
doc['history']                                           # the latest logged actions on the object

label is one of:

Label	Meaning
imported	loaded from a data dictionary
manual	written or corrected by a person
propagated	copied from a feature to a dataset column
generated or unknown (before 0.3.2.1)	written by the LLM, or by anyone before this version
imported (before 0.3.2.1)	a source table documented before this version (the LLM never writes those)

Protect, or hand back to the LLM¶

# protect documentation written by hand before 0.3.2.1 (it carries no mark)
tdfs4ds.mark_documentation_manual('DB_LAB.FEAT_ENG_CUST', ['BUSINESS_LOGIC', 'COLUMN:sum_Transaction_Amount'])

# let the next LLM run rewrite a part again
tdfs4ds.release_manual_documentation('DS_CUSTOMER', ['SECTION:BUSINESS_QUESTIONS'])

The LLM functions also take overwrite_manual=True to replace every human part in one run: document_process_incremental, document_dataset_incremental, rebuild_documentation and cascade_redocumentation.

With an LLM¶

Human text is kept. document_process_incremental, document_dataset_incremental, rebuild_documentation and the admin app's Document action keep every imported or manual part. The LLM is given that text as authoritative context, so what it writes around it stays consistent.
Regenerating what depended on an edit. regenerate_dependents=True on a set_* call regenerates the documentation the LLM had derived from the edited text (report['stale']): the process's business logic, the dataset sections, downstream objects. With no instruct model configured, or with AGENTIC_FEATURES off, they are only listed.
Only the gaps. rebuild_documentation() (force_update=False) documents only what is not documented yet. A store documented by hand therefore keeps its text when an LLM is configured later.

The vector index¶

Each set_* call refreshes the index once, at the end (refresh_index=True):

Chroma re-embeds only the rows whose text changed.
The Teradata Vector Store refreshes its collections.

Without an embedding model the refresh is skipped and reported; the documentation is saved, and the next tdfs4ds.build_vector_index() indexes it.

In the admin app¶

Documentation › Objects → select an object → ✎ Edit documentation (admin view).

Each field shows who wrote it: imported, manual (with the editor and date), propagated, or generated.
Your name is required. It is recorded with the database user, because the app connects with one account for everybody, so the name is declared, not verified.
Review changes… shows what will be written, the dataset columns that receive a copy, the human copies left alone, the LLM-written documentation that becomes stale, and the index refresh. Nothing is written yet.
Confirm and save writes it. Options: copy to datasets, overwrite human copies, refresh the index, and regenerate with the LLM (disabled, with the reason, when no model is configured).
History lists the logged actions on the object, Protect and Release to LLM act on one field, and the Report has a Documentation activity section.

Where it is recorded¶

The six documentation tables are unchanged. tdfs4ds adds two optional tables, created by setup() and connect() when missing:

Table	Holds
`FS_DOCUMENTATION_PROVENANCE` (VALIDTIME; view `FS_V_DOCUMENTATION_PROVENANCE`)	one row per part written by a person or copied: source (`IMPORTED`, `MANUAL`, `PROPAGATED`), database user, declared editor, and the log event that set it. A part with no row was written by the LLM, or before 0.3.2.1.
`FS_DOCUMENTATION_LOG` (view `FS_V_DOCUMENTATION_LOG`)	one row per documentation action: manual edit, import, propagation, LLM documentation, protect, release, index refresh. Each row has its status, the database user, the declared editor, where it came from (`api`, `app`, `upload_features`, `rebuild_documentation`, `cascade`, `migration`) and the `RUN_ID` / `PROCESS_ID` that join it to `FS_FOLLOW_UP`.

tdfs4ds.documentation_log(target='DB_LAB.FEAT_ENG_CUST').to_pandas()

Two rules about these tables:

They are separate from FS_FOLLOW_UP, because every reader of the follow-up table treats a row as a process run.
They are optional. A user who cannot create them still documents; the edits are then not protected and not logged, and a warning says so.

Backward compatibility¶

Existing documentation is read and used exactly as before. It is shown as before 0.3.2.1, and it stays overwritable by the LLM until you protect it with mark_documentation_manual.
upload_business_dictionary_*(df) keeps its signature. Rows loaded through it are now tagged IMPORTED; pass source=None to record nothing, as before.
An older tdfs4ds client sharing the store does not know the provenance table. Its LLM runs can overwrite protected parts.
migrate_objects carries the provenance of the objects it promotes, so imported and hand-written documentation stays protected in the target store.