tdfs4ds.genai.manual_documentation¶
Writing documentation by hand: features, processes, datasets, source data (0.3.2.1).
No LLM is needed for anything here. With an instruct model configured, the LLM remains the main author and a person corrects or completes it; without one, documentation is entirely written here. Either way:
- a part written here is marked MANUAL in
FS_DOCUMENTATION_PROVENANCEand is kept by every later LLM run (overwrite_manual=Trueon the LLM functions lets it be replaced); a load of an existing dictionary is IMPORTED (same protection); - a feature's description is written in BOTH places it lives - the process
documentation (read by dataset propagation) and the business dictionary on the
process view (read by the agent, the vector index and the admin app) - and,
with
propagate=True, copied to the column of every dataset exposing it (marked PROPAGATED; a dataset column a person wrote is left alone); - every call is logged in
FS_DOCUMENTATION_LOG(who, what, when, from where); - the report lists what was written, copied, skipped, the LLM-written
documentation that was derived from it (
stale), and the vector index refresh.regenerate_dependents=Trueregenerates that LLM documentation when an instruct model is configured; without one it is only listed.
dry_run=True returns the same report without writing anything (the admin
app's confirmation dialog shows it).
module-attribute
¶
DATASET_SECTIONS = ('OVERVIEW', 'ENTITY', 'FEATURE_THEMES', 'BUSINESS_QUESTIONS', 'INTENDED_AUDIENCE')
module-attribute
¶
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Split db.obj into (database, object), both de-quoted.
Quote-aware, so a dot inside a quoted part does not split the name:
split_qualified('"DEINF.CARLOSN"."FS_X"') == ('DEINF.CARLOSN', 'FS_X').
The naive rsplit('.', 1) this replaces returns ('"DEINF.CARLOSN"',
'"FS_X"') only by luck and ('"DEINF', 'CARLOSN"."FS_X"') in the
split('.') variant.
An unqualified name returns ('', name).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Whether LLM regeneration can run, and why not.
A process by PROCESS_ID or view name ('DB.VIEW' or 'VIEW').
A feature triplet (domain, entity, name) and the processes producing it.
{FEATURE_NAME_UPPER: catalog row} of the features a process produces.
Matched through the triplets the process produces (catalog_counts
producers), never by name within the data domain: a domain can hold the same
name for several entities, and a name match picked the wrong one (seen live).
A dataset by DATASET_ID, name or 'DB.NAME'.
_write_process_logic(process_id, business_logic=None, entity_description=None, entity_columns_json=None) -> str
Processes and datasets downstream of a view (persisted lineage).
Copy a table's new documentation onto the views declared to expose it (0.3.4.6).
_finish(report, refresh_index, dry_run, regenerate_dependents, process_ids=(), dataset_ids=(), cascade_from=())
set_feature_documentation(feature: str, description: str, *, entity=None, data_domain=None, propagate: bool = True, refresh_index: bool = True, regenerate_dependents: bool = False, overwrite_manual: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write the business description of one feature by hand.
The feature is resolved as the triplet (data domain, entity, name); a name
registered under several entities or domains raises with the candidates -
pass entity= / data_domain=. The text is written for every process
producing the feature (process documentation + business dictionary on the
process view), marked MANUAL, and copied to each dataset exposing it
(propagate). No LLM is used.
Example::
tdfs4ds.set_feature_documentation(
'nb_days_since_last_transactions',
'Days between the business date and the customer\'s latest purchase.',
entity='CustomerID', author='Jane Doe')
set_process_documentation(process: str, *, business_logic: Optional[str] = None, entity_description: Optional[str] = None, entity_columns: Optional[Dict[str, str]] = None, features: Optional[Dict[str, str]] = None, propagate: bool = True, refresh_index: bool = True, regenerate_dependents: bool = False, overwrite_manual: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write a process's documentation by hand (None keeps a part as it is).
process: PROCESS_ID or view name. business_logic: what the process
computes. entity_description: what one row (the entity) is.
entity_columns: {key column: description}. features:
{feature name: description} for the features this process produces.
Example::
tdfs4ds.set_process_documentation(
'DB_LAB.FEAT_ENG_CUST',
business_logic='Monthly spending profile of each customer.',
features={'mean_Transaction_Amount': 'Average amount of the customer\'s purchases.'})
set_dataset_documentation(dataset: str, *, overview: Optional[str] = None, sections: Optional[Dict[str, str]] = None, columns: Optional[Dict[str, str]] = None, refresh_index: bool = True, regenerate_dependents: bool = False, author: Optional[str] = None, dry_run: bool = False) -> dict
Write a dataset's documentation by hand (None keeps a part).
overview is the dataset's description (also its OVERVIEW section);
sections: {name: text} - OVERVIEW, ENTITY, FEATURE_THEMES,
BUSINESS_QUESTIONS, INTENDED_AUDIENCE or a name of your own; columns:
{column: description}.
set_object_documentation(database: str, name: str, *, description: Optional[str] = None, columns: Optional[Dict[str, str]] = None, sections: Optional[Dict[str, str]] = None, object_type: Optional[str] = None, source: str = 'MANUAL', refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Document a source table or a plain view by hand.
object_type: 'T' (table, default when unknown) or 'V'.
source='IMPORTED' when the text comes from an existing data dictionary.
Processes downstream of a deep table whose SQL (or an upstream view's) names one of tag_names.
Their LLM-written documentation was derived from the tag's old text: they are listed as stale, and only they are regenerated on request.
set_deep_table_documentation(database: str, table: str, *, id_columns=None, tag_column=None, payload_column=None, timestamp_column=None, timestamp_timezone=None, description: Optional[str] = None, columns: Optional[Dict[str, str]] = None, tags=None, tag_name_column=None, tag_text_column=None, unit_column=None, source: str = 'MANUAL', regenerate_dependents: bool = False, refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Document a deep (historian) table: its layout, its columns and each of its tags.
A deep table stores one row per (id columns, timestamp, tag) with the value in a payload column. Each tag - a value of the tag column, such as a sensor name - is documented as if it were a column of the pivoted table.
The layout (id_columns, tag_column, payload_column,
timestamp_column, timestamp_timezone - UTC by default) is required the
first time and read back afterwards, so a later call can pass tags= alone.
tags: {tag: description} or a DataFrame (tag_name_column,
tag_text_column, optional unit_column; by default its first two
columns). source='IMPORTED' for a historian tag list, 'MANUAL' for a
correction. columns documents the physical columns.
The report lists the processes whose SQL names a written tag (stale): their
documentation was written from the old text. regenerate_dependents=True
re-documents them when an instruct model is configured.
propagate_documentation(datasets=None, *, overwrite_manual: bool = False, refresh_index: bool = True, author: Optional[str] = None, dry_run: bool = False) -> dict
Fill dataset column descriptions from the feature documentation - no LLM.
datasets: a dataset (name / id) or a list; None = every dataset.
Columns a person wrote are kept unless overwrite_manual. This is how a
store without an LLM gets documented datasets.
For a deep table: layout, tags (names) and tag: True on each tag part. No-op otherwise.
get_documentation(target: str, kind: Optional[str] = None, *, entity=None, data_domain=None, history: int = 10) -> dict
Current documentation of a feature, process, dataset or object, with who wrote each part.
kind: 'feature' | 'process' | 'dataset' | 'object'; guessed when omitted
(process id or view, dataset, feature name, then 'DB.OBJECT'). Each part
carries source (MANUAL / IMPORTED / PROPAGATED, or 'LLM or legacy'),
label, editor (declared), db_user and since; history holds
the latest log events of the object.
mark_documentation_manual(targets, parts: Optional[Iterable[str]] = None, *, kind: Optional[str] = None, source: str = 'MANUAL', author: Optional[str] = None) -> dict
Protect existing documentation from LLM runs, without changing its text.
For documentation written by hand before 0.3.2.1 (it carries no mark).
parts narrows it: ['DESCRIPTION', 'BUSINESS_LOGIC', 'COLUMN:amount', 'SECTION:OVERVIEW'].
release_manual_documentation(target: str, parts: Optional[Iterable[str]] = None, *, kind: Optional[str] = None, author: Optional[str] = None) -> dict
Let the LLM rewrite parts again (closes their IMPORTED / MANUAL mark; history kept).