tdfs4ds.documentation_export¶

tdfs4ds.documentation_export ¶

Business-dictionary documentation, inside and outside the feature store.

Two jobs share one rule, which this module is the single authority for: is a documented table or view part of the feature store, or outside it?

inside: a registered process view (its columns are features), a registered dataset, a Time/FilterManager, a feature-store internal object (FS_*, *_HIDDEN);
outside: every other documented table or view - source tables, intermediate views. They are documented data, not governed features: no data domain, no entity, no refresh history, and relationships between their columns are not certified. An outside object can still feed feature processes (lineage), and the export and the agent say which.

:func:feature_store_index / :func:object_scope / :func:feature_processes_reading are pure and also used by the consumer agent. The export follows the catalog_export layers:

:func:collect_documentation - the only I/O;
:func:build_documentation_model - pure;
:func:render_documentation_html / :func:write_documentation_xlsx.

Public: :func:export_documentation (a file, with what is not documented yet in yellow) and :func:import_documentation (the filled-in workbook back into the business dictionary, through the hand-documentation API).

all `module-attribute` ¶

__all__ = ['export_documentation', 'import_documentation', 'read_documentation_workbook', 'plan_documentation_import', 'collect_documentation', 'build_documentation_model', 'render_documentation_html', 'write_documentation_xlsx', 'feature_store_index', 'object_scope', 'feature_processes_reading', 'SCOPE_INSIDE', 'SCOPE_OUTSIDE']

SCOPE_INSIDE `module-attribute` ¶

SCOPE_INSIDE = 'feature store'

SCOPE_OUTSIDE `module-attribute` ¶

SCOPE_OUTSIDE = 'outside the feature store'

SCOPES `module-attribute` ¶

SCOPES = {'all': None, 'feature_store': SCOPE_INSIDE, 'inside': SCOPE_INSIDE, 'outside': SCOPE_OUTSIDE}

KIND_PROCESS `module-attribute` ¶

KIND_PROCESS = 'process view'

KIND_DATASET `module-attribute` ¶

KIND_DATASET = 'dataset'

KIND_MANAGER `module-attribute` ¶

KIND_MANAGER = 'Time/FilterManager'

KIND_INTERNAL `module-attribute` ¶

KIND_INTERNAL = 'feature store internal object'

KIND_TABLE `module-attribute` ¶

KIND_TABLE = 'table'

KIND_VIEW `module-attribute` ¶

KIND_VIEW = 'view'

KIND_OBJECT `module-attribute` ¶

KIND_OBJECT = 'table or view'

KIND_ATTACHED_VIEW `module-attribute` ¶

KIND_ATTACHED_VIEW = 'attached view'

KIND_ATTACHED_TABLE `module-attribute` ¶

KIND_ATTACHED_TABLE = 'attached table'

_KIND_ORDER `module-attribute` ¶

_KIND_ORDER = [KIND_DATASET, KIND_PROCESS, KIND_ATTACHED_VIEW, KIND_ATTACHED_TABLE, KIND_MANAGER, KIND_INTERNAL, KIND_TABLE, KIND_VIEW, KIND_OBJECT]

OUTSIDE_NOTE `module-attribute` ¶

OUTSIDE_NOTE = 'Outside the feature store: documented data, not governed features. These objects have no data domain, no entity and no refresh history in the feature store, and the relationships between their columns are not certified.'

UNKNOWN `module-attribute` ¶

UNKNOWN = 'unknown'

NONE `module-attribute` ¶

NONE = 'none'

OBJECT_COLUMNS `module-attribute` ¶

OBJECT_COLUMNS = ['SCOPE', 'KIND', 'FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'STATUS', 'DESCRIPTION', 'DESCRIPTION_SOURCE', 'DOCUMENTED_COLUMNS', 'MISSING_COLUMNS', 'SECTIONS', 'READ_BY_FEATURE_PROCESSES', 'DOCUMENTED_SINCE']

COLUMN_COLUMNS `module-attribute` ¶

COLUMN_COLUMNS = ['SCOPE', 'KIND', 'FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'COLUMN_NAME', 'COLUMN_ROLE', 'STATUS', 'DESCRIPTION', 'DESCRIPTION_SOURCE', 'DOCUMENTED_SINCE']

SECTION_COLUMNS `module-attribute` ¶

SECTION_COLUMNS = ['SCOPE', 'KIND', 'FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'SECTION_NAME', 'STATUS', 'CONTENT', 'DESCRIPTION_SOURCE', 'DOCUMENTED_SINCE']

STATUS_DOCUMENTED `module-attribute` ¶

STATUS_DOCUMENTED = 'documented'

STATUS_MISSING `module-attribute` ¶

STATUS_MISSING = 'missing'

_SOURCE_LABELS `module-attribute` ¶

_SOURCE_LABELS = {'MANUAL': 'written by hand', 'IMPORTED': 'imported', 'PROPAGATED': 'propagated'}

_SECTION_ORDER `module-attribute` ¶

_SECTION_ORDER = ['OVERVIEW', 'ENTITY', 'FEATURE_THEMES', 'BUSINESS_QUESTIONS', 'INTENDED_AUDIENCE']

_REPORTED_KINDS `module-attribute` ¶

_REPORTED_KINDS = (KIND_PROCESS, KIND_DATASET, KIND_TABLE, KIND_VIEW, KIND_OBJECT)

_CSS `module-attribute` ¶

_CSS = '\n.td-dx{font-family:Inter,Arial,sans-serif;color:#14273A;background:#fff;margin:0;padding:24px 32px;line-height:1.45}\n.td-dx h1{color:#00233C;font-size:26px;margin:0 0 4px}.td-dx h2{color:#00233C;font-size:20px;margin:28px 0 10px}\n.td-dx .sub{color:#5A6B7B;margin:0 0 16px}.td-dx .bar{height:4px;background:#FF5F00;width:80px;margin:6px 0 18px}\n.td-dx table{border-collapse:collapse;width:100%;font-size:13px;margin:6px 0}\n.td-dx th{background:#00233C;color:#fff;text-align:left;padding:6px 8px}\n.td-dx td{border-bottom:1px solid #E2E8F0;padding:6px 8px;vertical-align:top}\n.td-dx tr:nth-child(even) td{background:#F5F7FA}.td-dx .u{color:#9AA9B8;font-style:italic}\n.td-dx .kv td:first-child{font-weight:600;width:240px;color:#00233C}\n.td-dx .obj{border:1px solid #E2E8F0;border-radius:10px;padding:12px 16px;margin:12px 0}\n.td-dx .obj h3{margin:0 0 6px;color:#00233C;font-size:16px}\n.td-dx .chip{display:inline-block;padding:1px 9px;border-radius:10px;font-size:12px;margin-right:6px;color:#fff}\n.td-dx .in{background:#0D9488}.td-dx .out{background:#64748B}.td-dx .kind{background:#00233C}\n.td-dx .note{background:#FFF1E8;border-left:4px solid #FF5F00;padding:8px 12px;margin:8px 0}\n.td-dx .sec{margin:6px 0}.td-dx .sec b{color:#00233C}.td-dx a{color:#B84400}\n.td-dx .toc li{margin:2px 0}\n.td-dx .miss{background:#FFF59D;color:#5C4A00;padding:1px 6px;border-radius:4px;font-style:italic}\n'

_MISS `module-attribute` ¶

_MISS = '<span class="miss">missing</span>'

IMPORT_HELP `module-attribute` ¶

IMPORT_HELP = "Write the text in the DESCRIPTION (Objects, Columns) or CONTENT (Sections) cells, yellow or not, keep the other columns as they are, save, then run tdfs4ds.import_documentation('<this file>') to see what would change, and tdfs4ds.import_documentation('<this file>', dry_run=False, author='<your name>') to write it. Only changed, non-empty texts are written (an emptied cell deletes nothing); objects of a read-only feature store, managers and internal objects are skipped."

_REQUIRED `module-attribute` ¶

_REQUIRED = {SHEET_OBJECTS: ['FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'DESCRIPTION'], SHEET_COLUMNS: ['FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'COLUMN_NAME', 'DESCRIPTION'], SHEET_SECTIONS: ['FEATURE_STORE', 'DATABASE_NAME', 'OBJECT_NAME', 'SECTION_NAME', 'CONTENT']}

_RESERVED_IMPORT_SECTIONS `module-attribute` ¶

_RESERVED_IMPORT_SECTIONS = {'DEEP_LAYOUT', 'FEATURE_GROUPS'}

_s ¶

_s(v) -> str

_u ¶

_u(v) -> str

_split ¶

_split(name, db=None)

(DB_U, NAME_U) of a possibly qualified name; DB '' when unknown.

feature_store_index ¶

feature_store_index(processes=None, datasets=None, attached=None) -> dict

{"fq": {(DB, NAME): kind}, "bare": {NAME: kind}} of the feature-store objects.

processes: process catalog rows (VIEW_NAME, qualified or not; the FILTER_* / TIME_* manager columns when present). datasets: dataset catalog rows (DATASET_DATABASE, DATASET_NAME). A name read without its database goes to bare and then matches whatever the database. attached: {(DB, OBJ): 'V' | 'T'} of the attached objects (attached_objects.attached_index, 0.3.5.2): inside, as attached view / table.

_internal ¶

_internal(name_u: str) -> bool

object_scope ¶

object_scope(db, name, index: dict, object_type=None) -> tuple

(scope, kind) of a documented object. Pure.

object_type is the business dictionary's OBJECT_TYPE ('T' / 'V' / 'D'), used only to name an outside object's kind.

graph_lookup ¶

graph_lookup(graph) -> dict

{(DB_U, NAME_U): node} of a lineage DiGraph (nodes carry database/object).

feature_processes_reading ¶

feature_processes_reading(graph, db, name, index: dict, lookup: Optional[dict] = None) -> list

Process views (DB.VIEW) that read db.name, directly or through views. Pure.

Lineage edges go consumer -> source, so the readers are the ancestors. [] when the graph is missing or does not hold the object.

collect_documentation ¶

collect_documentation(as_of=None, include_missing: bool = True) -> dict

The business dictionary, the catalogs and the lineage graph, at one valid time.

as_of None = current valid time, every attached store (read-only ones through their public views); a date = the primary store's base tables at that time. include_missing: also read from DBC which objects exist and their columns, so undocumented ones are listed (DBC is read now, at any as_of). A failed read is recorded in errors; its values become unknown.

_candidate_objects ¶

_candidate_objects(processes, datasets, graph, *frames) -> set

{(DB_U, NAME_U)} whose documentation the export reports, documented or not: the process views, the datasets, the tables and views of the lineage graph (managers excluded) and every object the dictionary documents.

_read_dbc ¶

_read_dbc(candidates: set, errors: dict) -> dict

Existence, kind and columns of candidates: two DBC reads per database (batches of 500 names). Raw SQL: a DataFrame over DBC needs WITH GRANT OPTION.

_ts ¶

_ts(v) -> str

_source_label ¶

_source_label(prov: dict, key, object_type, has_provenance: bool) -> str

_features_of ¶

_features_of(processes) -> dict

{(DB, VIEW): {FEATURE_U}} from the process catalog's FEATURE_NAMES.

_wanted ¶

_wanted(values) -> Optional[set]

column_role ¶

column_role(kind: str, column, features: set) -> str

What a column is: a feature or an entity key of a process view, a dataset column, a column.

build_documentation_model ¶

build_documentation_model(data: dict, scope: str = 'all', databases=None, objects=None, include_missing: bool = True) -> dict

Objects, Columns and Sections tables, each row tagged inside / outside. Pure.

scope: 'all', 'feature_store' (or 'inside') or 'outside'. databases / objects: keep only those (names, case-insensitive; objects accepts db.name or a bare name). include_missing: also list what is not documented (STATUS='missing', empty text): the process views, datasets and lineage tables/views found in DBC without a description, their undocumented columns, and the standard sections a dataset lacks. Needs data['dbc'] (collect_documentation).

_h ¶

_h(v) -> str

_anchor ¶

_anchor(r) -> str

_key ¶

_key(r) -> tuple

render_documentation_html ¶

render_documentation_html(model: dict) -> str

One self-contained page (no script, no external request).

write_documentation_xlsx ¶

write_documentation_xlsx(model: dict, path: str) -> str

Sheets About, Objects, Columns and Sections (Excel tables); returns path.

_norm_text ¶

_norm_text(v) -> str

read_documentation_workbook ¶

read_documentation_workbook(path: str) -> dict

{sheet: DataFrame} of an exported workbook (every cell as text).

Raises ValueError naming a missing sheet or column: the file must keep the export's sheets and headers (other columns and the row order may change).

plan_documentation_import ¶

plan_documentation_import(sheets: dict, current: dict, index: dict, primary_store: str) -> dict

What an import would write. Pure.

current: a model of the documentation now (build_documentation_model). A text is written when it is not empty and differs from the current one (whitespace ignored); an empty cell writes nothing (never a deletion). {"objects": {(DB_U, OBJ_U): {database, object, kind, description, columns, sections}}, "skipped": [{object, part, name, reason}], "unchanged": n}.

import_documentation ¶

import_documentation(path: str, *, dry_run: bool = True, author: Optional[str] = None, refresh_index: bool = True) -> dict

Write back the documentation filled in an exported workbook.

Reads a workbook written by :func:export_documentation (XLSX; sheets Objects, Columns and Sections, headers unchanged), compares each text with the current documentation and writes the new or changed ones through the hand-documentation API, so they are marked written by hand (MANUAL, with author) and LLM runs keep them:

a process view: its description = entity_description, a feature column = features (also written to the process documentation and copied to the datasets exposing it), another column = entity_columns;
a dataset: description = overview, sections, columns;
a table or view outside the feature store: set_object_documentation.

Never deletes: an empty cell is ignored. Skipped (listed in the report): objects of a read-only feature store, Time/FilterManagers and internal objects, sections of a process view, reserved sections. dry_run=True (default) writes nothing. The vector index is refreshed once at the end.

Returns {"dry_run", "written": [...], "skipped": [...], "unchanged": n, "errors": [{object, error}], "index"}.

_default_path ¶

_default_path(fmt: str, scope: str, as_of_literal) -> str

export_documentation ¶

export_documentation(path: Optional[str] = None, *, fmt: Optional[str] = None, scope: str = 'all', databases=None, objects=None, as_of=None) -> str

Write the business-dictionary documentation to an XLSX or HTML file.

Every documented table and view, its description, its documented columns and its sections, each tagged in the feature store (process views, datasets, managers) or outside the feature store (source tables and other views; for those, the feature processes reading them, from lineage).

Parameters¶

path : str, optional Where to write. Default: tdfs4ds.DASHBOARD_REPORT_DIR / tdfs4ds_documentation_<store>[_<scope>][_<date>]_<timestamp>.<fmt>. fmt : {'xlsx', 'html'}, optional Taken from the extension of path when omitted; 'xlsx' by default. scope : {'all', 'feature_store', 'outside'} Which objects to export. databases, objects : str or list of str, optional Keep only these databases / objects (db.name or a bare name). as_of : str or datetime, optional None = current documentation of every attached store; a date = the primary store's documentation at that valid time.

Returns¶

str The path written.

tdfs4ds.documentation_export¶