tdfs4ds.serve.doc_coverage¶

tdfs4ds.serve.doc_coverage ¶

Documentation coverage: what is documented, what is missing, what is in the vector index.

Backs the admin app's Documentation section (Coverage and Objects panels). Three layers, like efs_map.py:

collect_doc_inventory() -- the only function that reads the database or the vector index (one read per catalog / business-dictionary table / collection);
build_doc_model(...) -- pure: one row per object, plus the overview;
render_* -- scoped CSS + HTML, no <script>.

Objects counted: process views and their feature columns, datasets (description and sections), intermediate views and source tables of the lineage graph. The feature store's own technical objects (FS_T_* storage tables, _HIDDEN tables) are left out, with the rule documentation itself uses.

Vector index status:

Chroma -- exact. The expected documents come from the builders build_vector_index uses (vector_index.chroma_index_status), so a row reported indexed is one the next build would skip.
Teradata Vector Store -- inferred. No per-row membership can be read safely (the embeddings are keyed by TD_ID, a row position recomputed on refresh). A row counts as indexed when its collection is ready and the row was documented (ValidStart) before the collection's last_updated; a row documented after it is pending.

DATASET_SECTIONS `module-attribute` ¶

DATASET_SECTIONS = ['OVERVIEW', 'ENTITY', 'FEATURE_THEMES', 'BUSINESS_QUESTIONS', 'INTENDED_AUDIENCE']

KINDS `module-attribute` ¶

KINDS = ['process', 'feature', 'dataset', 'intermediate_view', 'source_table']

KIND_LABELS `module-attribute` ¶

KIND_LABELS = {'process': 'Processes', 'feature': 'Features (process outputs)', 'dataset': 'Datasets', 'intermediate_view': 'Intermediate views', 'source_table': 'Source tables'}

KIND_SINGULAR `module-attribute` ¶

KIND_SINGULAR = {'process': 'Process', 'feature': 'Feature', 'dataset': 'Dataset', 'intermediate_view': 'Intermediate view', 'source_table': 'Source table'}

SOURCES `module-attribute` ¶

SOURCES = ('objects', 'sections', 'columns')

_INDEX_ORDER `module-attribute` ¶

_INDEX_ORDER = ['collection_failed', 'missing', 'stale', 'pending', 'building', 'unknown', 'indexed_inferred', 'indexed']

INDEX_LABELS `module-attribute` ¶

INDEX_LABELS = {'indexed': 'indexed', 'indexed_inferred': 'indexed (inferred)', 'stale': 'stale (text changed)', 'pending': 'pending (documented after the last build)', 'missing': 'not indexed', 'collection_failed': 'collection failed', 'building': 'collection building', 'unknown': 'unknown', 'n/a': 'nothing to index'}

_BAD_INDEX `module-attribute` ¶

_BAD_INDEX = {'collection_failed', 'missing', 'stale', 'pending'}

_UPLOAD_HINT `module-attribute` ¶

_UPLOAD_HINT = 'tdfs4ds never generates documentation for source tables: write it in Objects > Edit, with tdfs4ds.set_object_documentation(database, name, description=..., columns={...}), or load an existing data dictionary with tdfs4ds.upload_business_dictionary_objects (and _columns).'

_CSS `module-attribute` ¶

_CSS = '<style>\n.doc-cov{--navy:#00233C;--orange:#FF5F00;--ok:#16a34a;--warn:#d97706;--fail:#dc2626;\n --muted:#64748b;--line:#E2E8F0;--chip:#F1F3F6;color:#1e293b;font-size:14px}\n.doc-cov h3{color:var(--navy);font-weight:600;margin:18px 0 8px;font-size:16px}\n.doc-cov .doc-kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(190px,1fr));gap:12px}\n.doc-cov .doc-kpi{border:1px solid var(--line);border-radius:10px;padding:12px 14px;background:#fff;min-width:0}\n.doc-cov .doc-kpi .lbl{font-size:12px;text-transform:uppercase;letter-spacing:.04em;color:var(--muted)}\n.doc-cov .doc-kpi .val{font-size:24px;font-weight:600;color:var(--navy);font-variant-numeric:tabular-nums}\n.doc-cov .doc-kpi .sub{font-size:12px;color:var(--muted)}\n.doc-cov .doc-bar{display:flex;height:8px;border-radius:4px;overflow:hidden;background:var(--chip);margin:8px 0 4px}\n.doc-cov .doc-bar span{display:block;height:100%}\n.doc-cov .b-ok{background:var(--ok)}.doc-cov .b-warn{background:var(--warn)}.doc-cov .b-fail{background:var(--fail)}\n.doc-cov .doc-chip{display:inline-block;padding:1px 8px;border-radius:999px;font-size:12px;\n border:1px solid var(--line);background:var(--chip);color:#334155;white-space:nowrap}\n.doc-cov .doc-chip.ok{background:#ecfdf5;border-color:#a7f3d0;color:#065f46}\n.doc-cov .doc-chip.warn{background:#fffbeb;border-color:#fde68a;color:#92400e}\n.doc-cov .doc-chip.fail{background:#fef2f2;border-color:#fecaca;color:#991b1b}\n.doc-cov .doc-wrap{overflow-x:auto}\n.doc-cov table{border-collapse:collapse;width:100%;font-size:13px}\n.doc-cov th{background:var(--navy);color:#fff;text-align:left;padding:6px 8px;font-weight:500}\n.doc-cov td{border-bottom:1px solid var(--line);padding:5px 8px;vertical-align:top}\n.doc-cov tr:nth-child(even) td{background:#F7F9FC}\n.doc-cov .doc-link{color:var(--navy);text-decoration:underline;cursor:pointer;background:none;border:0;\n padding:0;font:inherit;text-align:left}\n.doc-cov .doc-note{color:var(--muted);font-size:12px;margin:6px 0}\n.doc-cov .num{text-align:right;font-variant-numeric:tabular-nums}\n</style>'

_Lookup ¶

Business-dictionary rows by name: exact (db, name) first, else by name alone (the agent's readers ignore the database, so documentation stored under another database name still counts -- as it does for the agent).

init ¶

__init__(df, name_cols, text_col)

get ¶

get(db, *names)

_u ¶

_u(value) -> str

_txt ¶

_txt(value) -> str

_ts ¶

_ts(value) -> Optional[pd.Timestamp]

A comparable UTC timestamp, or None.

_sql_frame ¶

_sql_frame(sql, columns)

_primary_only ¶

_primary_only(df: DataFrame) -> pd.DataFrame

Keep the primary store's rows (see read_federation.primary_rows).

_with_store ¶

_with_store(df: DataFrame, store: str) -> pd.DataFrame

_read_only_bd ¶

_read_only_bd(view_const: str, columns: list, where: str = '') -> pd.DataFrame

One business-dictionary view read per attached read-only store (0.3.3.15), rows tagged FEATURE_STORE. Their views carry no ValidStart: the date a row was documented is then unknown. A store whose view cannot be read is skipped (logged): its objects show as undocumented rather than failing the panel.

_concat ¶

_concat(primary: DataFrame, others: DataFrame) -> pd.DataFrame

collect_doc_inventory ¶

collect_doc_inventory() -> dict

Read everything the model needs. Index reads are fail-open (reported, not raised).

_deep_tables ¶

_deep_tables(deep_layouts) -> dict

{(DB, NAME): {'layout', 'physical'}} of the tables declared deep (few; DBC read per table).

_existing_objects ¶

_existing_objects(graph) -> Optional[set]

{(DB, NAME)} of the lineage nodes that exist in DBC.TablesV, or None if unknown.

The lineage builder sometimes stores SQL tokens as objects (seen live: 1, APPLY, POSITION); without this check they would be reported as views missing documentation. One read per database, names filtered in Python.

collect_index_state ¶

collect_index_state(backend: str, frames: dict) -> dict

Per source: {'collection', 'exists', 'status', 'last_updated', 'rows', 'orphans', 'error'}.

With several stores read (frames tagged FEATURE_STORE), the primary's state stays at the top level and each read-only store's is under '_stores' ({STORE: {source: state}}): on Chroma each store has its own collections. On the Teradata backend a read-only store's index state is not read (its collections live in its own database): 'unknown'.

_rows_of ¶

_rows_of(df, store)

_index_state_one ¶

_index_state_one(backend: str, frames: dict, store) -> dict

row_id ¶

row_id(source: str, db, obj, sub=None) -> str

The id a business-dictionary row has in the vector index (same convention everywhere).

row_index_status ¶

row_index_status(source: str, rid: str, valid_start, index: dict, backend: str, store=None) -> str

Index status of one documented row (store: a read-only store's view database, whose own index state is in index['_stores']).

_worst ¶

_worst(statuses) -> str

build_doc_model ¶

build_doc_model(processes=None, datasets=None, graph=None, fs_tables=None, bd_objects=None, bd_sections=None, bd_columns=None, index=None, backend='chroma', schema=None, view_schema=None, existing_objects=None, catalog_features=None, deep_tables=None, stores=None) -> dict

One row per object plus the overview. Pure: no database access.

existing_objects ({(DB, NAME)} from DBC, or None when unknown): lineage nodes absent from it are not objects and are reported as phantom nodes. catalog_features: the feature catalog, to reconcile the feature count (process outputs here, (domain, entity, name) triplets in the catalog).

feature_note ¶

feature_note(catalog_features, processes) -> str

'362 in the catalog · 16 without a current producer · ...' - why the process-output count differs from the catalog's. Same rule as the Report.

_overview ¶

_overview(rows, index, backend) -> dict

gap_rows ¶

gap_rows(model: dict) -> list

Objects whose documentation or index entry is not complete.

_e ¶

_e(value) -> str

chip_class ¶

chip_class(status: str) -> str

_chip ¶

_chip(status, label=None) -> str

_name ¶

_name(r) -> str

render_coverage ¶

render_coverage(model: dict, max_rows: int = 200) -> str

render_object_header ¶

render_object_header(row: dict, inferred: bool) -> str

Summary block above an object's documentation.

gaps_csv ¶

gaps_csv(model: dict) -> str

tdfs4ds.serve.doc_coverage¶