tdfs4ds.genai.deep_tables¶
Deep (historian) source tables: documenting the table as if it were pivoted.
A deep table stores one row per (id columns, timestamp, tag) with the value in a payload column - a historian, an entity-attribute-value table. What a reader needs documented is each tag (each value of the tag column: a sensor, a signal), as if it were a column of the pivoted table.
Nothing here adds a table or alters one:
- the layout - which columns are the id columns, the tag, the payload and the
timestamp, and the timestamp's time zone - is a reserved section
DEEP_LAYOUTof the table in the business dictionary (JSON), like the agent'sFEATURE_GROUPS; - each tag is a
FS_BUSINESS_DICTIONARY_COLUMNSrow of the table whoseCOLUMN_NAMEis the tag value. A row is a tag when the table has a layout and the name is not a physical column. Provenance (IMPORTED / MANUAL), the log, the index and protection from LLM runs apply as to any column.
A table without a layout behaves exactly as before everywhere.
The LLM documenting a process that reads a deep table receives only the tags the
process's SQL names (its string and number literals), never the whole tag list:
see :func:scope_deep_context.
module-attribute
¶
module-attribute
¶
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Build db.obj with each part quoted only as needed.
qualified('TDB', 'FS_X') == 'TDB.FS_X' (unchanged for ordinary names);
qualified('SVC@CORP', 'FS_X') == '"SVC@CORP".FS_X'. An empty/None
database yields the bare object name, so unqualified callers keep working.
force=True quotes both parts unconditionally — only for the sites that
have always emitted quotes and whose exact text is already persisted (dataset
view DDL, returned volatile-table names).
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
'FEATURE_GROUPS', 'DEEP_LAYOUT' for a SECTION_NAME NOT IN (...) predicate.
build_layout(physical: Iterable[str], *, existing: Optional[dict] = None, id_columns=None, tag_column=None, payload_column=None, timestamp_column=None, timestamp_timezone: Optional[str] = None) -> dict
The layout of a deep table, checked against its physical columns.
Values not given are taken from existing (a later call can change one role). The tag, payload and timestamp columns are required; the id columns may be empty. Column names are returned with the table's own spelling.
The layout stored in the DEEP_LAYOUT section, or None (absent / unreadable).
One sentence for a reader or an LLM: the grain, where the value is, the time zone.
Why name cannot be stored as a tag, or None.
tags_from_input(tags, tag_name_column=None, tag_text_column=None, unit_column=None) -> Dict[str, str]
{tag: text} from a dict or a DataFrame (tag, description [, unit]).
Every string literal and bare number of sql, upper-cased, comments removed.
This is deliberately shape-blind: a tag can be named in a CASE, an IN list, a PIVOT, a join condition or a QUALIFY - the literal is what they share.
The documented tags that sql names, in the documented spelling.
scope_deep_context(ctx: dict, consumer_sql: str, layout: dict, physical: Iterable[str], max_tags: Optional[int] = None) -> dict
The upstream context of a deep table, reduced to what consumer_sql uses.
Physical columns keep their documentation; tags are kept only when the SQL
names them. When it names none (tags chosen through a join, or the tag kept
as a row), at most max_tags tags are kept so the LLM sees the shape without
the whole list. Returns a new dict with deep set; the input is unchanged.
String literals compared with the tag column (=, <>, IN (...)) that have no documentation.
Best effort, for a log line telling the user which tags to document next; it never decides what the LLM receives.
Prompt lines for a deep table (used by _format_upstream_context).
The stored layout of db.obj, or None. Fail-open (None) on any error.
The table's columns (DBC), in order. Raises when the object does not exist.
ctx scoped for consumer_sql when its object is a deep table; else ctx unchanged.
deep_table_tags(database: str, table: str, *, since=None, until=None, documented: Optional[bool] = None)
The tags present in a deep table, with their row counts and first/last timestamps.
One GROUP BY over the table, bounded by since / until on the timestamp
column when given (interpreted in the layout's time zone, UTC by default).
This reads the table: on a large historian, give a window. It is never run
by tdfs4ds on its own.
documented=False keeps the tags without documentation, True the
documented ones; None keeps all, with a DOCUMENTED column.