tdfs4ds.serve.panels¶
Gradio-free panel logic for the tdfs4ds admin web shell.
Each function here returns plain JSON-serialisable data or HTML strings — never
gradio objects — so they can back FastAPI routes (see
:mod:tdfs4ds.serve.app). The heavy lifting (Plotly figures, business-dictionary
queries, optimiser, evaluation) is delegated to the already-module-level builder
functions in :mod:tdfs4ds.agent.chatbot; this module only orchestrates them and
strips the Gradio presentation layer.
module-attribute
¶
_TD_MD_CSS = "<style>.td-md { font-family: system-ui, -apple-system, Segoe UI, sans-serif; color:#2c3e50; }.td-md h1, .td-md h2, .td-md h3 { color:#00233C; }.td-md h2 { border-bottom:2px solid #FF5F00; padding-bottom:4px; margin-top:18px; }.td-md h3 { margin-top:14px; }.td-md p { margin:6px 0; }.td-md pre { background:#0f172a; color:#e2e8f0; padding:12px 16px; border-radius:8px; overflow-x:auto; margin:8px 0; font-size:.85rem; }.td-md code { font-family:'Courier New', monospace; font-size:.88em; color:#c0392b; }.td-md pre code { color:#e2e8f0; background:transparent; }.td-md table { border-collapse:collapse; width:100%; margin:10px 0; font-size:.9rem; }.td-md th, .td-md td { border:1px solid #E3E8EE; padding:6px 12px; text-align:left; }.td-md th { background:#00233C; color:#fff; font-weight:600; }.td-md tr:nth-child(even) { background:#f8fafc; }.td-md blockquote { border-left:4px solid #FF5F00; margin:0; padding:4px 12px; background:#f8fafc; color:#555; }.td-md strong { color:#00233C; }.td-md th * { color:#fff; }.td-md ul, .td-md ol { margin:6px 0; padding-left:24px; }.td-md li { margin:2px 0; }</style>"
module-attribute
¶
module-attribute
¶
module-attribute
¶
_SECTION_ORDER = ['OVERVIEW', 'ENTITY', 'FEATURE_THEMES', 'BUSINESS_QUESTIONS', 'INTENDED_AUDIENCE']
module-attribute
¶
_DATASET_SECTION_LABELS = {'OVERVIEW': 'Overview', 'ENTITY': 'Entity', 'FEATURE_THEMES': 'Feature themes', 'BUSINESS_QUESTIONS': 'Business questions', 'INTENDED_AUDIENCE': 'Intended audience'}
module-attribute
¶
_DATASET_STATUS_COLOURS = {'OK': '#16a34a', 'REPAIR': '#FF5F00', 'REGISTER': '#2563eb', 'DUPLICATE': '#a16207', 'ORPHAN': '#dc2626', 'UNPARSEABLE': '#64748b'}
module-attribute
¶
_MAINTENANCE_LABELS = {'lineage': 'Lineage graph', 'documentation': 'Documentation', 'vector': 'Vector index', 'all': 'Lineage, documentation and vector index', 'statistics': 'Storage statistics', 'dataset_catalog': 'Dataset catalog', 'process_sql': 'Process SQL versions', 'resources': 'Execution history'}
module-attribute
¶
NO_STORE_STATUS = '⚠️ No feature store is selected. Choose one in Settings → Parameters, or set TDFS4DS_SCHEMA in the .env file passed to the app and restart it.'
module-attribute
¶
module-attribute
¶
_SWITCH_STATE = ('SCHEMA', 'SCHEMA_VIEW', 'DATA_DOMAIN', 'SCHEMA_READONLY', 'FOLLOW_UP_NAME', 'DATASET_CATALOG_NAME', 'DATA_DISTRIBUTION_TEMPORAL')
module-attribute
¶
Streamed live storage-count build: treemap + stacked bar + orphans + table.
progress is a callable(str) hook (per-table PROGRESS k/N lines).
module-attribute
¶
Actions of the Documentation panels.
op='document': document the objects named by keys (model row keys), or
every object whose documentation is missing or partial when keys is empty;
then, if refresh_index, update the vector index. op='index': update the
vector index only (build_vector_index(force_rebuild=False): Chroma embeds
only missing or changed rows; Teradata refreshes its collections).
module-attribute
¶
Preview (dry_run) or apply a hand edit from the Objects panel.
module-attribute
¶
Execute the staged migration (must call migration_stage first).
module-attribute
¶
Run the evaluation; return {results, report_html, radar_html, table_html}.
module-attribute
¶
Run the benchmark for dataset; return the rendered HTML report.
module-attribute
¶
The entity scan (one pass per entity over its largest storage table), then
the whole model. Streamed: progress gets PROGRESS k/N lines.
module-attribute
¶
What each step of the last plan would remove: counts and existence only.
'1. a? 2. b? 3. c' (one line) -> three lines, when the numbers follow on.
LLM-written documentation stores numbered questions on one line; Markdown then renders a single list item. Only a run starting at 1 and counting up is split, so a sentence quoting "version 2. something" is left alone.
Adapt LLM/GFM-style Markdown to Python-Markdown's stricter rules.
Python-Markdown (unlike GitHub, the chatbot and every LLM's habits) needs a blank line before a list, and 4 spaces to nest one; without them a list renders as one paragraph with literal dashes, and sub-items flatten. Also splits inline "1. … 2. …" enumerations. Fenced code is never touched.
Render a Markdown string to a styled, self-contained HTML fragment.
Uses the markdown package (declared in the serve extra) with the
fenced_code, tables and sane_lists extensions so GFM pipe tables,
fenced code blocks and lists render properly — unlike the minimal client-side
marked() in app.js. Scoped via the .td-md class. Fail-open: returns an
escaped <pre> block on any error so the panel never breaks, and says why
(the fallback was silent at DEBUG, so a server without the markdown
package showed raw Markdown everywhere with nothing in the log).
Domain + process lists for the storage-build cascading selectors.
{PROCESS_ID: {"DB.TABLE", ...}} - which storage tables each process writes.
Delegates to the package (feature_store.statistics.process_storage_tables),
the one authority shared with check_storage_statistics. Fail-open.
Full run-log table HTML for a single process (drill-down).
Consistency model for a snapshot, with the cached storage scan when it was taken at the same valid time. Fail-open: None.
Streamed storage scan (button 'Scan storage for orphans'); cached for the report and the EFS map at that valid time.
Assemble the plain data dict the inline-SVG report renders from.
Catalog + run-history + cached storage build → JSON-serialisable records (no
Plotly). Storage block is the last streamed build (_LAST_DASHBOARD_STORAGE)
or None (→ the report shows a 'build first' note). window_days bounds the
run-history timeline/calendar. as_of reads the catalogs at that valid time
(None = current); every count states the valid time it was read at.
Fail-open per section.
Latest documentation events + counts per action (FS_DOCUMENTATION_LOG). Fail-open.
Bare view name from a possibly-quoted FQN: '"db"."vw_x"' → 'vw_x'.
{TABLE_UPPER: {view_name, view_short, process_id}} — the first process that writes each storage table. Fail-open: {} on error.
Nested {table: {label, nonseq, processes:[{view, process_id, n_features, rows_share, features:[{feature, type, entity, orphan}]}]}} for tile drill-down.
Built in pandas from the already-loaded catalogs (no per-row DB query). Row share = table nonseq rows split evenly across the table's features.
Run history bounded to the last window_days.
Returns {rows, t_min, t_max, calendar, last_runs} where rows is a flat
list of {view, start, end, status, filter} (ISO strings) the inline-JS
timeline consumes; filter carries APPLIED_FILTER / VALIDTIME_DATE for the
hover. calendar is {view:{day:ok}}. Fail-open → None.
For each process, flag the stalest segment when it runs segmented.
Per process/dataset upstream + downstream counts from the persisted graph.
Render the exportable report inline for the panel.
Write the report to disk and return its path (for download).
Node ids ('DB.OBJECT') of the registered process views. Fail-open.
Return the lineage graph HTML + the node choices for inspection.
layout: 'flow' (default; left-to-right, zoomable, focus on click,
search, filters) or 'network' (free pyvis network). 'sankey' is
still accepted and now shows the Flow view: the Sankey did not scale.
db.name of every registered dataset absent from the lineage graph.
Record the lineage of datasets missing from the graph (scoped, diff-based).
datasets=None takes every registered dataset the graph lacks. A dataset
whose view is gone is reported, never fatal for the others.
The EFS map as HTML (+ KPIs). exact=True counts rows with COUNT(*);
as_of reads the catalogs at that valid time (None = current).
Rows per feature of one storage table, as HTML (bar chart).
Write the EFS map to a standalone HTML file and return its path (download).
The lineage graph used for the SQL chains, loaded once per catalog build (None if unreadable).
feature_catalog_page(page: int = 0, size: int = FEATURE_CATALOG_PAGE, q: str = '', domains=(), as_of: str = '', refresh: bool = False, processes=()) -> dict
One page of the feature catalog (feature x producing process) for the panel.
domains: names (or '|'-joined); empty = all. processes: PROCESS_IDs
(or '|'-joined); empty = all. q: text searched in feature, entity,
description and process view. as_of: blank = current.
processes_all lists the processes of the selected domains, for the filter.
{(domain, entity, name): [{pid, view}]} over the feature rows (rows without a producer left out).
The chain as the panel shows it: highlighted steps, the computation of feature in the process view, the inputs.
The SQL versions of a process for the feature report.
state: off (PROCESS_SQL_VERSIONING False), not_installed (no
FS_PROCESS_SQL on this store), none (nothing stored yet: before
0.3.3.12), ok. versions: one row per version, with the runs whose
valid time falls in it. drift: the live SQL differs from the current
stored version (captured at the next run or upload_features).
A process of the feature catalog with its SQL.
The SQL shown: the stored version when given; at a past valid time, the
process version in force then (FS_PROCESS_SQL); otherwise the live view
definitions (one SHOW VIEW per view). versions lists the stored process
versions (see :func:_process_versions).
With feature, the SQL comes back with the feature's computation and its
mentions highlighted, and computation gives each level's expression and
the grain (sql_context.computation_context).
{data_domain: 'db.view'} of the domains that have a clock (0.3.6.0). Fail-open: {}
(no FS_DATA_DOMAIN_SETTINGS table, an older store, or an unreadable one).
How a process stands with the clock of its data domain, for the process card, or None
when the domain has no clock: {clock, relation, text} with relation clock (reads the
clock itself), clone (its own soft clone of it), none (no TimeManager) or other
(another calendar). Fail-open: None.
A unified diff of two stored process versions, as text and highlighted HTML.
Capture the SQL beneath a process now (a new process version when it changed).
A 'no feature store selected' payload when SCHEMA is unset (the app can start connected but without a store), else None.
The Execution log panel: executions of the window with their resource
use, per-process and per-version statistics (serve/execution_log).
The statements of one execution (DBQL, else the stored history).
The feature catalog card's "Runs and consumption" block: the process's runs and resource use in the window ending at as_of, per process version, and the version in force at as_of.
Copy finished executions out of DBQL now (Maintenance / the panel's button).
The Execution log as an XLSX file (executions + processes); returns its path.
Write the export (with SQL) for the panel's filters; returns the path.
Feature rows that can go in a dataset (a producing process), current valid time.
The entities features can be combined on, per data domain, with their datasets.
The features of one entity: description, type, producing processes (latest run first), and the existing datasets on that entity with the features each one exposes.
May a dataset be created as name in the view database?
status: available, invalid (not a plain identifier), registered
(a dataset of the catalog), exists (a table or view of that name: creating
would replace it) or check_failed (never treated as free). Reads the
database now, never a cache: a dataset created a minute ago must count.
The dataset kind from the form: snapshot (current / as of / TimeManager) or history.
The form's choice as build_dataset's selected_features.
One process for a feature: {feature: PROCESS_ID}, the column keeps the
feature's name. Several processes (several versions side by side):
{feature: [{"process_id", "process_view_name"}, ...]}, one column per
version named <feature>_<process view> (the rule of build_dataset).
aliases {feature: {PROCESS_ID: column name}}: a version with a name of
its own gets alias (one version: {"process_id", "alias"}). Without
aliases the result is exactly what it was.
The column names a feature gets in the dataset: its name for one version,
<name>_<process view> for each of several (as build_dataset names them).
The form's offsets as build_dataset's feature_offsets ({} when none).
raw: {feature: "0, -1 day, +1 month"} (a comma list) or a list of
strings. Each value is checked here, so a typo is reported with the feature
it belongs to rather than as a failed creation. Raises ValueError.
The column names the form's choice gives, per feature, and the offset errors.
Same rule as build_dataset (version suffix, then offset suffix, then
_2/_3 for a repeat): tdfs4ds.dataset.builder.preview_column_names.
No database access beyond the cached builder rows (process view names).
Create (or, with preview, only render the SQL of) a dataset from the builder form.
payload: domain, entity, features ({feature name: PROCESS_ID}),
name, kind (current / as_of / time_manager / history) and its
parameters, join_type, document. The name is checked again here (the
button state is only a hint): an existing dataset or object is never replaced.
The Python that creates the form's dataset, and what each argument does.
The arguments come from the same parsing as :func:dataset_builder_create, so the
code is the call Create makes (skill_build_dataset passes them on unchanged).
Nothing is run. The password is never written: the code asks for it.
The registered datasets (all, or those on entity): kind, features, entity.
Same reading as the chat's dataset list (skill_list_datasets), read now:
the agent's per-turn cache is reset first, so a dataset created a minute ago shows.
Run sql (a SELECT, read only) and return {columns, rows} as text.
The SELECT of a REPLACE VIEW x AS <body> statement (what the view would return).
20 rows of a registered dataset (name), or of the dataset the builder
form describes (payload), before it is created: its query is run with
SAMPLE n and nothing is created.
name must be a registered dataset: the panel never runs a SELECT on an arbitrary object.
A short cause for a failed dataset read, from the database message's
wording (never from an error number alone). name: the dataset
(db.view), to tell its own view missing from an object it reads. Pure.
EXPLAIN SELECT * FROM every registered dataset (no data read): flags
those whose query no longer compiles, reads a missing object or is refused.
{(DB, OBJ)} of the named objects present in DBC.TablesV (one read per
database, raw SQL), or None when the dictionary cannot be read.
Remove a dataset of the primary store from the catalog (with its documentation, index entries and lineage); dry_run lists what would go.
Markdown blocks of an object's business documentation: description,
sections in _SECTION_ORDER, and a table of documented columns.
Shared by the Lineage node detail and the Documentation panels, so both show the same thing. A failed read is said, never silently dropped; when nothing is stored, a block says so.
Assemble DDL + business documentation for a lineage node as styled HTML.
The documentation is composed as Markdown (description, sections, a GFM pipe
table of documented columns, a fenced SQL DDL block) then rendered to HTML via
:func:_md_to_html_fragment so tables/lists/code render properly in the admin
shell (the client-side marked() cannot handle pipe tables).
The documentation model, collected once and cached until refreshed or changed.
Overview: documented / missing per kind, collection status, gap lists.
Who wrote the documentation: parts imported / edited by people, copies, the rest.
Every object as a JSON row (the panel filters client-side).
Summary + documentation of one object (key from the model rows).
Write the gaps (documentation and index) to a CSV file and return its path.
Write the documentation export (business dictionary, inside / outside the feature store) and return its path. Raises ValueError on a bad fmt/scope.
Generate documentation for rows (model rows), one object at a time.
Process (or one of its features) -> document_process_incremental; dataset -> document_dataset_incremental; intermediate view -> _document_intermediate_view over the persisted graph, as rebuild_documentation does. Source tables are never generated (their documentation is user-supplied) and are skipped.
Column names of a table or view (DBC), so undocumented columns can be written.
The editable parts of one object, with who wrote each (Objects panel > Edit).
Route the changed fields to the manual-documentation API.
Protect (mark as written by a person) or release one part to the LLM.
Object names from a list or from text (one per line, or comma-separated).
The attached objects, the data domains and whether the LLM can document.
Views of databases that read a dataset or a process view, attached or not (read only).
Preview (dry_run) or attach objects. Each object is attached on its own, so a
refused one (a process view, a dataset, a name not found) does not stop the others.
End the attachment of objects; their documentation stays.
n rows of an attached object or of a view that reads the feature store, read only after its EXPLAIN shows the query stays within the limits.
SAMPLE applies to the view's result, so the view's whole query runs: the
EXPLAIN of SELECT * FROM <obj> SAMPLE n is what the sample will cost. Above a
limit nothing is read and the plan is returned; force reads it anyway.
Only an attached object or a suggested view can be sampled, never an
arbitrary name.
Run sql in a worker thread on its OWN raw connection, so that
connection.cancel() stops it in the database (verified live: Error 3110,
the session idle within a second). Returns the job token.
State of a cancellable sample: running (with elapsed seconds), done (rows), cancelled, failed.
What changed in the attached objects (dry_run), or apply the refresh.
Comma-separated seed string -> list, or None (whole store) when blank.
Selectable seeds for the Execution plan panel: datasets + process views.
Compact HTML summary of the plan (waves, groups, conflicts, unify, unresolved).
Build the plan for the given seeds and render the interactive workflow + summary.
parallel_script_file(seeds: str | None = None, mode: str = 'process_pool', force_compute: bool = False, max_workers: Optional[int] = None, wave_on_failure: str = 'skip_descendants') -> Optional[str]
Build the plan and write the executable .py script; return its path.
Attached read-only stores available as migration sources (any schema name can also be typed — plan_migration attaches it read-only).
Source-scoped datasets + process views selectable as migration seeds.
migration_plan(source: str | None = None, seeds: str | None = None, target_data_domain: str | None = None, data_strategy: str = 'recompute') -> dict
Build the migration manifest and render the approval report + conflict controls. READ-ONLY (nothing is written).
migration_stage(source: str, seeds: str | None, target_data_domain: str | None, data_strategy: str, approvals: dict) -> dict
Re-plan freshly, validate the approvals, and stash {plan, approvals} for the streamed run. Returns the validation verdict.
migration_script_file(source: str, seeds: str | None = None, target_data_domain: str | None = None, data_strategy: str = 'recompute', approvals: dict | None = None) -> Optional[str]
Build the plan and write a standalone migration .py with baked-in approvals.
Run the optimiser; return the rendered report + a pending-replacement token.
Apply the approved REPLACE VIEW (uses the last optimisation's pending state).
Write the last optimisation report to a temp file and return its path.
eval_generate(skills=None, multi_turn=True, use_pronouns=True, per_skill=1, fact_first=True, temporal=True, seed=0, mcp_enabled=False) -> dict
Import scenarios JSON (replace/append) and merge into the current set.
Write the current scenario set to a temp JSON and return its path.
List datasets (+ auto-derived entity columns) for the Benchmark dropdown.
Re-render the most recent benchmark report (panel revisit).
Write the last benchmark result to a temp file for download; return path.
Forward 'tdfs4ds' logger records to emit(str) for the duration.
The rebuild cores already log per-object lines ("documenting X", "skipped",
"absent from graph — fallback", per-seed lineage, per-collection indexing).
Capturing them is how the Maintenance panel streams progress WITHOUT adding a
progress= param to those cores. Temporarily forces DISPLAY_LOGS=True so
the lines actually emit.
Current lineage / documentation / vector-index sizes for the status card.
Registered datasets with how they were built (Maintenance → Datasets card).
One row per dataset: name, strategy, join type and time mode. They come from
the build table; a dataset registered before it existed shows
not recorded (its strategy is still detected when converting).
dataset_grouped(view: str, in_place: bool = False, target: str = '', overwrite: bool = False, preview: bool = False) -> dict
Build the grouped version of a dataset — a new view, or in place.
preview=True only returns the DDL. The package logs the reason of a
refusal (target exists, not a dataset view); it is captured and returned so
the panel shows it instead of a bare failure.
The overview row naming qualified (db.name or bare name), or None.
Case-insensitive, quote-tolerant: the catalog stores raw names while the caller may pass a quoted or differently-cased one.
The dataset-catalog check/restore report as an HTML table (Maintenance panel).
The storage-statistics report as an HTML table (Maintenance panel).
The backfill report: one row per process, its status and version.
The stores the Maintenance vector-index step can re-index, with what the local index holds for each.
Every store found by the feature-store discovery (as in Parameters), the attached ones first and ticked. A store that is not attached can be indexed too: it is read through its public views, nothing is written to it. On the Teradata Vector Store backend the list is informative only (that backend indexes the primary, in its own database).
Wait for a background build of the same store to end (two builds of one store's collections at once would race).
Run a rebuild operation, streaming the cores' log lines via progress.
op ∈ {'lineage', 'documentation', 'vector', 'all'}. Flags in opts:
force_update, force_rebuild, all_data_domains, extra_views,
stores (the vector-index step on Chroma: the stores to re-index, a list
or a comma-separated string; default every attached store).
The run is shown in the status bar while it lasts.
Re-index the Chroma collections of each store in stores, one status-bar
job per store; {store: {'embedded', 'skipped', 'errors'}}.
A store whose build fails is reported and the next one still runs.
Current Teradata connection status for /health and the UI status dot.
Pre-fill values for the Connect form (password is never returned).
Open (or re-open) the Teradata context from the Connect form.
Return a 'not connected' payload when disconnected, else None.
For the few panels that work without a feature store (the store list, the Parameters toggles).
Return a payload naming what is missing (connection, feature store), else None.
Every panel that reads the store calls this. Without a store, its SQL would read
FROM .FS_FEATURE_CATALOG and fail with Error 3707, a syntax error that names
nothing (seen at a site whose .env set the connection but not TDFS4DS_SCHEMA).
Snapshot of the live tdfs4ds config used to pre-fill the Parameters form.
One check of the report as the Parameters panel shows it (status, detail, error, fix hint, and the time it took when it is noticeable).
Run tdfs4ds.check_configuration() and render it for the Parameters panel.
Returns the raw report plus an html fragment, so the panel shows the
per-check status and — the point of the feature — the fix hint for anything
that did not pass. progress is passed through ((event, data), see
diagnostics.check_configuration) so the panel shows each row as it comes.
The databases holding a feature store, for the Parameters list.
Looked for once at app startup and again on demand (refresh=True); the
result is kept so opening the panel costs nothing. Never raises: a failed
search returns the reason, and the panel then offers free text.
What the status bar shows: the connection, the stores read, and the jobs.
No database read: the store names are the session's, the jobs come from
tdfs4ds.serve.activity.
View databases of the attached read-only stores, in attach order.
Forget every panel result computed for the previous set of stores.
Features (DATA_DOMAIN, ENTITY_NAME, FEATURE_NAME) present in more than one attached store: the assistant's internal identity is store-blind, so each is treated as one (the primary's). 0 with a single store or on a read error.
Make db_name the primary feature store and, when readonly is given (a list of database names, possibly empty), read from exactly those stores too.
readonly=None keeps the stores already attached. The vector index of each
store is then brought up to date in the background (only missing or changed
rows are embedded: each store has its own collections, so nothing is wiped).
Switch the query band and the execution history on or off for this app process; when on, create the history tables if missing and say whether this user's runs can be measured (query band settable, DBQL readable and logging).
The model, with the cached entity scan when one exists (or a new scan).
{"html", "status", "summary"} of the cheap checks (+ the cached scan).
The assessment as html (standalone page), md or csv, written
under tdfs4ds.DASHBOARD_REPORT_DIR. Reuses the panel's last model when it
is recent and for the same scope (with its entity scan), else the cheap checks.
The Report's 'Workflow assessment' section: the panel's last model when it is recent and for every domain, else the cheap checks. Fail-open: ''.
The objects of the panel: a JSON list of process ids, dataset ids and
features [name, [entity], domain, process_id], or a comma list of ids.
The processes, datasets and features the panel lists (current catalogs).
The ordered removal plan of objects and of everything depending on them.
After the script ran: what each step of the last plan left behind (catalog rows, documentation, provenance, SQL versions, vector-index entries). Read only.
Write the script of the last plan; return its path (None without a plan).
Plan (read only) the relocation of features, or preview the rewrite of a dataset's table references, and show each step's dry run. Nothing is changed; the script does it.
Write the script of the last relocation plan; return its path (None without a plan).
Show a streamed panel job in the status bar while it runs.
The job's own progress messages become the bar's detail, and a
PROGRESS k/N (or PHASE k/N) line its count. A returned status
starting with ❌ marks the job failed; so does an exception (re-raised).