tdfs4ds.utils.execution_resources¶
Resource usage of process executions, kept beyond DBQL's retention (0.3.3.19).
Every ingestion runs under a query band (:mod:tdfs4ds.utils.query_band), so its
DBQL rows can be joined to its follow-up row. DBQL (DBC.QryLogV) is purged
after a few days on most systems, so the joined figures are copied into two
optional tables of the store:
FS_EXECUTION_RESOURCES (plain)
One row per execution (one follow-up row = one FilterManager segment or one
TimeManager date): the follow-up columns, the process version in force at
its valid time (FS_PROCESS_SQL, NULL before versioning), and the summed
DBQL figures. Keyed by EXECUTION_ID.
FS_EXECUTION_STATEMENTS (plain)
One row per DBQL statement of a stored execution: type, times, CPU, I/O,
spool, error. The SQL text is NOT copied (size, literals): it is read from
DBQL while DBQL still holds it.
:func:harvest_execution_resources copies, in the database, the executions
whose DBQL rows are complete (DBQL_STATUS = 'LOGGED') and not yet stored. It
runs at the start of run() / roll_out (never fatal), when the admin
app's Execution log opens, and from Maintenance. Readers return the stored rows
plus the live join for executions not stored yet, so a run that just ended is
listed at once (PENDING) and an old one survives DBQL's purge.
Everything is fail-open: a missing or unwritable table gives one warning and ingestion is never affected. Primary store only.
module-attribute
¶
METRICS = ('N_QUERIES', 'N_ERRORS', 'AMP_CPU_SEC', 'MAX_AMP_CPU_SEC', 'IMPACT_CPU_SEC', 'CPU_SKEW_PCT', 'PARSER_CPU_SEC', 'TOTAL_IO', 'PHYSICAL_IO', 'IO_SKEW_PCT', 'MAX_SPOOL_BYTES', 'FIRST_QUERY_START', 'LAST_RESPONSE')
module-attribute
¶
COLUMNS = ('EXECUTION_ID', 'RUN_ID', 'PROCESS_ID', 'EXEC_SEQ', 'PROCESS_TYPE', 'START_DATETIME', 'END_DATETIME', 'VALIDTIME_DATE', 'APPLIED_FILTER', 'STATUS', 'PROCESS_VERSION') + METRICS + ('HARVESTED_AT',)
module-attribute
¶
STATEMENT_COLUMNS = ('EXECUTION_ID', 'QUERY_ID', 'STATEMENT_TYPE', 'START_TIME', 'FIRST_RESP_TIME', 'AMP_CPU_SEC', 'MAX_AMP_CPU_SEC', 'N_AMPS', 'PARSER_CPU_SEC', 'TOTAL_IO', 'PHYSICAL_IO', 'SPOOL_BYTES', 'ERROR_CODE', 'ERROR_TEXT', 'HARVESTED_AT')
module-attribute
¶
NO_STORE_MESSAGE = 'no feature store is selected (tdfs4ds.SCHEMA is not set): choose one in Parameters, or call tdfs4ds.connect(database=...)'
Wrapper around the global logger that only emits logs when
tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.
Escape value for use inside a single-quoted SQL string literal.
For the DBC probes and catalog predicates that compare a name as data
(WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.
Forget what was learnt about the tables (tests, or after setup()).
Create the two tables (and views) when missing. Never raises.
existing is the lower-cased object set connect() read from DBC: tables
found there cost no CREATE. A failure is remembered and not retried.
(select_expr, join_clause) adding PROCESS_VERSION to a row set with
PROCESS_ID / VALIDTIME_DATE / START_DATETIME: the version whose period
contains the execution's valid time (its business date, else its start - the
runs_per_version rule). NULL without the FS_PROCESS_SQL table.
(insert_executions, insert_statements) - set-based, run in the database.
Days back to scan: from two days before the last stored execution, else None (everything DBQL still holds).
Copy the executions whose DBQL rows are complete into the history tables.
One INSERT ... SELECT per table, run in the database (no row comes to
the client); only executions not stored yet. days bounds the DBQL scan
(default: from two days before the last stored execution). dry_run=True
returns the SQL without running it. Never raises: returns {"ok",
"executions", "statements", "error", "sql"}.
The run() hook: harvest, at most once per :data:HARVEST_EVERY_SECONDS
in this Python process (a roll_out calls run() once per date), never
fatal, never noisy.
Follow-up rows alone (DBQL unreadable): runs are still counted.
Executions in the window (as_of - days, as_of] with their resource use.
Returns (frame, info): one row per execution (stored history first, then
the live follow-up <-> DBQL join for executions not stored yet), with
ELAPSED_SEC, PROCESS_VERSION and DBQL_STATUS (STORED /
LOGGED / PARTIAL / PENDING / NOT LOGGED / UNAVAILABLE).
info: {"state", "error", "hint", "history", "live"}. Never raises on
a DBQL failure: the follow-up alone is returned with the reason.
The statements of one execution: (frame, info).
DBQL first (it has the SQL text, from DBC.QryLogSQLV when readable),
else the stored copy (no SQL text). Never raises.
Per process (and per process version when by_version): executions, runs, failures, how often it runs, last run, run duration and consumption.
frame: :func:read_execution_resources' rows. A run groups the segments
of one execution (:func:~tdfs4ds.process_store.process_followup.execution_run_ids),
so frequency and per-run figures do not count FilterManager segments as
runs. Returns {pid: stats} or {pid: {version|None: stats}}. Pure.
'about every 1.0 day' / 'about every 6.0 h' / 'once' / ''. Pure.