tdfs4ds.utils.query_band¶

tdfs4ds.utils.query_band ¶

Query band for process executions, and the follow-up ↔ DBQL join.

Every query of one process execution (the follow-up INSERT, the ingestion CTAS, the MERGE, the statistics refresh, the follow-up UPDATE) runs with a session query band::

TDFS4DS_RUN_ID=<RUN_ID>;TDFS4DS_PROCESS_ID=<PROCESS_ID>;TDFS4DS_EXEC_SEQ=<n>;
TDFS4DS_EXECUTION_ID=<uuid>;TDFS4DS_VALIDTIME=<time or CURRENT>;TDFS4DS_VERSION=<v>;

so DBQL (DBC.QryLogV.QueryBand) can be joined to FS_FOLLOW_UP to get the resources each execution used: AMP CPU, parser CPU, I/O, spool.

The join key. FS_FOLLOW_UP has no execution id, and one cannot be added: its INSERT … SELECT lists no columns, so a new column makes every older tdfs4ds client sharing the store fail with Error 3812 (verified). The row is identified instead by (RUN_ID, PROCESS_ID) and its rank by START_DATETIME — TDFS4DS_EXEC_SEQ is that rank, counted in this Python process (RUN_ID is generated once per import, so all of a run's rows come from here). A gap in DBQL (logging switched on mid-run) therefore does not shift the pairing. Known limits: an execution whose follow-up INSERT failed leaves no row and shifts the later ones of the same run; two executions of the same process starting within the same second (START_DATETIME has whole-second precision) may swap.

The caller's own band is kept. Our pairs are added with UPDATE FOR SESSION (which keeps the other pairs) and the session band read beforehand is restored afterwards — SET … = 'NAME=NONE' would store the literal NONE rather than remove the pair (verified).

Never fatal: if the band cannot be set, the execution runs untagged and a warning says so once.

PREFIX `module-attribute` ¶

PREFIX = 'TDFS4DS_'

NAMES `module-attribute` ¶

NAMES = ('RUN_ID', 'PROCESS_ID', 'EXEC_SEQ', 'EXECUTION_ID', 'VALIDTIME', 'VERSION')

_SEQ `module-attribute` ¶

_SEQ: dict = {}

_LOCK `module-attribute` ¶

_LOCK = Lock()

_WARNED `module-attribute` ¶

_WARNED = {'set': False}

DEFAULT_FLUSH_MINUTES `module-attribute` ¶

DEFAULT_FLUSH_MINUTES = 10

logger_safe ¶

logger_safe(level, message, *args, **kwargs)

Wrapper around the global logger that only emits logs when tdfs4ds.DISPLAY_LOGS is True. level is a string like "info", "error", etc.

sql_literal ¶

sql_literal(value)

Escape value for use inside a single-quoted SQL string literal.

For the DBC probes and catalog predicates that compare a name as data (WHERE DatabaseName = '…'). The caller supplies the surrounding quotes.

_run_sql ¶

_run_sql(sql)

_clean_value ¶

_clean_value(value, limit=128) -> str

A query band value may not contain ; or =, nor a quote here.

session_band_from_get_query_band ¶

session_band_from_get_query_band(text) -> str

The SESSION part of GetQueryBand()'s =S> a=b;=T> c=d; text.

build_query_band ¶

build_query_band(pairs: dict) -> str

next_exec_seq ¶

next_exec_seq(run_id, process_id) -> int

execution_pairs ¶

execution_pairs(process_id, run_id=None, feature_store_time=None, seq=None) -> dict

process_query_band ¶

process_query_band(process_id)

Tag the queries of one process execution; restore the session band after.

Yields the pairs that were set, or None when tagging is off or failed.

_restore_session_band ¶

_restore_session_band(previous) -> None

Put the session band read before back (NONE when there was none).

forced_incremental_planning ¶

forced_incremental_planning()

DynamicPlan=SYSTEMX for the session while the block runs, then the previous band. A DYNAMIC EXPLAIN then shows the plan built with the values a subquery returns (a FilterManager's segment), which a static plan leaves as parameters: seen live, a filter on a partitioning column read all rows statically and 365 of 2,555 partitions with incremental planning. The request must be EXPLAIN-only: the band is a planning hint, not a test switch. Yields True when the band was set.

_band_value ¶

_band_value(name, width=255, column='QueryBand') -> str

dbql_unreadable_hint ¶

dbql_unreadable_hint(user='<user>') -> str

What to ask for when DBC.QryLogV cannot be read (one wording, also used by check_configuration).

dbql_not_logged_hint ¶

dbql_not_logged_hint(user='<user>') -> str

What to ask for when nothing of a user is logged in DBQL.

query_band_capability ¶

query_band_capability(run, user: str = '<user>') -> dict

Can this session tag its queries, and read them back from DBQL?

run: run(sql) -> rows (a teradataml or a raw teradatasql connection). Returns {"band": bool, "dbql_readable": bool | None, "dbql_queries": int | None, "last": ..., "error": str | None, "hint": str | None}. Only a refused SET QUERY_BAND makes band False; DBQL unreadable or empty leaves the runs working, and the hint names the grant / logging statement to ask for. The session band is restored. Never raises.

days_ago_sql ¶

days_ago_sql(days) -> str

CURRENT_TIMESTAMP minus days days. INTERVAL 'n' DAY takes 2 digits (99 days at most) and DAY(4) inside the literal is Error 3706, so a longer span goes through a CAST.

_in_list ¶

_in_list(values) -> str

followup_resource_usage_sql ¶

followup_resource_usage_sql(process_id=None, run_id=None, days=None, flush_minutes=DEFAULT_FLUSH_MINUTES, only_logged=False, process_ids=None) -> str

The SELECT joining FS_V_FOLLOW_UP to DBC.QryLogV per execution.

One row per follow-up execution: all its follow-up columns, then the resource usage of its queries, summed over them:

N_QUERIES, N_ERRORS;
CPU — AMP_CPU_SEC (total), MAX_AMP_CPU_SEC (busiest AMP of the heaviest query), IMPACT_CPU_SEC (busiest AMP × active AMPs: what the system effectively spent), CPU_SKEW_PCT = 1 − AMP_CPU / IMPACT_CPU (0 = even), PARSER_CPU_SEC;
I/O — TOTAL_IO (logical), PHYSICAL_IO, IO_SKEW_PCT (same rule on MaxAmpIO);
spool — MAX_SPOOL_BYTES (peak of any one query);
FIRST_QUERY_START, LAST_RESPONSE (DBQL times, system clock);

and DBQL_STATUS:

LOGGED — its DBQL rows were found, and it ended long enough ago for all of them to be written;
PARTIAL — rows were found, but the execution is still running or ended less than flush_minutes ago: the totals may still grow;
PENDING — nothing found yet, and its last activity (end, or start when it never ended) is less than flush_minutes old: DBQL writes from a cache, flushed every 10 minutes by default (DBQLFlushRate);
NOT LOGGED — older than that and still absent: logging was off for the user, the rule excluded these queries, or the run was not tagged. A row left RUNNING by a crash lands here once its start is old enough, since its first query (the follow-up insert) would have been flushed.

days bounds both the follow-up rows and the DBQL scan; only_logged=True keeps the LOGGED rows only; process_ids (a list) restricts to several processes.

followup_resource_usage ¶

followup_resource_usage(process_id=None, run_id=None, days=30, flush_minutes=DEFAULT_FLUSH_MINUTES, only_logged=False)

Resources used by each process execution: follow-up rows joined to DBQL.

Needs SELECT on DBC.QryLogV and DBQL logging for the user that ran the processes (tdfs4ds.check_configuration() checks both).

DBQL is not immediate. Logged queries sit in a cache and are written to the DBQL tables at the flush interval (DBQLFlushRate, 10 minutes by default), so an execution that just ended has no DBQL row yet. Every follow-up row is returned with DBQL_STATUS — LOGGED, PARTIAL (totals may still grow), PENDING (nothing written yet: ask again later) or NOT LOGGED — so a missing or incomplete measure is never silent. Set flush_minutes to the system's flush rate; a DBA can also force the write with FLUSH QUERY LOGGING WITH ALL;.

Returns a pandas DataFrame.

create_followup_resource_view ¶

create_followup_resource_view(view_name='FS_V_FOLLOW_UP_RESOURCES')

Create the join as a view in the view database (no days bound).

A view over DBC.QryLogV works only when its database holds SELECT ... WITH GRANT OPTION on it — usually a DBA grant. Returns the view name, or raises with the database message.

tdfs4ds.utils.query_band¶