tdfs4ds.agent.sql_context¶

tdfs4ds.agent.sql_context ¶

Where a feature's expression is computed: grain and filters (0.3.1.240).

An EXPLAIN answer gives the formula of a feature; this module says in which context the formula is evaluated, read from the process view DDL by code:

the grain: the GROUP BY keys of the aggregation it sits in, or the window (PARTITION BY / ORDER BY) of an analytic function;
the filters deciding which rows reach it - explicit (WHERE, HAVING, QUALIFY) and implicit: an INNER join keeps only rows with a match, a semi-join (IN (SELECT ...) / EXISTS) keeps rows found elsewhere, a predicate on the outer side of a LEFT join written in WHERE turns it into an inner join, and a TimeManager / FilterManager read restricts the rows to the current business date or segment.

The walk starts at the feature's item in the view's outer SELECT and follows the derived tables and CTEs the expression reads, level by level. It reuses the tokenizer of :mod:tdfs4ds.agent.sql_qualify (no new parser, no dependency). Pure and fail-open: an unreadable view gives None.

_CLAUSES `module-attribute` ¶

_CLAUSES = ('WHERE', 'GROUP', 'HAVING', 'QUALIFY', 'ORDER')

_JOIN_WORDS `module-attribute` ¶

_JOIN_WORDS = ('INNER', 'LEFT', 'RIGHT', 'FULL', 'CROSS', 'OUTER', 'JOIN', 'NATURAL')

_SET_OPS `module-attribute` ¶

_SET_OPS = ('UNION', 'INTERSECT', 'EXCEPT', 'MINUS')

_AGG_RE `module-attribute` ¶

_AGG_RE = compile('\\b(SUM|AVG|COUNT|MIN|MAX|STDDEV_SAMP|STDDEV_POP|VAR_SAMP|VAR_POP|MEDIAN|PERCENTILE_CONT|PERCENTILE_DISC|CORR|COVAR_SAMP|REGR_\\w+)\\s*\\(', I)

_Level ¶

init ¶

__init__(select_idx, depth, end)

_bare ¶

_bare(tok) -> str

Return the identifier text with surrounding double quotes stripped.

_collect_ctes ¶

_collect_ctes(toks: list) -> dict

Map CTE name → index of the ( opening its body.

Handles WITH a AS (...), b AS (...). A CTE with an explicit column list (WITH a (c1, c2) AS (...)) is recorded with its declared columns.

_find_matching_close ¶

_find_matching_close(toks: list, open_idx: int) -> int

Index of the ) matching the ( at open_idx (or len(toks)).

_scope_extent ¶

_scope_extent(toks: list, select_idx: int) -> int

Index one past the last token belonging to this SELECT's scope.

_tokenize ¶

_tokenize(sql: str) -> list

Split SQL into tokens, tracking paren depth.

Comments and whitespace are skipped. String literals and quoted identifiers are emitted as single opaque tokens so their contents can never be mistaken for column references.

_text ¶

_text(sql, toks, a, b) -> str

Original text of tokens [a, b).

_at ¶

_at(toks, i, d, *words) -> bool

_scopes ¶

_scopes(toks)

_paren_before ¶

_paren_before(toks, sc) -> Optional[int]

Index of the '(' directly enclosing a scope (derived table / CTE body).

_clause_bounds ¶

_clause_bounds(toks, sc) -> Dict[str, tuple]

{clause: (start_token, end_token)} for FROM, WHERE, GROUP, HAVING, QUALIFY, ORDER at the scope's depth (the keyword itself excluded).

_split_top ¶

_split_top(toks, a, b, d, sep) -> List[tuple]

Split tokens [a, b) on sep (',' or 'AND') at depth d.

_between_and ¶

_between_and(toks, a, i, d) -> bool

True when the AND at i belongs to a BETWEEN ... AND ... at depth d.

_items ¶

_items(toks, sc, bounds)

SELECT list items: [(name_upper, start, end)].

_relations ¶

_relations(sql, toks, sc, bounds, ctes)

FROM / JOIN entries: [{alias, name, kind, join, on, paren}].

_refs ¶

_refs(toks, a, b)

(qualified, bare) column references in tokens [a, b): qualified as (ALIAS, COL), bare as COL.

_window ¶

_window(sql, toks, a, b) -> Optional[dict]

_manager_kind ¶

_manager_kind(name: str, managers) -> Optional[str]

computation_context ¶

computation_context(ddl: str, feature_name: str, managers: Optional[Dict[str, str]] = None) -> Optional[Dict[str, Any]]

The grain and the filters a feature's expression is computed under.

managers: {'DB.VIEW' or 'VIEW': 'time' | 'filter'} (e.g. from :func:tdfs4ds.lineage.time_drivers.manager_index). Returns None when the DDL cannot be read or the feature is not in the outer SELECT list; else::

{"feature", "expression", "levels": [{"level", "relation", "span", "group_by",
 "window", "where", "having", "qualify", "distinct", "aggregated", "joins",
 "semi_joins", "outer_join_as_inner", "managers"}],
 "grain", "filter_columns", "join_filters", "has_filters"}

_context ¶

_context(sql, feature_name, managers)

column_origin ¶

column_origin(ddl: str, column: str) -> Dict[str, Any]

Where an output column of a view comes from, unchanged (0.3.3.25).

{"relation": <name as written>, "column": <COL>} when the column is a plain reference (optionally aliased) all the way down through derived tables and CTEs to a named relation; else {"reason": ...}: "transformed" (an expression or a CAST changes the values, so a hash on it is another hash), "ambiguous" (a bare name with several relations), "not found", "unreadable". Pure, same tokenizer as :func:computation_context.

_simple_ref ¶

_simple_ref(seg, d)

(qualifier or None, column) when seg (an item's tokens, alias removed) is a bare or qualified column reference, else None.

_origin ¶

_origin(sql, column)

column_lineage ¶

column_lineage(graph, texts: Dict[str, str], view_node: str, column: str) -> Dict[str, Any]

Follow :func:column_origin across views until a table: {"node", "column"} or {"reason"}. texts: {node id: view DDL}; a relation is matched to the current node's successors in the lineage graph by name, so an unqualified name resolves to the object the graph recorded. Pure.

render_computation_context ¶

render_computation_context(ctx: Optional[dict], french: bool = False) -> str

The fixed "Computed at" block of an EXPLAIN answer (markdown).

tdfs4ds.agent.sql_context¶