tdfs4ds.agent.sql_invariants

tdfs4ds.agent.sql_invariants

Structural invariants for detecting business-logic drift in a SQL rewrite.

Why this exists

The optimizer accepts a rewrite when EXPLAIN compiles and user_score does not regress. Neither condition has anything to do with the query still meaning the same thing — and the score is actively anti-correlated with safety: it is derived from the count of [You] items in the EXPLAIN plan, so a candidate that drops a predicate, loses a DISTINCT or removes a join produces a simpler plan, scores higher, and is accepted preferentially.

Column qualification (:mod:tdfs4ds.agent.sql_qualify) removes one failure mode — silent rebinding of a bare column when scopes merge. It does nothing about the others, all of which compile cleanly:

• a dropped WHERE/ON predicate
• an INNER join silently becoming LEFT (or vanishing) during a flatten
• a changed aggregation grain when two GROUP BY levels collapse
• a lost DISTINCT / QUALIFY
• a narrowed window PARTITION BY
• a dropped source table

This module extracts a coarse structural fingerprint from a query and diffs two fingerprints. It is deliberately conservative in one direction only: it can miss a semantic change (it is a fingerprint, not a prover), but a violation it does report is a real structural difference that a simplification is not entitled to make on its own.

It is a cheap, offline, always-on gate — not a substitute for executing both queries and comparing rows, which remains the only complete oracle.

logger module-attribute

logger = getLogger(__name__)

__all__ module-attribute

__all__ = ['extract_invariants', 'diff_invariants', 'InvariantViolation', 'describe_violations']

_STR_RE module-attribute

_STR_RE = compile("'(?:[^']|'')*'")

_LINE_COMMENT_RE module-attribute

_LINE_COMMENT_RE = compile('--[^\\n]*')

_BLOCK_COMMENT_RE module-attribute

_BLOCK_COMMENT_RE = compile('/\\*.*?\\*/', DOTALL)

_JOIN_RE module-attribute

_JOIN_RE = compile('\\b(?:(LEFT|RIGHT|FULL)\\s+(?:OUTER\\s+)?JOIN|(INNER)\\s+JOIN|(CROSS)\\s+JOIN|(?<!\\w)(JOIN))\\b', IGNORECASE)

_TABLE_REF_RE module-attribute

_TABLE_REF_RE = compile('(?:FROM|JOIN)\\s+\\"?(\\w+)\\"?\\s*\\.\\s*\\"?(\\w+)\\"?', IGNORECASE)

_AGG_RE module-attribute

_AGG_RE = compile('\\b(SUM|COUNT|AVG|MIN|MAX|STDDEV_SAMP|STDDEV_POP|VAR_SAMP|VAR_POP|MEDIAN|PERCENTILE_CONT|PERCENTILE_DISC|REGR_SLOPE|CORR)\\s*\\(', IGNORECASE)

_WINDOW_PARTITION_RE module-attribute

_WINDOW_PARTITION_RE = compile('PARTITION\\s+BY\\s+(.*?)(?:\\bORDER\\s+BY\\b|\\bROWS\\b|\\bRANGE\\b|\\))', IGNORECASE | DOTALL)

_GROUP_BY_RE module-attribute

_GROUP_BY_RE = compile('\\bGROUP\\s+BY\\b(.*?)(?=\\bHAVING\\b|\\bQUALIFY\\b|\\bORDER\\s+BY\\b|\\bUNION\\b|\\bINTERSECT\\b|\\bEXCEPT\\b|\\bMINUS\\b|\\)|\\Z)', IGNORECASE | DOTALL)

_QUALIFIER_RE module-attribute

_QUALIFIER_RE = compile('\\b[A-Za-z_]\\w*\\s*\\.\\s*(?=[A-Za-z_\\"])')

InvariantViolation

One structural difference between an original query and a rewrite.

__slots__ class-attribute instance-attribute

__slots__ = ('kind', 'detail', 'severity')

kind instance-attribute

kind = kind

detail instance-attribute

detail = detail

severity instance-attribute

severity = severity

__init__

__init__(kind: str, detail: str, severity: str = 'blocking')

__repr__

__repr__() -> str

__eq__

__eq__(other) -> bool

_strip_noise

_strip_noise(sql: str) -> str

Remove comments and blank out string literals (keeping length stable-ish).

_norm_ws

_norm_ws(s: str) -> str

_strip_qualifiers

_strip_qualifiers(expr: str) -> str

Drop alias. prefixes from an expression.

Grain and partitioning must be compared alias-insensitively: renaming c to cl is a legitimate rewrite the LLM performs constantly, and comparing the raw text would flag every one of them as a grain change, making the guard useless.

The cost is precision when two relations expose the same column name and the rewrite switches which one it groups by. That is exactly the case the qualifier pass (:mod:tdfs4ds.agent.sql_qualify) makes explicit upstream, so the two layers cover each other.

_split_top_level

_split_top_level(expr: str, sep: str = ',') -> list

Split on sep at paren depth 0.

_count_predicates

_count_predicates(sql: str) -> int

Count boolean conjuncts in WHERE / ON / HAVING / QUALIFY clauses.

A count, not a set: predicate text legitimately changes under rewriting (a.x = b.y ↔ b.y = a.x), but a rewrite that reduces the number of filtering conditions has almost certainly dropped one.

extract_invariants

extract_invariants(sql: str) -> dict

Return a coarse structural fingerprint of sql.

Keys

tables set of DB.OBJ referenced in FROM/JOIN join_kinds multiset (as a sorted list) of join flavours n_predicates count of boolean conjuncts across WHERE/ON/HAVING/QUALIFY has_distinct bool has_qualify bool group_by_keys sorted list of normalised GROUP BY expressions n_group_by number of GROUP BY clauses partition_keys sorted list of normalised window PARTITION BY key lists agg_funcs sorted list of aggregate function names used set_ops sorted list of UNION/INTERSECT/EXCEPT operators used

diff_invariants

diff_invariants(original_sql: str, candidate_sql: str, allow: Optional[set] = None) -> list

Compare two queries structurally.

Parameters:

Name	Type	Description	Default
original_sql	str	The trusted query.	required
candidate_sql	str	The proposed rewrite.	required
allow	Optional[set]	Violation kind strings to downgrade to 'warn' (for strategies that legitimately change that property — e.g. an aggregation-pushdown strategy changing GROUP BY).	None

Returns:

Type	Description
list	A list of :class:InvariantViolation. Empty means the rewrite is
list	structurally consistent with the original.

Direction matters: only losses are blocking. A rewrite that adds a predicate or a GROUP BY key is doing something intentional and is reported as a warning, not a block — the dangerous direction is silently doing less work than the original.

describe_violations

describe_violations(violations) -> str

One-line-per-violation rendering, blocking first.

has_blocking

has_blocking(violations) -> bool