Deepening opportunities

Architecture review — dbt-meta

20 Jul 2026 · 16.8k LOC · 41 CLI commands · scope: CLI seam, schema-resolution cluster, compiled-SQL fetch, optimisation advisors

module seam leak / duplication deep module

Strong Worth exploring Speculative

Strong in-process Candidate 1

Collapse the two model-resolution stacks into one resolver

command_impl/base.py · fallback.py · command_impl/schema.py · command_impl/columns.py · utils/state_detector.py · utils/model_state.py · command_impl/column_source.py

Before · two parallel stacks

flowchart TD
  cmdA["11 commands
schema/config/context…"] --> B["BaseCommand
get_model_with_fallback"]
  B --> F["FallbackStrategy
prod→dev→BQ"]
  SC["SchemaCommand"] -. copy-pastes .-> B
  cmdB["columns command"] --> SD["ModelStateDetector"]
  SD --> MS["detect_model_state
14-state tree"]
  SD --> CF["ColumnSourceFactory"]
  CF --> BQ["BigQuery / Catalog
source"]
  B -. dev-schema override .-> INV1(("schema=
dev")):::leak
  SD -. dev-schema override .-> INV2(("schema=
dev")):::leak
  SC -. dev-schema override .-> INV3(("schema=
dev")):::leak
  classDef leak fill:#fee2e2,stroke:#dc2626,stroke-width:2px;
  class SC leak

Two disconnected answers to "where does the data come from". The dev-schema invariant is re-implemented at 5+ sites.

After · one deep resolver

all commands

ModelResolver.resolve()

returns ResolvedModel{ location, level, state }

dev-schema override · fallback ordering · state detection — all inside, once

prod/dev manifest

catalog

BigQuery

Problem

The invariant "found in dev → use dev schema, never re-search prod" is honoured by schema.py only — generic prod-mode dev-fallback silently keeps the wrong schema.

Solution

FallbackStrategy returns its FallbackResult (with level + resolved location) to every caller; the dev-schema override lives once behind that interface.

Wins

Locality: invariant lives in one module
Deletes schema.py's copy of _get_model_prod_mode
Correctness: dev schema enforced for all commands
Collapses shallow ColumnSourceFactory (one predicate)
columns.py stops bypassing its own base class
Leverage: one resolver, N call sites

Evidence — dev-schema override duplicated verbatim: base.py:179-180 ≡ state_detector.py:74-75. schema.py:75-118 copy-pastes base.py:_get_model_prod_mode only to capture result.level. columns.py:58-60 — dead process_model stub (bypasses base fallback).

Strong in-process Candidate 2

Revive the render seam — the CLI already has one, unused

cli.py (3516 lines · 41 commands)

Before · 5-step ritual, copy-pasted ×41

try/except

×28

get_manifest_path

×20

Config.from_env

×17

if json_output

×45

handle_error

×28

handle_command_output() · cli.py:716

the render seam — 0 call sites (dead code)

After · one decorator + the revived seam

flowchart TD
  C["41 commands
(body = build result)"] --> D{{"@meta_command
decorator"}}
  D --> M["resolve manifest
+ Config"]
  D --> E["catch DbtMetaError
+ not-found"]
  D --> R["handle_command_output
json | formatter"]
  R --> P["15 _print_* formatters
(kept as args)"]
  classDef deep fill:#0f172a,stroke:#1e293b,color:#e2e8f0;
  class D deep

Each command shrinks to option decls + "build the result"; the ritual moves behind one interface.

Problem

The command layer is shallow — every command re-implements manifest resolution, error catching and JSON-vs-Rich branching. A seam built for exactly this (handle_command_output) sits dead.

Solution

A @meta_command decorator resolves manifest + config, catches errors, and routes output through the revived handle_command_output.

Wins

Deletes 45 inline if json_output branches
Deletes 28 hand-rolled try/except blocks
Locality: error handling in one place
Merge parents/children — one command, direction param
Annotated alias for the 3-option triad
Non-DbtMetaError stops leaking as tracebacks

Evidence — handle_command_output at cli.py:716-743, zero call sites. parents (1188-1233) and children (1252-1279) differ only in labels/variable names (~30 dup lines).

Strong in-process Candidate 3

One CompiledSqlResolver — three copies have silently diverged

utils/compiled_sql.py · usage/advisor_refresh.py · cli.py (_ensure_manifest_compiled) · lineage/builder.py

Before · same concept, 2½ implementations

flowchart TD
  S1["compiled_sql.py
_infer_project_root (manifest walk)
_run_dbt_compile (shutil.which)"]:::a
  S2["advisor_refresh.py
_infer_project_root (manifest+cwd)
_find_dbt_executable (venv-aware)"]:::b
  S3["cli._ensure_manifest_compiled
reuses S2 helpers,
re-inlines subprocess.run"]:::c
  S1 -. diverged .-> S2
  S3 -. half-reuse .-> S2
  classDef a fill:#fee2e2,stroke:#dc2626;
  classDef b fill:#fee2e2,stroke:#dc2626;
  classDef c fill:#fef3c7,stroke:#d97706;

Project-root inference and dbt-executable discovery differ between copies — same bug fixed in one, not the other.

After · one deep resolver, three modes

CompiledSqlResolver

venv-aware discovery · project-root walk · once

.single()
scan, validate

.bulk()
refresh advisor

.whole()
cli optimize

Problem

"Get compiled SQL, auto-compile if missing" exists as two full copies plus a partial third, with divergent project-root and dbt-exec logic.

Solution

One resolver with single / bulk / whole-project modes and the venv-aware discovery behind a single interface.

Wins

Locality: project-root + dbt discovery in one place
Kills the silent divergence between copies
Folds in cli._ensure_manifest_compiled
Leverage: scan/validate/refresh/optimize share it

Evidence — _read_disk_compiled (advisor_refresh.py:555) ≈ _read_compiled_file (compiled_sql.py:124), same body. Two _infer_project_root (compiled_sql.py:93 vs advisor_refresh.py:515) differ in strategy. cli.py:2902-2905 imports refresh's helpers but re-inlines subprocess.run.

Worth exploring local-substitutable Candidate 4

An Advisor seam for cluster + partition (two adapters justify it)

usage/advisor_cluster.py · usage/advisor_partition.py · (usage/advisor_refresh.py stays separate)

Before · pipeline duplicated per advisor

flowchart LR
  subgraph CL["ClusterAdvisor.recommend"]
    a1["resolve → direct_downstream
→ collect_events → score
→ rank → diagnose"]
  end
  subgraph PA["PartitionAdvisor.recommend"]
    a2["resolve → direct_downstream
→ collect_events → score
→ rank → diagnose"]
  end
  a1 -. byte-identical _base_type
+ diagnose block .-> a2

Same seven-step body twice; only the scoring weight table genuinely differs.

After · one template, a scorer strategy

flowchart TD
  T["Advisor.recommend()
shared pipeline template"]:::deep --> SC{{"ColumnScorer
(seam)"}}
  SC --> W1["ClusterScorer
weights"]
  SC --> W2["PartitionScorer
weights + type bonus"]
  classDef deep fill:#0f172a,stroke:#1e293b,color:#e2e8f0;

The pipeline is the deep module; the scorer is the only thing that varies across the seam.

Problem

Cluster and partition advisors share constructor shape, method name and result contract, yet duplicate the whole recommend pipeline and a byte-identical _base_type.

Solution

One shared Advisor template with a ColumnScorer strategy at the seam. Refresh has a different interface (plan(changes)) — leave it out.

Wins

Two adapters justify one real seam
Deletes duplicated _base_type + diagnose block
Locality: pipeline changes hit one template
Refresh stays separate — not forced into the shape

Evidence — _base_type byte-identical at advisor_cluster.py:46-59 and advisor_partition.py:78-85. recommend() bodies mirror each other (cluster:126-183, partition:167-201). _partition_cols (advisor_refresh.py:423) duplicates _common.model_partition_columns.

Worth exploring in-process Candidate 5

Point analyze at the deep extractor; delete the regex twin

command_impl/analyze.py · utils/monitoring.py (fetch_downstream_filter_patterns) · usage/extractor.py

Before · a shallow regex duplicate

analyze.py — "which columns do children filter on?"

fetch_downstream_filter_patterns re.search WHERE · caps at 10 children, top-5

Solves — badly — the exact problem the deep extractor already solves properly.

After · reuse the deep module

analyze.py

ColumnUsageExtractor

scope-aware · alias-resolved · operator-classified

One filter-usage engine feeds advisors and analyze.

Problem

analyze re-derives downstream filter columns with a regex that misses aliases, functions and scope — a lower-fidelity twin of ColumnUsageExtractor.

Solution

Feed analyze from collect_events / the extractor; delete fetch_downstream_filter_patterns.

Wins

Leverage: one filter-usage engine, both callers
Higher fidelity (alias + operator aware)
Deletes a shallow regex module
Removes the 10-child / top-5 caps

Evidence — analyze.py:118 _analyze_downstream_filters → monitoring.py:507-531 re.search WHERE parser. The extractor (extractor.py:134) is the strongest, deepest module in the set.

Top recommendation

Start with Candidate 1 — unify model resolution

It is the only candidate that fixes a correctness defect, not just duplication: the dev-schema invariant is silently unenforced whenever a generic command finds a model via prod-mode dev fallback. Two parallel resolution stacks mean the concept "where does a model's data come from" has no single interface to test. Give FallbackStrategy a result that carries the level and resolved location, put the dev-schema override behind it once, and the copy-pasted schema.py override, the shallow ColumnSourceFactory, and columns.py's bypass of its own base class all fall away.