# dbt-testpilot — Build Progress Report

**Project:** dbt-testpilot — an LLM-assisted dbt test generator
**Owner:** Kushal Mishra
**Report date:** 2026-07-07
**Covers:** naming decision → environment research → Part 0 → Week 1 (profiler) → Week 2 (proposals) → Week 3 (write + run) → Week 3.5 (custom-test macro generation)
**Status:** Published to PyPI — **v0.2.0** (hardened) · Part 0 → Week 4 complete · roadmap done

---

## 1. What dbt-testpilot is

A `pip`-installable CLI that scans a dbt project, profiles the *actual* data (via DuckDB), and auto-proposes the dbt tests you're missing — `not_null`, `unique`, `accepted_values`, `relationships`, and custom — with a human approve/reject step. It's the second piece of a small **dbt reliability toolkit**, pairing with an existing SQL→dbt converter.

The core skills it exercises: parsing dbt's machine-readable artifacts (`manifest.json` / `catalog.json`) and producing strict, trustworthy structured output.

---

## 2. Naming decision

The roadmap's working title was **dbt-testgen**. Research surfaced a problem and a fix:

- A tool literally named **`dbt-testgen`** already exists ([kgmcquate/dbt-testgen](https://github.com/kgmcquate/dbt-testgen)) and does a similar job — but it's a *dbt package* (a Jinja macro, heuristics-only, no LLM) and is **not on PyPI**.
- Candidate PyPI names were checked; all were free. The name **`dbt-testpilot`** was locked.

| Candidate | On PyPI | Chosen |
|---|---|---|
| dbt-testgen | free on PyPI, but name/concept already used by an existing dbt package | no |
| **dbt-testpilot** | **free** | **yes** |
| dbt-testsmith / dbt-testforge / dbt-testwise / dbt-testscout / dbt-testcraft | free | alternates |

**Why `dbt-testpilot`:** it signals the LLM + human-in-the-loop angle ("you're the pilot, it copilots your test suite"), avoids the collision with the existing heuristic-only `dbt-testgen`, and dodges the trademark baggage of "copilot."

All references were renamed across the roadmap, README, and repo. (The desktop folder was later renamed to `dbt-testpilot` too.)

---

## 3. Environment research (verified July 2026)

The roadmap flagged that several facts drift over time. They were re-verified against current sources:

| Area | Finding (July 2026) | Impact |
|---|---|---|
| dbt distributions | Now **three**: Fusion engine (Rust, proprietary), Core v2 (Rust, OSS), Core v1 (Python, OSS, still maintained ~1.11.x/1.12) | Install `dbt-core dbt-duckdb` in a venv; **never** `pip install dbt` (that pulls the platform/Fusion CLI and shadows Core) |
| Python | dbt Core supports 3.10–3.13; **3.12** is the safe middle | Roadmap choice confirmed |
| Artifacts | `manifest.json` is **v12**, `catalog.json` is **v1**; version can bump per dbt minor | Parser reads each artifact's own `dbt_schema_version` rather than assuming a shape |
| Gemini free tier | ~1,500 req/day on Flash (volatile through 2026) | Fine for dev |
| Groq free tier | Llama-3.1-8B ~14,400 req/day; **tokens-per-minute** usually binds first | Backup lane |
| Sandbox | `jaffle_shop_duckdb` + `jafgen` still work as described | Used as-is |

**Architecture recommendation adopted:** heuristics-first, LLM-augmented (not LLM-only) — cheaper, works offline, and more trustworthy. The LLM's job is rationale, cross-table relationships, and custom tests.

---

## 4. Part 0 — setup (complete)

| Item | Status |
|---|---|
| Accounts: GitHub (2FA), Google AI Studio (Gemini), Groq | done (user) |
| Homebrew, git, uv, DuckDB CLI, VS Code | done (user) |
| Python 3.12 venv (`uv`), project scaffolding (`pyproject.toml`, `uv.lock`) | done (user) |
| `dbt-core` + `dbt-duckdb` installed (inside `jaffle_shop_duckdb/venv`) | done (user) |
| `jaffle_shop_duckdb` cloned, `dbt build` succeeds | done (user) |
| `.env` created from `.env.example` with LLM keys | done (user) |
| `git init` + commits | done (user) |
| Repo hygiene: `.gitignore`, `.env.example`, README stub | added |
| `check_llm.py` — one-shot Gemini/Groq key verifier | added |

Build artifacts confirmed present: `jaffle_shop_duckdb/target/manifest.json`, `.../catalog.json`, and `jaffle_shop_duckdb/jaffle_shop.duckdb`.

---

## 5. Week 1 — data profiler (complete)

**Goal:** get structured facts about each model, ready for the LLM.
**Done-when:** for any model, print its columns, types, null %, and cardinality. ✅

### Architecture — the `dbt_testpilot/` package

| Module | Responsibility |
|---|---|
| `artifacts.py` | Parse `manifest.json` (model list + already-existing tests) and `catalog.json` (declared column types) into `Model` objects |
| `profiler.py` | For each model, read the **actual** columns/types from DuckDB's `information_schema`, then run **one aggregate pass** per table for row count, null %, cardinality, min/max; capture the value list for low-cardinality columns |
| `report.py` | Render each `ModelProfile` as a JSON-ready dict and a readable console view |
| `cli.py` / `__main__.py` | `python -m dbt_testpilot profile --project-dir <dir> --out profiles.json` |

### Key design decisions

- **DuckDB is the source of truth for columns.** The manifest tells us *which relations are models*; the actual columns and types are read from the warehouse, so the profiler never trips over stale or undocumented columns.
- **Existing tests are captured.** Each column records which tests it already has (`unique`, `not_null`, `accepted_values`, `relationships`), so Week 2 proposes only what's *missing*.
- **One aggregate query per table** keeps profiling efficient even on larger projects.
- **Heuristic signals are pre-computed** (`is_unique`, `null_pct`, low-cardinality value lists) — the exact inputs the Week 2 proposer needs.

### How to run

```
cd ~/Desktop/Claude/dbt-testpilot
source .venv/bin/activate
pip install duckdb
python -m dbt_testpilot profile --project-dir jaffle_shop_duckdb --out profiles.json
```

---

## 6. Verification & results

Run against jaffle_shop (5 models). The profiler printed columns, types, null %, and cardinality for every model and wrote a valid `profiles.json`.

| Model | Rows | Highlights |
|---|---|---|
| `main.customers` | 100 | `customer_id` unique + not-null (PK); order-derived columns ~38% null (customers who never ordered) |
| `main.orders` | 99 | `order_id` unique; `status` ∈ {completed, placed, shipped, returned, return_pending}; `customer_id` already `relationships`-tested |
| `main.stg_customers` | 100 | `customer_id` unique + not-null |
| `main.stg_payments` | 113 | `payment_id` unique + not-null; `payment_method` has 4 values |
| `main.stg_orders` | 99 | `order_id` unique + not-null; `status` has 5 values |

**Existing coverage detected:** 20 test nodes in the manifest (e.g. `customer_id → unique, not_null`; `status → accepted_values`; `orders.customer_id → relationships`). These will be excluded from future proposals.

**One fix during verification:** low-cardinality value lists initially included `NULL` (e.g. `number_of_orders` showed `None(38)`); the query now filters nulls, so value lists are clean and ready for `accepted_values` proposals.

Each column in `profiles.json` carries: `name`, `data_type`, `null_count`, `null_pct`, `distinct_count`, `distinct_pct`, `is_unique`, `min`, `max`, `top_values`, `existing_tests`.

---

## 7. Week 2 — test proposals (complete)

**Goal:** turn the profiles into *proposed* dbt tests as strict structured output.
**Done-when:** the tool suggests sensible tests on jaffle_shop that a human would mostly agree with. ✅

### Two layers

**Heuristics (default, deterministic, no API key)** — rules over `profiles.json`:

| Test | Proposed when |
|---|---|
| `not_null` | column has zero nulls |
| `unique` | distinct count = non-null count (a real key) |
| `accepted_values` | categorical column (text/boolean), ≤ 15 distinct values |
| `relationships` | `*_id` column that isn't this table's key but is unique in another model |

**LLM (opt-in, `--llm`)** — each model's profile is sent to Gemini/Groq one model at a time, asking for the missing tests as strict JSON. Every item is validated (the column must exist; `accepted_values` needs `values`; `relationships` needs `to`/`field`); malformed or hallucinated items are dropped; a failed call degrades to heuristics-only. The LLM may also propose custom tests.

Proposals from both layers are merged, de-duplicated by `(model, column, test)`, and anything already in a column's `existing_tests` is removed.

### Modules added

| Module | Responsibility |
|---|---|
| `proposals.py` | `Proposal` schema, heuristic rules, LLM-output validation, merge/dedupe |
| `llm.py` | Per-model prompt, Gemini/Groq JSON-mode call, parse + validate, graceful fallback |
| `cli.py` | new `propose` subcommand |

### Verification

Heuristics on jaffle_shop produced **14 proposals**, correctly skipping every column that already had a `unique` / `accepted_values` / `relationships` test:

| Test | Count | Examples |
|---|---|---|
| `not_null` | 12 | `orders.order_date`, `orders.status`, `customers.first_name`, `stg_payments.amount` |
| `relationships` | 2 | `stg_payments.order_id → orders`, `stg_orders.customer_id → customers` |

The LLM layer's parsing/validation was checked with a mocked response: it stripped code fences, dropped a hallucinated column and an `accepted_values` item missing its `values`, and kept the valid proposals. The **live** LLM call uses the user's API key and runs on the user's machine (not in the sandbox).

### Run it

```
python -m dbt_testpilot propose --profiles profiles.json --out proposals.json         # heuristics only
python -m dbt_testpilot propose --profiles profiles.json --out proposals.json --llm    # + Gemini/Groq
```

---

## 8. Week 3 — write tests + run (complete)

**Goal:** close the loop from suggestion to running test.
**Done-when:** go end to end — scan → review → approve → tests running. ✅

### Flow

`apply` reads `proposals.json`, presents each proposal for **approve/reject** (interactive by default — nothing is written without a human OK), writes the approved tests into the models' `schema.yml`, and can run `dbt test`.

### Writing into schema.yml

- Each model's YAML file is located via the manifest's `patch_path`.
- Tests are merged with **round-trip YAML** (ruamel), preserving existing descriptions, comments, ordering, and formatting.
- The file's existing key is matched (`tests:` vs `data_tests:`), and dbt's `arguments:` wrapper is used for `accepted_values` / `relationships`.
- Missing columns/models are created; duplicate tests are skipped.
- **Only built-in tests** (`not_null`, `unique`, `accepted_values`, `relationships`) are written by default; custom LLM tests are skipped unless `--include-custom` — which now generates vetted macros for supported patterns (see §9).

### Modules added

| Module | Responsibility |
|---|---|
| `yaml_writer.py` | Locate + merge tests into schema.yml (round-trip, no clobbering) |
| `apply.py` | Approve/reject flow, built-in/custom filtering, optional `dbt test` |
| `cli.py` | new `apply` subcommand (`--yes` / `--dry-run` / `--min-confidence` / `--include-custom` / `--run`) |

### Verification (end-to-end)

On a non-destructive copy of jaffle_shop, approving the built-in proposals wrote **16 tests** across the two `schema.yml` files (descriptions preserved), then `dbt test` ran:

```
Done. PASS=36 WARN=0 ERROR=0 SKIP=0 TOTAL=36
```

All 36 (20 pre-existing + 16 generated) passed — generated `not_null`, `unique`, `accepted_values`, and `relationships` tests all execute. The 13 custom LLM tests were correctly skipped. New dependency: `ruamel.yaml`.

### Run it

```
python -m dbt_testpilot apply --proposals proposals.json --project-dir jaffle_shop_duckdb             # interactive
python -m dbt_testpilot apply --proposals proposals.json --project-dir jaffle_shop_duckdb --yes --run  # accept all + dbt test
```

---

## 9. Week 3.5 — custom-test macro generation (complete)

**Why:** the LLM proposes semantically useful checks (e.g. `is_not_negative`, `positive_amount`, `date_in_past_or_present`) by *name*, but dbt runs a generic test `foo` via a macro `test_foo`. Only four built-ins ship with dbt, so those names have no macro and fail to compile — proven: force-writing all 13 turned `dbt test` into `PASS=36 ERROR=13`.

**Fix — Option 1 (generate macros), chosen over adding dbt_utils/dbt_expectations:** self-contained (no target-project package setup), and the reliability-safe choice.

### How `--include-custom` now works

- For each approved custom test we either **reuse** an existing project macro, **generate** a vetted one, or **skip** it.
- Vetted templates cover the common patterns — **value ≥ 0** (`non_negative`: `is_not_negative`, `positive_amount`, `positive_number`, `clv_non_negative`) and **date not in the future** (`not_future`: `date_in_past_or_present`). Macros are written to `macros/dbt_testpilot/test_<name>.sql`.
- Genuinely **bespoke** logic (multi-column or regex — `first_order_before_most_recent_order`, `order_details_consistency`, `is_total_amount_correct`, `is_initial_format`) is **not** auto-written; guessing that SQL would risk false confidence, so it's left for the user to implement.

### Module added

| Module | Responsibility |
|---|---|
| `macrogen.py` | Classify a custom test → vetted template; reuse existing macros; generate `test_<name>.sql`; report what's skipped |

### Verification

- On a **pristine** jaffle_shop copy, `apply --yes --include-custom` generated **5 macros**, wrote **25 tests** (16 built-in + 9 supported custom), skipped 4 bespoke, and `dbt test` → **PASS=45 WARN=0 ERROR=0**.
- The **real** project had 13 custom tests force-written earlier by the *old* `--include-custom` (no macros → `ERROR=13`). It was reverted to pristine and re-applied with the new code → **PASS=45/45**, green.

### Run it

```
python -m dbt_testpilot apply --proposals proposals.json --project-dir jaffle_shop_duckdb --yes --include-custom
```

---

## 10. Repository layout

```
dbt-testpilot/
├─ dbt_testpilot/            # the tool (Python package)
│  ├─ __init__.py
│  ├─ __main__.py            # python -m dbt_testpilot
│  ├─ artifacts.py           # parse manifest.json + catalog.json
│  ├─ profiler.py            # DuckDB profiling (Week 1)
│  ├─ proposals.py           # heuristic proposals + schema/validation (Week 2)
│  ├─ llm.py                 # LLM proposals, strict JSON (Week 2)
│  ├─ yaml_writer.py         # merge approved tests into schema.yml (Week 3)
│  ├─ apply.py               # approve/reject flow + run dbt test (Week 3)
│  ├─ macrogen.py            # generate macros for supported custom tests (Week 3.5)
│  ├─ verify.py              # verify tests against data before writing (v0.2.0)
│  ├─ report.py              # JSON + console rendering
│  └─ cli.py                 # profile + propose + apply commands
├─ jaffle_shop_duckdb/       # sample dbt project (sandbox) — gitignored
├─ check_llm.py              # Part 0: one-shot LLM key verifier
├─ dbt-testpilot_Roadmap.md  # the 4-week plan
├─ docs/progress-report.md   # this report
├─ README.md
├─ pyproject.toml, uv.lock   # uv project scaffolding
├─ .env / .env.example / .gitignore
├─ profiles.json             # generated profiler output — gitignored
└─ proposals.json            # generated proposals — gitignored
```

---

## 11. Current status & next steps

- **Part 0 / Week 1 / Week 2 / Week 3 (+ custom-test macros):** all complete and verified.
- **Latest result:** the real jaffle_shop project runs `dbt test` → **45/45** (20 original + 16 built-in + 9 macro-backed custom), with 5 generated macros in `macros/dbt_testpilot/`.
- **Commit:** the package (incl. `yaml_writer.py`, `apply.py`, `macrogen.py`), the `apply` command, and doc updates are staged; recommended message: `Week 3 + custom-test macro generation`.
- **New dependency:** `ruamel.yaml` (round-trip YAML) — `pip install ruamel.yaml`.

**Week 4 — packaged & published (complete).**
- `pyproject.toml` finalized (hatchling, console entry point, core deps duckdb + ruamel.yaml, optional `[llm]` extra, MIT license, dynamic version) + `LICENSE`.
- Built, `twine check` passed, **published to TestPyPI then PyPI** — `pip install dbt-testpilot` verified in a clean venv (v0.1.0; v0.1.1 refreshes README/metadata). License **MIT** (copyright retained).
- Validated on jaffle_shop, a synthetic bookshop, and the real Olist warehouse (caught genuine orphan-FK issues).
- **Post-launch:** make the repo public, record a demo GIF, add to LinkedIn. `docs/`, `check_llm.py`, the roadmap, and `test-llms.py` are gitignored (local only).

---

## 12. v0.2.0 — hardening & accuracy (post-launch)

An end-user running the guide on **Windows** surfaced two issues, fixed here along with a broader accuracy push.

**The crash:** files were written in the OS-default encoding. On Windows that's cp1252, so `são paulo` (Olist data) was written as byte `0xe3` — invalid UTF-8. dbt reads files as UTF-8 and crashed during parse, so `dbt test` never ran. **Fix:** all file I/O now forces `encoding="utf-8"` (+ ruamel `allow_unicode`, JSON `ensure_ascii=False`).

**Verify-before-write (`verify.py`):** `apply` runs each candidate test's violation query against the DuckDB and writes **only** tests with zero violations; tests the data would fail are reported (`would FAIL: N rows`) as findings, not written. New flags: `apply --db`, `--no-verify`.

**LLM:** the Gemini free tier is ~20 requests/day and the tool called it once per model — a 25-model project hit `429 RESOURCE_EXHAUSTED`. Now all models go in **one batched request** with 429 retry/backoff, behind a stricter, data-grounded prompt.

**Macros:** `positive` (>0) is now distinct from `non_negative` (≥0); added `not_future`, `not_empty_string`, `is_integer`. Each macro and its verification share one predicate.

**Verified (Olist, real 100k-row data):** verify correctly dropped `unique` on a duplicated column, `positive` on a column with zeros (keeping `non_negative`), and an orphan-FK `relationships` test; `dbt test` came back green and the generated macros ran.

---

## 13. Operational notes

- **Folder renames break the sandbox mount.** After renaming the project folder, the Linux sandbox pointed at a stale path; reconnecting the folder fixed it. The tool itself runs on the user's machine, not in the sandbox.
- **DuckDB catalog stats are minimal** (`has_stats` only), so all row/null/cardinality numbers are computed by the profiler's own SQL — not read from `catalog.json`.
- **Free-tier limits move.** Provider is read from `.env` so switching Gemini ⇄ Groq ⇄ Ollama is a one-line change.
