# dbt-testpilot — Context Freeze

**Frozen:** 2026-07-07 (updated 2026-07-09 — v0.2.0: verify-before-write + UTF-8/Windows fix) · **Owner:** Kushal Mishra · **Location:** `~/Desktop/Claude/dbt-testpilot/`

A complete point-in-time snapshot of the project and the build session, so any future session (or reader) has full context without re-deriving it.

---

## TL;DR

- **What:** `dbt-testpilot` — a pip-installable CLI that scans a dbt project, profiles the real data (DuckDB), proposes the dbt tests you're missing (heuristics + optional LLM), writes the ones you approve into `schema.yml`, and runs `dbt test`.
- **Status:** Shipped and **hardened** — full pipeline published to PyPI as **v0.2.0** (`pip install dbt-testpilot`): adds verify-before-write, a UTF-8/Windows crash fix, batched LLM calls, and more accurate macros. MIT licensed, copyright retained.
- **Proof:** on jaffle_shop the full pipeline runs; with `--include-custom` the real project's `dbt test` passes **45/45** (20 original + 16 built-in + 9 macro-backed custom).
- **Owner context:** Kushal — data engineer, M1 Mac (16 GB), ₹0 budget (free/open-source only), building a public reliability/Lakehouse portfolio. Prefers concise, direct answers.

---

## The project

A tool for a chronically under-tested problem: analysts rarely test their data. dbt-testpilot reads a dbt project's metadata, profiles the actual tables, and suggests the missing tests — with a human approve step. It's the second half of a small **dbt reliability toolkit**, pairing with Kushal's existing SQL→dbt converter. Core skills exercised: parsing dbt artifacts (`manifest.json`/`catalog.json`) and producing strict, trustworthy structured output.

---

## Key decisions (and why)

- **Name: `dbt-testpilot`** (locked, verified free on PyPI). The roadmap's working title `dbt-testgen` collides with an existing heuristic-only dbt package ([kgmcquate/dbt-testgen](https://github.com/kgmcquate/dbt-testgen)); "testpilot" signals the LLM + human-in-the-loop angle. Free alternates checked: dbt-testsmith / forge / wise / scout / craft.
- **Heuristics-first, LLM-augmented** (not LLM-only): a deterministic base is cheaper, works offline, and is more trustworthy; the LLM adds rationale, cross-table relationships, and custom tests.
- **Python dbt Core v1 + `dbt-duckdb`** — never `pip install dbt` (that installs the Rust Fusion/platform CLI and shadows Core). Stable, parseable artifacts.
- **DuckDB is the source of truth for columns** — actual columns/types come from `information_schema`; the manifest only tells us which relations are models and what's already tested.
- **Never auto-write without a human OK** — `apply` is interactive by default. Only dbt built-in tests are written by default; custom LLM tests are skipped (they need a macro) unless `--include-custom`.
- **Round-trip YAML (ruamel)** — writing into `schema.yml` preserves existing descriptions, comments, ordering, and formatting.
- **Custom tests → generate vetted macros, don't guess** — `--include-custom` generates macros for recognised patterns (value ≥ 0, not-future) so the LLM's custom tests actually run; bespoke logic is skipped rather than auto-writing SQL we can't guarantee. Chosen over depending on dbt_utils/dbt_expectations (keeps the tool self-contained).

---

## Environment facts (verified July 2026)

| Area | Finding |
|---|---|
| dbt distributions | Three: Fusion engine (Rust, proprietary), Core v2 (Rust, OSS), Core v1 (Python, OSS, maintained ~1.11.x/1.12) — use Core v1 |
| Python | dbt Core supports 3.10–3.13; **3.12** chosen |
| Artifacts | `manifest.json` v12, `catalog.json` v1; schema can bump per dbt minor |
| Gemini free tier | ~1,500 req/day on Flash (volatile) |
| Groq free tier | Llama-3.1-8B ~14,400 req/day (tokens-per-minute usually binds first) |
| Sandbox | `jaffle_shop_duckdb` + `jafgen` work as described |

Working versions: dbt-core 1.11.12, dbt-duckdb 1.10.1, duckdb 1.5.4, google-genai 1.7.0, groq 1.5.0.

---

## What's built — the tool

Package `dbt_testpilot/`, three commands:

```
python -m dbt_testpilot profile --project-dir jaffle_shop_duckdb --out profiles.json
python -m dbt_testpilot propose --profiles profiles.json --out proposals.json [--llm]
python -m dbt_testpilot apply   --proposals proposals.json --project-dir jaffle_shop_duckdb [--yes] [--dry-run] [--include-custom] [--run]
```

| Module | Responsibility |
|---|---|
| `artifacts.py` | Parse manifest (models + existing tests) + catalog (declared types) |
| `profiler.py` | Read real columns from DuckDB `information_schema`; one aggregate pass → row count, null %, cardinality, min/max; low-cardinality value lists |
| `report.py` | JSON-ready dicts + console rendering (profiles and proposals) |
| `proposals.py` | `Proposal` schema; heuristic rules (not_null/unique/accepted_values/relationships); LLM-output validation; merge/dedupe; skip existing |
| `llm.py` | Per-model prompt → Gemini/Groq JSON mode → parse + validate → graceful fallback |
| `yaml_writer.py` | Locate model YAML via manifest `patch_path`; round-trip merge into the right column; match `tests:`/`data_tests:` + `arguments:` wrapper; create missing columns; skip dups |
| `apply.py` | Approve/reject flow; macro-gen; **verify-before-write**; optional `dbt test` |
| `verify.py` | Run each test's violation query on DuckDB; write only 0-violation tests (v0.2.0) |
| `macrogen.py` | Generate vetted generic-test macros for supported custom patterns (value ≥ 0, not-future) |
| `cli.py` / `__main__.py` | `profile` + `propose` + `apply` subcommands |

Supporting: `check_llm.py` (Week 0 one-shot key verifier).

---

## Verification by phase

- **Part 0:** `dbt build` succeeds on jaffle_shop; `check_llm.py` returns `[ok]`.
- **Week 1:** profiler prints columns/types/null %/cardinality for all 5 models; `profiles.json` valid.
- **Week 2:** heuristics → 14 proposals (12 `not_null`, 2 `relationships`), correctly skipping already-tested columns; LLM parse/validate verified with a mock (drops hallucinated columns + malformed args). Live LLM run added custom tests (`positive_amount`, etc.).
- **Week 3:** `apply` wrote **16** built-in tests into a jaffle_shop copy (descriptions preserved); `dbt test` → **PASS=36 WARN=0 ERROR=0** (20 pre-existing + 16 generated); 13 custom LLM tests skipped.
- **Custom-test macros:** `--include-custom` generates vetted macros (value ≥ 0, not-future) for supported customs. On a pristine copy → 5 macros, 25 tests written, `dbt test` **45/45**. The real project (which had 13 macro-less customs force-written by the old code → ERROR=13) was reverted + re-applied → **45/45 green**.

---

## Repository layout

```
dbt-testpilot/
├─ dbt_testpilot/            # the tool
│  ├─ __init__.py  __main__.py
│  ├─ artifacts.py  profiler.py  report.py          # Week 1
│  ├─ proposals.py  llm.py                           # Week 2
│  ├─ yaml_writer.py  apply.py  macrogen.py  verify.py  # Week 3 + v0.2.0 verify
│  └─ cli.py                                          # profile + propose + apply
├─ jaffle_shop_duckdb/       # sandbox (gitignored)
├─ check_llm.py              # Week 0 key check
├─ dbt-testpilot_Roadmap.md  # the 4-week plan
├─ docs/progress-report.md   # detailed build log
├─ docs/context-freeze.md    # this file
├─ README.md                 # onboarding (Weeks 0–3 detailed)
├─ pyproject.toml  uv.lock  requirements.txt
├─ .env / .env.example / .gitignore
└─ profiles.json / proposals.json   # generated (gitignored)
```

---

## Dependencies

- Runtime: `duckdb` (profiling), `ruamel.yaml` (Week 3 YAML edits).
- Optional (for `--llm`): `google-genai`, `groq`.
- Target project needs `dbt-core` + `dbt-duckdb`.
- Note: the env has `PyYAML` but `apply` uses `ruamel.yaml` for comment-preserving round-trips.

---

## Git / commit state (as of freeze)

- Staged (not yet committed): `dbt_testpilot/*`, `docs/*`, `README.md`, `.gitignore`, `pyproject.toml`, `uv.lock`, `requirements.txt`, `check_llm.py`.
- Ignored (never committed): `.env`, `.venv/`, `jaffle_shop_duckdb/`, `profiles.json`, `proposals.json`, `*.whl`.
- The code is interdependent (the CLI imports the proposer/writer/macro-gen), so a single commit is cleanest. Suggested: `Week 3 + custom-test macro generation` (or a combined `Weeks 1–3` message).

---

## Operational notes / gotchas

- **Renaming the project folder mid-session** orphans the Linux sandbox mount (it pointed at the old path and looked empty). Fix: reconnect the folder; host-path file tools keep working regardless.
- **The sandbox is Linux, separate from the Mac** — it can't run the Mac venv's binaries. The tool runs on the user's machine; for verification, dbt/duckdb were installed in the sandbox and run against a *copy* of jaffle_shop (real project untouched).
- **DuckDB catalog stats are minimal** (`has_stats` only) — all stats are computed by the profiler's own SQL.
- **Free-tier limits move** — provider is read from `.env` (`gemini`/`groq`/`ollama`), a one-line switch.
- **Custom LLM tests need a dbt macro** — skipped by default; `--include-custom` generates vetted macros (value ≥ 0, not-future) into `macros/dbt_testpilot/` for supported patterns and skips bespoke ones. (An earlier *old* `--include-custom` force-wrote macro-less customs and turned `dbt test` red — the current build fixes that.)
- **jaffle_shop quirk:** `schema.yml` documents `total_order_amount` but the built `customers` model has `customer_lifetime_value` — the tool profiles the *actual* columns, so this mismatch is handled.

---

## v0.2.0 — hardening & accuracy (2026-07-09)

After launch, an end-user on **Windows** hit a crash and the LLM quota; v0.2.0 fixes both plus a broad accuracy push:

- **UTF-8 everywhere** — files were written in the OS-default encoding; on Windows that's cp1252, so `são paulo` (Olist data) became an invalid byte and dbt's UTF-8 parser crashed → `dbt test` never ran. All file I/O is now UTF-8 (+ ruamel `allow_unicode`, JSON `ensure_ascii=False`).
- **Verify-before-write** (`verify.py`) — `apply` runs each candidate test's violation query against the DuckDB and writes **only** tests with zero violations; would-fail tests are reported as findings. New flags: `apply --db`, `--no-verify`.
- **Batched LLM + retry** — Gemini free tier is ~20 req/day and the tool called it once per model; now all models go in one request with 429 backoff, behind a stricter data-grounded prompt.
- **Accurate macros** — `positive` (>0) vs `non_negative` (≥0) distinguished; added `not_future`, `not_empty_string`, `is_integer`; each macro and the verifier share one predicate.
- **Verified on Olist** (100k rows): verify dropped `unique`-on-dupes, `positive`-on-zeros (kept `non_negative`), and orphan-FK relationships; `dbt test` green; generated macros run.

---

## Session arc (chronological)

1. Read the roadmap; deep research verified/updated its time-sensitive facts and surfaced the `dbt-testgen` name collision.
2. Locked the name `dbt-testpilot` (checked PyPI availability of several candidates).
3. Renamed the name everywhere; scaffolded Part 0 hygiene (`.gitignore`, `.env.example`, README, `check_llm.py`).
4. Kushal completed Part 0 on the Mac (accounts, installs, `uv init`, jaffle_shop `dbt build`, `.env`, git). Folder later renamed to `dbt-testpilot` (mount reconnected).
5. **Week 1** — built the profiler; verified on jaffle_shop → `profiles.json`.
6. Produced the detailed progress report; made the README onboarding-grade.
7. **Week 2** — heuristic + opt-in LLM proposer; verified; documented in detail.
8. Commit guidance (Weeks 1–2 were uncommitted).
9. **Week 3** — YAML writer + approve/reject `apply`; end-to-end `dbt test` → **36/36**; documented in detail.
10. **Custom-test macro generation** — `macrogen.py`; `--include-custom` generates vetted macros so custom tests run; real project reverted + re-applied → **45/45**.
11. This context freeze (updated).

---

## Week 4 — packaged & published (complete)

- `pyproject.toml` finalized (hatchling backend, console entry point `dbt-testpilot`, core deps duckdb + ruamel.yaml, optional `[llm]` extra, MIT license, dynamic version); `LICENSE` added.
- Built + `twine check` passed; **published to TestPyPI then PyPI** — `pip install dbt-testpilot` verified in a clean venv (v0.1.0; v0.1.1 refreshes the README/metadata). License stays **MIT** — Kushal retains copyright.
- Validated on three schemas: jaffle_shop, a synthetic bookshop, and the real **Olist** warehouse (where the generated `relationships` tests caught genuine orphan-FK issues).
- **Done ✅** — a brand-new venv → `pip install dbt-testpilot` → runs against a dbt project.

**Post-launch remaining:** make the GitHub repo public (`github.com/orgkushal/dbt-testpilot`), record a demo GIF, add to LinkedIn. `docs/`, `check_llm.py`, `dbt-testpilot_Roadmap.md`, and `test-llms.py` are gitignored — kept local, not in the public repo.
