# Getting Started with dbt-testpilot

A from-scratch guide to installing and using **dbt-testpilot** on any DuckDB-backed dbt project — including the LLM setup, every command, and how custom tests work.

> **What it is:** a `pip`-installable CLI that scans a dbt project, profiles the real data (DuckDB), proposes the dbt tests you're missing (`not_null`, `unique`, `accepted_values`, `relationships`, and — with an LLM — custom ones), writes the ones you approve into `schema.yml`, and lets `dbt test` run them.

---

## Contents

- [How it works](#how-it-works)
- [Requirements](#requirements)
- [1. Install](#1-install)
- [2. Add your Gemini key (.env)](#2-add-your-gemini-key-env)
- [3. The workflow (commands)](#3-the-workflow-commands)
- [Custom tests explained](#custom-tests-explained)
- [Command reference](#command-reference)
- [Worked example A — Olist (clone & run)](#worked-example-a--olist-clone--run)
- [Worked example B — bookshop (create from scratch)](#worked-example-b--bookshop-create-from-scratch)
- [Gotchas & tips](#gotchas--tips)

---

## How it works

```
profile  →  propose  →  apply  →  dbt test
```

1. **profile** — reads the project's `manifest.json`/`catalog.json` and profiles each model's real data in DuckDB (row counts, null %, cardinality, ranges, value lists).
2. **propose** — turns those profiles into proposed tests. Deterministic **heuristics** by default; add `--llm` for AI-suggested rationale, relationships, and custom tests.
3. **apply** — you review each proposal (approve/reject) and the approved ones are written into the models' `schema.yml`.
4. **dbt test** — runs them.

The design is **heuristics-first, LLM-augmented**: obvious tests come from rules; the LLM adds the rest. Nothing is written without your approval.

---

## Requirements

- **Python 3.10–3.13**
- A **DuckDB-backed dbt project** (uses the `dbt-duckdb` adapter). The tool reads your data from the project's `.duckdb` file.
- A **Gemini** (or Groq) API key — only needed for `--llm`.

---

## 1. Install

```bash
python3 -m venv .venv && source .venv/bin/activate

pip install dbt-duckdb          # to build the dbt project
pip install dbt-testpilot       # the tool
pip install google-genai        # only needed for --llm (Gemini)
```

`dbt-testpilot`'s only hard dependencies are `duckdb` and `ruamel.yaml`. The LLM SDKs are optional — `pip install google-genai` (Gemini) and/or `pip install groq` (Groq). You can also get both at once with `pip install "dbt-testpilot[llm]"`.

---

## 2. Add your Gemini key (.env)

The `--llm` step reads its provider and key from a `.env` file in the project directory.

**Get a key:** go to <https://aistudio.google.com> → **Get API key** → **Create API key** (no credit card needed).

**Create `.env`** in your dbt project folder:

```bash
cat > .env <<'ENV'
LLM_PROVIDER=gemini
GEMINI_API_KEY=paste-your-gemini-key-here
ENV
```

Then replace `paste-your-gemini-key-here` with your real key.

- To use **Groq** instead, set `LLM_PROVIDER=groq` and add `GROQ_API_KEY=...`.
- **Never commit `.env`.** Add it to `.gitignore`:

```bash
echo ".env" >> .gitignore
```

**Verify the key works** (optional): `dbt-testpilot` will tell you if the key is missing when you run `--llm`. If a call fails, it's almost always the key not being read from `.env`.

---

## 3. The workflow (commands)

Run these from inside your dbt project (after it's built with `dbt run` / `dbt build` and `dbt docs generate`):

```bash
# 1) profile the real data
dbt-testpilot profile --project-dir . --out profiles.json
#    if the .duckdb file lives outside the project dir, point at it:
#    dbt-testpilot profile --project-dir . --db path/to/your.duckdb --out profiles.json

# 2) propose the missing tests (with LLM augmentation)
dbt-testpilot propose --profiles profiles.json --out proposals.json --llm
#    drop --llm for heuristics only (no key needed)

# 3) review + write the ones you approve  (prompts: y / n / a / q)
dbt-testpilot apply --proposals proposals.json --project-dir .
#    to also write LLM custom tests (and auto-generate their macros):
#    dbt-testpilot apply --proposals proposals.json --project-dir . --include-custom

# 4) run them
dbt test --profiles-dir .
```

---

## Custom tests explained

dbt ships **four** built-in generic tests: `not_null`, `unique`, `accepted_values`, `relationships`. dbt runs a test named `foo` by looking up a macro `test_foo`.

When you use `--llm`, the model also proposes **custom** tests by name — e.g. `is_not_negative`, `positive_amount`, `date_in_past_or_present`, or bespoke ones like `is_total_amount_correct`. dbt has no macro for these, so:

- **By default, `apply` skips custom tests** (with a note) — this keeps `dbt test` green.
- **With `--include-custom`, dbt-testpilot generates the macros for you.** It writes vetted macros into `macros/dbt_testpilot/` for the patterns it recognizes:
  - `non_negative` (value ≥ 0) and `positive` (value > 0) — kept distinct, chosen from the column's minimum
  - `not_future` (a date/timestamp not in the future)
  - `not_empty_string` (text that must not be blank)
  - `is_integer` (a numeric column of whole numbers)
- **Genuinely bespoke** custom tests (multi-column or regex logic — e.g. comparing two columns, checking a total equals the sum of parts) are **still skipped** and listed for you to implement, because auto-writing SQL that can't be guaranteed would risk false confidence.

So `--include-custom` turns the *recognizable* LLM suggestions into real, passing tests, and leaves the rest to you. The generated macros live in your project under `macros/dbt_testpilot/test_<name>.sql`.

## Verified before writing

`apply` checks every approved test against your real data before writing it — it runs the equivalent query on the DuckDB and **only writes tests with zero violations**, so `dbt test` comes back green. Anything the data would fail (`unique` on a duplicated column, `positive` on a column with `0`s, a `relationships` FK with orphans) is **not written** and is reported as `would FAIL: N rows` — treat those as real data issues to investigate. The DuckDB is auto-detected from `--project-dir`; pass `--db path/to/your.duckdb` if it lives elsewhere, or `--no-verify` to skip (not recommended).

---

## Command reference

**`profile`**

| Flag | Meaning |
|---|---|
| `--project-dir` | dbt project dir (has `target/` + the `.duckdb`). Default `.` |
| `--db` | Path to the DuckDB file (auto-detected from `--project-dir` if omitted) |
| `--out` | Write per-model profiles to this JSON file |
| `--max-values` | Max distinct values captured for low-cardinality columns (default 25) |

**`propose`**

| Flag | Meaning |
|---|---|
| `--profiles` | The `profiles.json` from `profile` (default `profiles.json`) |
| `--out` | Write proposals to this JSON file |
| `--llm` | Add LLM-suggested proposals (rationale, relationships, custom tests) |
| `--provider` | `gemini` or `groq` (default: `LLM_PROVIDER` from `.env`) |
| `--env` | Path to the `.env` with your keys (default `.env`) |

**`apply`**

| Flag | Meaning |
|---|---|
| `--proposals` | The `proposals.json` from `propose` (default `proposals.json`) |
| `--project-dir` | dbt project whose `schema.yml` files get updated |
| `--yes` | Approve all (non-interactive) |
| `--min-confidence` | Only consider proposals at/above `low` / `medium` / `high` |
| `--dry-run` | Preview what would be written; change nothing |
| `--include-custom` | Generate macros for supported custom tests and write them |
| `--db` | DuckDB file used to verify tests before writing (auto-detected from `--project-dir` if omitted) |
| `--no-verify` | Skip data verification (not recommended) |
| `--run` | Run `dbt test` after writing |

---

## Worked example A — Olist (clone & run)

A real 100k-row Brazilian e-commerce warehouse (8 CSV sources → staging → dim/fact → reporting, 25 models).

```bash
mkdir -p ~/dbt-tests && cd ~/dbt-tests
git clone https://github.com/nelsonic1/duckdb-dbt.git
cd duckdb-dbt/dbt

python3 -m venv .venv && source .venv/bin/activate
pip install dbt-duckdb dbt-testpilot google-genai

# Gemini key for --llm
cat > .env <<'ENV'
LLM_PROVIDER=gemini
GEMINI_API_KEY=paste-your-gemini-key-here
ENV

# build the project (Olist CSVs ship inside the repo)
dbt deps                              # installs dbt_utils
dbt run --profiles-dir .              # builds all 25 models
dbt docs generate --profiles-dir .

# run dbt-testpilot — its .duckdb lives one level up, so pass --db
dbt-testpilot profile --project-dir . --db ../duckdb/database.duckdb --out profiles.json
dbt-testpilot propose --profiles profiles.json --out proposals.json --llm
dbt-testpilot apply   --proposals proposals.json --project-dir . --include-custom
dbt test --profiles-dir .
```

Expect ~150+ proposals and a mostly-green `dbt test` with a few **`relationships` failures** — those are real orphan-foreign-key findings in the Olist data (e.g. payments/reviews referencing missing orders), not bugs.

---

## Worked example B — bookshop (create from scratch)

A small synthetic project (authors / books / sales) — nothing to clone, you create it.

```bash
mkdir -p ~/dbt-tests/bookshop && cd ~/dbt-tests/bookshop
python3 -m venv .venv && source .venv/bin/activate
pip install dbt-duckdb dbt-testpilot google-genai

# Gemini key for --llm
cat > .env <<'ENV'
LLM_PROVIDER=gemini
GEMINI_API_KEY=paste-your-gemini-key-here
ENV

mkdir -p seeds models/staging

cat > dbt_project.yml <<'YML'
name: 'bookshop'
version: '1.0.0'
config-version: 2
profile: 'bookshop'
model-paths: ["models"]
seed-paths: ["seeds"]
YML

cat > profiles.yml <<'YML'
bookshop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: 'bookshop.duckdb'
      threads: 4
YML

cat > seeds/raw_authors.csv <<'CSV'
author_id,author_name,country
1,Jane Austen,UK
2,Charles Dickens,UK
3,Mark Twain,USA
4,Ernest Hemingway,USA
5,Leo Tolstoy,Russia
6,Fyodor Dostoevsky,Russia
7,Haruki Murakami,Japan
8,Yukio Mishima,Japan
9,Chinua Achebe,Nigeria
10,Wole Soyinka,Nigeria
CSV

cat > seeds/raw_books.csv <<'CSV'
book_id,title,author_id,genre,price,published_date
1,Pride and Prejudice,1,fiction,12.99,1813-01-28
2,Great Expectations,2,fiction,14.00,1861-08-01
3,Adventures of Huckleberry Finn,3,fiction,10.99,1884-12-10
4,The Old Man and the Sea,4,fiction,9.99,1952-09-01
5,War and Peace,5,fiction,19.99,1869-01-01
6,Crime and Punishment,6,mystery,15.99,1866-01-01
7,Norwegian Wood,7,fiction,13.50,1987-09-04
8,Confessions of a Mask,8,nonfiction,13.00,1949-07-05
9,Things Fall Apart,9,fiction,12.00,1958-06-17
10,Selected Poems,10,poetry,8.99,1990-01-01
CSV

cat > seeds/raw_sales.csv <<'CSV'
sale_id,book_id,quantity,amount,sale_date,status
1,1,2,25.98,2024-01-15,completed
2,3,1,10.99,2024-01-20,completed
3,5,1,19.99,2024-02-02,refunded
4,2,3,42.00,2024-02-11,completed
5,6,1,15.99,2024-02-15,pending
6,1,1,12.99,2024-03-01,completed
7,9,2,24.00,2024-03-05,completed
8,7,1,13.50,2024-03-09,refunded
9,10,4,35.96,2024-03-12,completed
10,4,1,9.99,2024-03-18,completed
CSV

cat > models/staging/stg_authors.sql <<'SQL'
select author_id, author_name, country from {{ ref('raw_authors') }}
SQL
cat > models/staging/stg_books.sql <<'SQL'
select book_id, title, author_id, genre, price, published_date from {{ ref('raw_books') }}
SQL
cat > models/staging/stg_sales.sql <<'SQL'
select sale_id, book_id, quantity, amount, sale_date, status from {{ ref('raw_sales') }}
SQL
cat > models/books_enriched.sql <<'SQL'
select b.book_id, b.title, b.genre, b.price, a.author_name, a.country
from {{ ref('stg_books') }} b
join {{ ref('stg_authors') }} a on b.author_id = a.author_id
SQL

cat > models/schema.yml <<'YML'
version: 2
models:
  - name: stg_books
    description: One row per book, with pricing and author.
    columns:
      - name: book_id
        description: Primary key.
        tests:
          - unique
          - not_null
YML

# build
dbt seed --profiles-dir .
dbt run  --profiles-dir .
dbt docs generate --profiles-dir .

# run dbt-testpilot (.duckdb is in this folder, so no --db needed)
dbt-testpilot profile --project-dir . --out profiles.json
dbt-testpilot propose --profiles profiles.json --out proposals.json --llm
dbt-testpilot apply   --proposals proposals.json --project-dir . --include-custom
dbt test --profiles-dir .
```

On this tiny 10-row dataset you'll see some over-eager `unique`/`accepted_values` proposals (small samples make columns look distinct) — **reject those at the `apply` prompt.** That's the human-in-the-loop step doing its job.

---

## Gotchas & tips

- **`.duckdb` outside the project dir?** Pass `--db path/to/file.duckdb` to `profile`.
- **Small datasets over-propose.** With few rows, columns look unique/categorical, so you'll get noisy `unique`/`accepted_values` suggestions — reject them. On real row counts the proposals are clean.
- **`relationships` failures on real data are usually real findings** (orphan foreign keys), not bugs — that's the tool earning its keep.
- **Views can be memory-heavy to test.** If a model is an expensive view, many parallel tests re-compute it; use fewer `--threads` or materialize it as a table if you hit memory limits.
- **Scope: DuckDB projects.** The profiler reads data from the `.duckdb` file, so it works with the `dbt-duckdb` adapter. Other warehouses aren't supported yet.
- **`--llm` errors?** Check that `.env` has `LLM_PROVIDER` + a valid `GEMINI_API_KEY`, and that `google-genai` is installed.

---

**Install:** `pip install dbt-testpilot` · **PyPI:** <https://pypi.org/project/dbt-testpilot/> · **License:** MIT
