# dbt-testpilot — Olist demo (from scratch)

A self-contained demo project for recording a `dbt-testpilot` walkthrough. It ships a
compact, **Olist-shaped** dataset with deliberately planted data-quality problems, so the
tool has something real to catch:

- some `payment_value` / `freight_value` rows are `0.0` → `positive` fails but `non_negative` passes
- `order_status` is a small categorical set → `accepted_values` fits
- a handful of `order_items` / `payments` are **orphans** (FK to a non-existent order) → `relationships` fails on those rows
- one city is `São Paulo` (unicode) → exercises the UTF-8 path

`dbt-testpilot` profiles the data, proposes the missing tests, **verifies each one against the
data, and writes only the ones that pass** — then `dbt test` runs them.

---

## What's in this folder

```
olist-demo/
├── seed_data.py                 # generates the DuckDB sample (olist.duckdb)
├── dbt_project.yml
├── profiles.yml                 # duckdb target -> olist.duckdb
├── models/staging/
│   ├── sources.yml
│   ├── stg_customers.sql  stg_orders.sql  stg_order_items.sql  stg_payments.sql
│   └── schema.yml               # model/column docs; tests get written in here
├── demo.tape                    # VHS script to record demo.gif (with --llm)
├── .env.example                 # copy to .env, add your GEMINI_API_KEY
├── demo-preview.gif             # ready-made preview (heuristics run)
└── README-DEMO.md               # this file
```

---

## 1. Prerequisites

```bash
# from inside olist-demo/
python3 -m venv .venv && source .venv/bin/activate

pip install dbt-duckdb dbt-testpilot
pip install "dbt-testpilot[llm]" google-genai      # for the --llm step
```

For the `--llm` step, add your key:

```bash
cp .env.example .env
# edit .env  ->  GEMINI_API_KEY=your_key_here
```

Keep dbt's profile lookup local to this folder:

```bash
export DBT_PROFILES_DIR=$PWD
```

## 2. Run the workflow

```bash
# a) load the sample data into olist.duckdb
python seed_data.py

# b) build the dbt models + the catalog dbt-testpilot reads
dbt build
dbt docs generate

# c) profile the real data
dbt-testpilot profile  --project-dir . --out profiles.json

# d) propose missing tests (heuristics + Gemini custom checks)
dbt-testpilot propose  --profiles profiles.json --out proposals.json --llm

# e) verify against the data, write only the passing tests
dbt-testpilot apply    --proposals proposals.json --project-dir . --yes --include-custom

# f) run the tests it just wrote
dbt test
```

Expected: `apply` reports the 2 orphan-FK `relationships` as **would FAIL — NOT written**, writes
the verified tests into `models/staging/schema.yml`, and `dbt test` finishes **PASS with 0 errors**.

Drop `--llm` (and skip the `.env` step) for a heuristics-only run — that's what
`demo-preview.gif` shows.

## 3. Record the GIF

```bash
brew install vhs          # one-time; also pulls in ttyd + ffmpeg
vhs demo.tape             # -> writes demo.gif
```

Re-run `python seed_data.py` first if you've already applied the tests, so the demo starts clean.

## 4. Turn it into a narrated video (optional)

1. Connect **Descript** (in this app).
2. Import `demo.gif` (or a screen recording of the run).
3. Add an AI voiceover / captions describing each step, trim dead time, export MP4.

`demo-preview.gif` is a lightweight preview to check pacing before you record the real thing.
